<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Serverless | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/serverless/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/serverless/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Tue, 29 Sep 2026 06:05:28 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Serverless | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/serverless/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>RDS vs DynamoDB: The Benchmark Isn&#8217;t the Decision</title>
		<link>https://john-nessime.com/blog/technical-guides/rds-vs-dynamodb-access-patterns/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 29 Sep 2026 06:05:28 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Databases]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Aurora]]></category>
		<category><![CDATA[Amazon RDS]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Capacity Planning]]></category>
		<category><![CDATA[Connection Pooling]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Modeling]]></category>
		<category><![CDATA[DynamoDB]]></category>
		<category><![CDATA[Hot Partition]]></category>
		<category><![CDATA[Over-Engineering]]></category>
		<category><![CDATA[PostgreSQL]]></category>
		<category><![CDATA[Provisioned Throughput]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Throttling]]></category>
		<category><![CDATA[Workload Shape]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=738</guid>

					<description><![CDATA[<p>Latency charts don't decide this. The query nobody wrote down at the start does. How to build an access pattern inventory, where DynamoDB and RDS each genuinely win and hurt, a six-question decision procedure, and the arguments on both sides that fall apart under a real workload.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/rds-vs-dynamodb-access-patterns/">RDS vs DynamoDB: The Benchmark Isn&#8217;t the Decision</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone drops a benchmark link into the design review. Flat latency line for DynamoDB, a rising one for Postgres, and the data store is chosen before anybody has described what the application does.</p>



<p class="wp-block-paragraph">That argument settles architecture decisions all the time, and it&#8217;s the wrong argument. Both services return a primary-key lookup faster than your network hop and your ORM overhead. What hurts six months later is the query nobody wrote down at the start, and no benchmark will warn you about it.</p>



<p class="wp-block-paragraph">So this post skips the latency charts. It covers the access pattern inventory you write before picking anything, where each service genuinely wins and genuinely hurts, a decision procedure you can run in an afternoon, and the arguments on both sides that fall apart on contact with a real workload.</p>



<h2 class="wp-block-heading">Why the benchmark answers a question you don&#8217;t have</h2>



<p class="wp-block-paragraph">A benchmark measures a fixed operation under fixed conditions. Your application is not a fixed operation. It&#8217;s a set of reads and writes that grows every time somebody asks a new question about the data.</p>



<p class="wp-block-paragraph">The failure mode that decides RDS vs DynamoDB is almost never throughput. It&#8217;s a product manager asking, eight months in, for every account that hasn&#8217;t logged in this quarter, broken down by plan tier. On Postgres that&#8217;s an awkward query and maybe an index. On DynamoDB, if you didn&#8217;t model for it, it&#8217;s a full table scan, a new global secondary index, a backfill, and a standing write-capacity tax to keep it in sync.</p>



<p class="wp-block-paragraph">That asymmetry is the whole decision. A relational database defers the modeling cost and charges a little on every query for the flexibility. DynamoDB front-loads it: design keys around known patterns, get cheap predictable reads for exactly those, and pay dearly for anything else.</p>



<h2 class="wp-block-heading">Write the access pattern inventory first</h2>



<p class="wp-block-paragraph">Before comparing anything, write the list. Every read and write the application performs, in plain language, with the shape of the operation and how fast it needs to be. This much is enough:</p>



<pre class="wp-block-code"><code>PATTERN                                SHAPE           FREQ    LATENCY
get order by id                        point read      high    p99 &lt; 50ms
list a customer's orders, newest first range read      high    p99 &lt; 100ms
orders by status, across customers     cross-entity    low     seconds fine
revenue by product category, monthly   aggregate       low     minutes fine
find orders by partial address         text match      medium  p99 &lt; 300ms
mark order shipped + write audit row   multi-item write high   p99 &lt; 100ms</code></pre>



<p class="wp-block-paragraph">Now read the SHAPE column. If nearly every row is a point or range read, and the exceptions tolerate latency, DynamoDB is a live option. If the column is full of cross-entity reads, aggregates and text matching, you&#8217;re describing a relational workload and can stop shopping.</p>



<h3 class="wp-block-heading">The rows people leave out</h3>



<p class="wp-block-paragraph">The inventory is only useful if it&#8217;s honest, and the omissions are predictable:</p>



<ul class="wp-block-list">
<li>The admin back office. Support staff search by email, phone, partial name, anything. Ad hoc queries, and they never appear in the first inventory.</li>



<li>Reporting. Someone will want a monthly rollup. Decide now whether it lives in the operational store or streams out to a warehouse.</li>



<li>Deletion. Regulatory deletes cascade across entities. In a normalized schema that&#8217;s a transaction. In a denormalized single table it&#8217;s a fan-out you have to write and test yourself.</li>



<li>Backfills. Every schema change becomes a migration job when no schema is enforced at the storage layer.</li>
</ul>



<p class="wp-block-paragraph">Write them down even if the answer is &#8220;we&#8217;ll handle it elsewhere.&#8221; A pattern you consciously exile to a warehouse is fine. One you forgot is a rewrite.</p>



<h2 class="wp-block-heading">Where DynamoDB wins, and where it doesn&#8217;t</h2>



<p class="wp-block-paragraph">Give it its genuine case first, because it&#8217;s a strong one.</p>



<h3 class="wp-block-heading">Where it wins</h3>



<ul class="wp-block-list">
<li><strong>Connection model.</strong> It&#8217;s an HTTPS API with SigV4 auth, so there&#8217;s no pool to exhaust. That&#8217;s why it pairs cleanly with Lambda. A relational database behind high Lambda concurrency needs RDS Proxy or a pooler, which is another component you operate.</li>



<li><strong>Predictable reads at any size.</strong> A query against a well-chosen partition key costs the same at ten thousand items or ten billion. Nothing in Postgres gives you that for free.</li>



<li><strong>Idle cost.</strong> On-demand mode bills per request unit consumed, so a table with no traffic costs storage and nothing else. Useful for tenant-per-table designs and low-traffic side services.</li>



<li><strong>Operational surface.</strong> No minor version upgrades, no vacuum, no failover window, no parameter groups to argue about.</li>
</ul>



<p class="wp-block-paragraph">The query API is narrow on purpose. Give it an exact partition key plus an optional sort key condition, and it returns an ordered slice of an item collection:</p>



<pre class="wp-block-code"><code>aws dynamodb query 
  --table-name AppData 
  --key-condition-expression "PK = :pk AND begins_with(SK, :sk)" 
  --expression-attribute-values '{":pk":{"S":"CUSTOMER#42"},":sk":{"S":"ORDER#"}}'</code></pre>



<p class="wp-block-paragraph">One customer&#8217;s orders, in sort order, in one request, no join. If that call shape covers most of your inventory, you&#8217;re looking at the right service.</p>



<h3 class="wp-block-heading">Where it doesn&#8217;t</h3>



<ul class="wp-block-list">
<li><strong>Anything you didn&#8217;t model.</strong> New patterns mean new keys or a new index, and a new global secondary index means a backfill plus write capacity on every future write.</li>



<li><strong>Filtering is not querying.</strong> A <code>FilterExpression</code> runs after the read, so you pay capacity for every item examined and only save bandwidth. A filter that discards most of what it reads is a design smell.</li>



<li><strong>Aggregates.</strong> There is no <code>GROUP BY</code> and no <code>COUNT(*)</code> over a table. You maintain aggregates yourself, usually with DynamoDB Streams feeding a Lambda that updates a counter item, which means you own the correctness problem when that consumer fails.</li>



<li><strong>Hard physical limits.</strong> An item caps at 400 KB. A Query or Scan page returns at most 1 MB before you paginate. A single partition sustains roughly 3,000 read and 1,000 write units per second, so a hot key throttles while the table sits well under its quota.</li>



<li><strong>Index budget.</strong> Global secondary indexes default to twenty per table; local secondary indexes cap at five and can only be defined at table creation. You can&#8217;t index your way out of a bad key design forever.</li>
</ul>



<h2 class="wp-block-heading">Where RDS wins, and where it doesn&#8217;t</h2>



<h3 class="wp-block-heading">Where it wins</h3>



<ul class="wp-block-list">
<li><strong>Unknown future queries.</strong> The entire point. Joins, aggregates, window functions, ad hoc analysis, and a new index in one statement when something turns out slow.</li>



<li><strong>Constraints the database enforces.</strong> Foreign keys, uniqueness, check constraints, transactions across arbitrary tables. In DynamoDB each of those is application code you write and maintain.</li>



<li><strong>Extensions.</strong> PostGIS for geospatial, pgvector for embeddings, full-text search built in, all against the same rows in the same transaction.</li>



<li><strong>Portability.</strong> Postgres and MySQL run on RDS, in a container, or on a plain VPS from a provider like Contabo or InterServer, so migrating off is an export and an import. DynamoDB&#8217;s data model doesn&#8217;t port without rewriting your data access layer.</li>



<li><strong>Everyone already knows SQL.</strong> Underrated on a small team. Teaching four developers single-table design costs real review time.</li>
</ul>



<h3 class="wp-block-heading">Where it doesn&#8217;t</h3>



<ul class="wp-block-list">
<li><strong>Connections under serverless concurrency.</strong> Each Lambda execution environment opens its own connection, and the database refuses new ones once <code>max_connections</code> is reached. RDS Proxy multiplexes to fix that, but some operations pin a client to a backend connection and pinned connections aren&#8217;t reused. Watch <code>DatabaseConnectionsCurrentlySessionPinned</code> before trusting the pool.</li>



<li><strong>Instances are always on.</strong> A provisioned instance bills whether or not anyone queries it. Aurora Serverless v2 changes that with a minimum of zero capacity units on a supported engine version, but resuming from paused is slow enough that the first user after an idle period feels it. AWS documents resume as taking roughly fifteen seconds.</li>



<li><strong>Vertical scaling has a ceiling.</strong> Writes go to one primary and read replicas help reads only. Sharding is a project, not a setting.</li>



<li><strong>Maintenance is yours.</strong> Version upgrades, failover testing, long transactions blocking vacuum, connection storms on restart. Managed does not mean absent.</li>
</ul>



<p class="wp-block-paragraph">If you want a database that hibernates properly, it lives in the scaling configuration and the idle timeout is yours to set:</p>



<pre class="wp-block-code"><code>aws rds create-db-cluster 
  --db-cluster-identifier app-cluster 
  --engine aurora-postgresql 
  --serverless-v2-scaling-configuration 
      MinCapacity=0,MaxCapacity=4,SecondsUntilAutoPause=600 
  --manage-master-user-password</code></pre>



<p class="wp-block-paragraph">A minimum of zero only works on engine versions supporting automatic pause and resume, so check yours before planning a budget around it. If the cluster refuses the setting, that&#8217;s usually why.</p>



<h2 class="wp-block-heading">A decision procedure for RDS vs DynamoDB</h2>



<p class="wp-block-paragraph">Run these in order against the inventory you wrote. The first clear answer wins.</p>



<ol class="wp-block-list">
<li><strong>Can you enumerate the access patterns, and are they stable?</strong> If the product is still moving and nobody can say what the admin tools will need, pick the store that tolerates being wrong. That&#8217;s RDS.</li>



<li><strong>Does any pattern need a cross-entity aggregate on the hot path?</strong> A user-facing screen needing a count or sum across entities in under a second, with no way to precompute it, means RDS.</li>



<li><strong>Is there a natural partition key with even distribution?</strong> Tenant ID, user ID, device ID. If traffic concentrates on one key, the per-partition ceiling finds you regardless of capacity mode. No good key means no DynamoDB.</li>



<li><strong>What&#8217;s the concurrency shape of the compute tier?</strong> Thousands of short-lived Lambda invocations argue for DynamoDB. A few long-running containers with a warm pool remove the strongest argument against RDS.</li>



<li><strong>Does the workload go quiet?</strong> Bursty and mostly idle favors on-demand DynamoDB or a scale-to-zero Aurora cluster. Steady traffic makes provisioned capacity or a right-sized instance cheaper.</li>



<li><strong>Who maintains this in a year?</strong> If it&#8217;s one part-time developer, weight familiarity heavily. A design nobody on the team can debug is a liability whatever the benchmark said.</li>
</ol>



<h2 class="wp-block-heading">The cost model nobody actually models</h2>



<p class="wp-block-paragraph">The two bill on axes that don&#8217;t compare. RDS charges instance hours, storage, and depending on engine and storage type, I/O, busy or not. DynamoDB charges request units and storage, so an idle table is nearly free and a chatty one isn&#8217;t.</p>



<p class="wp-block-paragraph">The number that matters is request units per user action, and almost nobody calculates it. Reads bill in 4 KB increments and writes in 1 KB increments, strongly consistent reads cost twice what eventually consistent ones do, and transactional operations double again. Every index you project into consumes write units on every write that touches it. A screen firing six queries plus an audit write can cost many times the naive estimate.</p>



<p class="wp-block-paragraph">Work it out per action, multiply by real traffic, and sanity check the instance side with my <a href="https://john-nessime.com/tools/aws-cost-estimator/">AWS cost estimator</a> before the decision hardens. AWS recommends on-demand as the default capacity mode for most workloads, but steady predictable traffic is still cheaper provisioned, and you can flip between them:</p>



<pre class="wp-block-code"><code># which mode is this table on?
aws dynamodb describe-table --table-name AppData 
  --query "Table.BillingModeSummary.BillingMode"

# what is it actually consuming?
aws cloudwatch get-metric-statistics 
  --namespace AWS/DynamoDB 
  --metric-name ConsumedWriteCapacityUnits 
  --dimensions Name=TableName,Value=AppData 
  --statistics Sum --period 3600 
  --start-time $(date -u -d '1 day ago' +%Y-%m-%dT%H:%M:%SZ) 
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ)</code></pre>



<p class="wp-block-paragraph">Switching billing mode is rate limited, so treat it as a planned change rather than an incident-time knob.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<ul class="wp-block-list">
<li><strong>&#8220;DynamoDB scales infinitely.&#8221;</strong> The service does. Your partition key might not. Uneven access concentrates load on one partition, and adaptive capacity only helps within limits.</li>



<li><strong>&#8220;Postgres can&#8217;t handle our scale.&#8221;</strong> Usually said before anyone has run <code>EXPLAIN (ANALYZE, BUFFERS)</code> on the slow query. A missing index and an N+1 in the ORM explain most claims of outgrowing relational.</li>



<li><strong>&#8220;We&#8217;ll add the indexes later.&#8221;</strong> True in Postgres, expensive in DynamoDB. A new global secondary index backfills across the whole table and then taxes every subsequent write.</li>



<li><strong>&#8220;Serverless database means no operations.&#8221;</strong> You trade instance operations for capacity and key-design operations. Throttling, hot partitions and stream consumer lag all need dashboards, exactly like a relational instance does.</li>



<li><strong>&#8220;We can just use both.&#8221;</strong> Sometimes right, often a way to avoid deciding. Two stores means two consistency models, two restore procedures, and a sync path that becomes the least reliable component you own.</li>
</ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<p class="wp-block-paragraph">My default for a new application with an unfinished product definition is managed Postgres. Not because it&#8217;s better, but because being wrong about a relational schema costs an afternoon and being wrong about a key design costs a migration.</p>



<p class="wp-block-paragraph">I&#8217;d reach for DynamoDB first when the workload is a small closed set of patterns keyed off one identifier, with high write volume, tolerance for eventual consistency, and concurrency that would flatten a connection pool. Session stores, telemetry ingestion, per-user feeds, idempotency keys, feature flags. It&#8217;s excellent at all of those.</p>



<p class="wp-block-paragraph">The hybrid worth doing is narrow: relational as the system of record, DynamoDB for one high-volume pattern that joins nothing. The hybrid to avoid splits related entities across both stores and reconciles them in application code.</p>



<p class="wp-block-paragraph">Whichever you land on, instrument it on day one. Throttled requests and consumed capacity on the DynamoDB side, connection counts and slow queries on the relational side, on a CloudWatch or Grafana Cloud dashboard someone actually opens. If you&#8217;re restricting item-level access with IAM condition keys such as <code>dynamodb:LeadingKeys</code>, run the policy through my <a href="https://john-nessime.com/tools/aws-iam-policy-checker/">IAM policy checker</a> before it ships.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is DynamoDB always faster than RDS?</h3>



<p class="wp-block-paragraph">For a primary-key lookup at large scale its latency is more consistent, because that operation&#8217;s cost doesn&#8217;t grow with table size. For a query it wasn&#8217;t modeled for it can be far slower, because the fallback is a scan. Speed depends on the match between query and key design, not on the service.</p>



<h3 class="wp-block-heading">Can I run analytics on DynamoDB?</h3>



<p class="wp-block-paragraph">Not directly, and you shouldn&#8217;t try. Export the table to S3 or stream changes out, then query with Athena or load a warehouse. Treat operational and analytical stores as separate systems.</p>



<h3 class="wp-block-heading">What about Aurora Serverless as the middle ground?</h3>



<p class="wp-block-paragraph">It solves idle cost, not connections or write scaling. If you&#8217;re considering DynamoDB because of Lambda concurrency or single-primary write limits, Aurora Serverless doesn&#8217;t address that. If it&#8217;s because you&#8217;re paying for an idle dev database, it does.</p>



<h3 class="wp-block-heading">How do I migrate from DynamoDB to RDS if I get it wrong?</h3>



<p class="wp-block-paragraph">Export to S3, reshape denormalized items into normalized tables, load, and dual-write during cutover. The data movement is routine. The hard part is that your application code assumed a key-value access shape everywhere, so the rewrite lands in the service layer, not the pipeline.</p>



<h3 class="wp-block-heading">Is DynamoDB cheaper than RDS for a small application?</h3>



<p class="wp-block-paragraph">Usually yes when traffic is low and bursty, because on-demand charges per request against an idle table while an instance bills continuously. The advantage narrows as traffic steadies and reverses once throughput justifies provisioned capacity or a reserved instance.</p>



<h3 class="wp-block-heading">Do I need RDS Proxy if I use Lambda with Postgres?</h3>



<p class="wp-block-paragraph">At meaningful concurrency, yes. Below that, a direct connection reused across invocations is fine, and the proxy adds cost plus a pinning failure mode. Add it when connection counts approach your limit, not preemptively in development.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">The one thing worth remembering about RDS vs DynamoDB: you&#8217;re not choosing a performance profile, you&#8217;re choosing when to pay the modeling cost. DynamoDB takes it up front and pays you back in predictability. A relational database lets you defer it and charges interest later in query complexity.</p>



<p class="wp-block-paragraph">Write the access pattern inventory. Be honest about the admin screens and the reporting nobody has asked for yet. If the list is short, stable and keyed on one identifier, DynamoDB will serve you for years. If it isn&#8217;t, take the relational database and spend the saved effort on indexes and connection handling.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second opinion on the data store decision?</h2>



<p class="wp-block-paragraph">Cheap to review now, expensive to revisit later. Things I help with:</p>



<ul class="wp-block-list">
<li>Building the access pattern inventory with your team and pressure-testing it against the queries your roadmap implies</li>



<li>Reviewing a DynamoDB key and index design before it ships, including partition key distribution and index write amplification</li>



<li>Diagnosing throttling and hot partitions on an existing table, and whether the fix is a key change, a capacity change or a cache</li>



<li>Sizing RDS or Aurora for a serverless compute tier, including connection handling, RDS Proxy configuration and pinning</li>



<li>Modeling the cost of both options against your real request volume instead of a rule of thumb</li>



<li>Planning a migration in either direction, with a dual-write cutover and a rollback that works</li>
</ul>



<p class="wp-block-paragraph">Send me your access pattern list, a slow query plan, or your throttling metrics and I&#8217;ll tell you what I&#8217;d do with it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/rds-vs-dynamodb-access-patterns/">RDS vs DynamoDB: The Benchmark Isn&#8217;t the Decision</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Turning External Documents Into Structured Data With Amazon Textract</title>
		<link>https://john-nessime.com/blog/technical-guides/amazon-textract-structured-data/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 22 Sep 2026 06:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon SNS]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Bedrock Data Automation]]></category>
		<category><![CDATA[Boto3]]></category>
		<category><![CDATA[Confidence Scoring]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Human In The Loop]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[OCR]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[Serverless]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=390</guid>

					<description><![CDATA[<p>Amazon Textract almost never fails loudly. A working guide to turning external PDFs and scans into structured data: picking the right operation per document family, parsing the block graph, routing on per-field confidence, and catching the limits that silently truncate your records.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/amazon-textract-structured-data/">Turning External Documents Into Structured Data With Amazon Textract</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A vendor sends over a year of delivery notes. It arrives as one PDF, several hundred pages, scanned on a machine that was clearly overdue for service. Someone asks how long it will take to get the reference numbers into the warehouse table, and the honest answer is that the OCR call is the easy part.</p>



<p class="wp-block-paragraph">Document pipelines are not hard because optical character recognition is inaccurate. Amazon Textract reads text well. They are hard because everything after the API call belongs to you, and because Textract almost never fails loudly. You get a 200 response, a list of blocks, a confidence score, and a record with an empty field where the purchase order number should have been. Nothing alerts. The row lands in the warehouse with a null, and someone in finance notices three weeks later while reconciling.</p>



<p class="wp-block-paragraph">That silent gap is the whole problem. Getting reliable Amazon Textract structured data out of files you did not design has less to do with OCR accuracy than with everything around it: which operation to call for which document family, why the response is a graph rather than a record, how to route on confidence instead of trusting it, and the limits that quietly truncate your data. All of it assumes documents you cannot ask anyone to change.</p>



<h2 class="wp-block-heading">Why external documents break pipelines that passed testing</h2>



<p class="wp-block-paragraph">Internal documents are generated by a template you control. External documents vary along three axes at once, and each one has its own failure mode:</p>



<ul class="wp-block-list">
<li><strong>Container.</strong> One logical document per file, or forty stapled into one PDF, or a single page split across three JPEGs.</li>

<li><strong>Text origin.</strong> Born-digital PDFs with a real text layer, flatbed scans, and photographs taken at an angle in bad light. Same file extension, wildly different accuracy.</li>

<li><strong>Layout stability.</strong> The vendor redesigns their invoice template and moves the PO number from the header to a footer block. Nobody tells you.</li>
</ul>



<p class="wp-block-paragraph">You cannot fix any of that upstream. You can only detect it at ingest and route accordingly. The sections below are organized by the failure family each one produces.</p>



<h2 class="wp-block-heading">Failure family one: the file is out of spec before Textract sees it</h2>



<p class="wp-block-paragraph">Textract accepts JPEG, PNG, PDF, and TIFF. It does not support XFA-based PDFs, and it will not open a password-protected PDF. Images are accepted up to 10,000 pixels on any side. For PDFs, the maximum page dimensions are 40 inches and 9,000 points.</p>



<p class="wp-block-paragraph">Two more constraints catch people out. Text detection covers English, French, German, Italian, Portuguese, and Spanish, and several capabilities are narrower still: handwriting, invoices and receipts, identity documents, and Queries are English only. Vertical text, common in Japanese and Chinese documents, is not supported at all. Rotation is fine, including heavy in-plane rotation. Vertical writing direction is not.</p>



<p class="wp-block-paragraph">Build an ingest gate that checks encryption, page count, byte size, and dimensions before it spends an API call, and give each rejection a reason code a human can act on. Then catch exceptions properly instead of wrapping the call in a bare retry. <code>BadDocumentException</code> means Textract could not read the file, and retrying fails identically. <code>DocumentTooLargeException</code> means route to the asynchronous path or split the file. <code>InvalidS3ObjectException</code> is usually IAM or a bucket-region mismatch, not a document problem. Only <code>InternalServerErrorException</code> deserves a retry with backoff.</p>



<h2 class="wp-block-heading">Failure family two: synchronous versus asynchronous is a correctness gate</h2>



<p class="wp-block-paragraph">This is the one that silently eats data, so it goes early.</p>



<p class="wp-block-paragraph">The synchronous operations, <code>DetectDocumentText</code> and <code>AnalyzeDocument</code>, cap at 10 MB in memory. For PDF and TIFF input they also cap at <strong>one page</strong>. Not one page per call with automatic paging. One page, full stop. Hand a synchronous call a 40-page PDF and you get page one, and a pipeline that reports success.</p>



<p class="wp-block-paragraph">The asynchronous operations are a different shape. PDF and TIFF go up to 500 MB and 3,000 pages, while JPEG and PNG stay at 10 MB. You start a job, Textract works in the background, and you collect results by job ID.</p>



<p class="wp-block-paragraph">A synchronous call looks like this. The document lives in S3, and you name the features you want:</p>



<pre class="wp-block-code"><code>aws textract analyze-document 
  --document '{"S3Object":{"Bucket":"my-doc-bucket","Name":"inbound/note-0042.png"}}' 
  --feature-types '["FORMS","TABLES"]' 
  --region us-east-1</code></pre>



<p class="wp-block-paragraph">The asynchronous version returns a job identifier immediately and publishes completion to an SNS topic. That is what you want in production, because polling a job inside a Lambda function means paying for a function that sleeps:</p>



<pre class="wp-block-code"><code>import boto3

textract = boto3.client("textract")

start = textract.start_document_analysis(
    DocumentLocation={
        "S3Object": {"Bucket": "my-doc-bucket", "Name": "inbound/batch-0042.pdf"}
    },
    FeatureTypes=["FORMS", "TABLES"],
    NotificationChannel={
        "SNSTopicArn": "arn:aws:sns:us-east-1:111122223333:textract-jobs",
        "RoleArn": "arn:aws:iam::111122223333:role/TextractSnsPublishRole",
    },
)

job_id = start["JobId"]</code></pre>



<p class="wp-block-paragraph">Results come back paginated, and forgetting the continuation token is a second, quieter way to lose most of a file:</p>



<pre class="wp-block-code"><code>blocks = []
next_token = None

while True:
    kwargs = {"JobId": job_id}
    if next_token:
        kwargs["NextToken"] = next_token

    page = textract.get_document_analysis(**kwargs)
    blocks.extend(page["Blocks"])

    next_token = page.get("NextToken")
    if not next_token:
        break</code></pre>



<p class="wp-block-paragraph">Check <code>JobStatus</code> before you trust <code>Blocks</code>. A job can come back partially successful, which means some pages processed and some did not. If you treat that as success you will publish a document that is missing its middle.</p>



<h2 class="wp-block-heading">Failure family three: the response is a graph, not a record</h2>



<p class="wp-block-paragraph">Textract does not return fields. It returns a flat list of <code>Block</code> objects, each with an ID and relationships pointing at other IDs. Reassembling those into a row is your code, and it is where most of the real work lives.</p>



<p class="wp-block-paragraph">For form data, the shape is specific. A key-value pair is two blocks of type <code>KEY_VALUE_SET</code>, distinguished by their <code>EntityTypes</code> array containing either <code>KEY</code> or <code>VALUE</code>. The key block carries a relationship of type <code>VALUE</code> holding the ID of its value block, and a relationship of type <code>CHILD</code> holding the IDs of the <code>WORD</code> blocks that spell out the label. The value block carries only <code>CHILD</code> relationships pointing at its own words. So reading one field means three hops.</p>



<pre class="wp-block-code"><code>blocks_by_id = {b["Id"]: b for b in blocks}

def child_text(block):
    parts = []
    for rel in block.get("Relationships", []):
        if rel["Type"] != "CHILD":
            continue
        for cid in rel["Ids"]:
            child = blocks_by_id[cid]
            if child["BlockType"] == "WORD":
                parts.append(child["Text"])
            elif child["BlockType"] == "SELECTION_ELEMENT":
                parts.append(child["SelectionStatus"])
    return " ".join(parts)

fields = {}
for b in blocks:
    if b["BlockType"] != "KEY_VALUE_SET":
        continue
    if "KEY" not in b.get("EntityTypes", []):
        continue

    value_text = ""
    for rel in b.get("Relationships", []):
        if rel["Type"] == "VALUE":
            for vid in rel["Ids"]:
                value_text = child_text(blocks_by_id[vid])

    fields[child_text(b)] = (value_text, b["Confidence"])</code></pre>



<p class="wp-block-paragraph">Two things about that snippet. The <code>SELECTION_ELEMENT</code> branch matters because checkboxes and radio buttons are values too, and their content is a selection status rather than text. And the confidence on the key block is not a separate key confidence: Textract returns the same value for both halves of a pair. If you planned to compare label confidence against value confidence to catch mismatches, that signal does not exist.</p>



<p class="wp-block-paragraph">Tables follow the same pattern with different block types, and cells carry row and column indices plus span values for merged cells. Layout adds another set, including title, header, footer, section header, page number, list, and figure blocks, and it sequences text into human reading order. That last property is why Layout is worth requesting when the output is headed for a language model rather than a database: chunking on reading order beats chunking on raw line order.</p>



<p class="wp-block-paragraph">Before hand-rolling any of this, look at the AWS-maintained parsers. The <code>amazon-textract-response-parser</code> package and the higher-level <code>amazon-textract-textractor</code> library both turn the block graph into objects with sensible properties. Writing your own traversal is a good way to understand the model and a poor way to spend two sprints.</p>



<h2 class="wp-block-heading">Getting Amazon Textract structured data out of the right API</h2>



<p class="wp-block-paragraph">Textract is a family of operations, not one endpoint, and the biggest single accuracy win available is usually picking a narrower one. Features are billed per page and they stack, so requesting <code>TABLES</code> on a corpus with no tables costs money and returns noise.</p>



<ul class="wp-block-list">
<li><strong>DetectDocumentText.</strong> Raw text and geometry. Use it when you only need a search index or a text layer.</li>

<li><strong>AnalyzeDocument with FORMS.</strong> Key-value pairs discovered from labels on the page. Works when the label is physically present and stable.</li>

<li><strong>AnalyzeDocument with TABLES.</strong> Grid reconstruction with row and column indices. The right call for line-item grids and statements.</li>

<li><strong>AnalyzeDocument with QUERIES.</strong> You ask natural language questions and get answers back. This is the fix for layout drift.</li>

<li><strong>AnalyzeDocument with LAYOUT and SIGNATURES.</strong> Reading order and structural elements; signature locations for compliance checks.</li>

<li><strong>AnalyzeExpense.</strong> Purpose-built for invoices and receipts, with normalized field types.</li>

<li><strong>AnalyzeID.</strong> Purpose-built for identity documents.</li>
</ul>



<h3 class="wp-block-heading">Why Queries beats FORMS on documents you don&#8217;t control</h3>



<p class="wp-block-paragraph">FORMS gives you whatever labels the document happens to contain. That means your downstream schema is hostage to a vendor&#8217;s wording. When they change &#8220;P.O. No.&#8221; to &#8220;Order Reference&#8221;, your parser looks for a key that no longer exists and writes a null.</p>



<p class="wp-block-paragraph">Queries inverts this. You supply the question and an alias, and the alias comes back attached to the answer, so your output key is fixed by you rather than by the document:</p>



<pre class="wp-block-code"><code>aws textract analyze-document 
  --document '{"S3Object":{"Bucket":"my-doc-bucket","Name":"inbound/note-0042.png"}}' 
  --feature-types '["QUERIES"]' 
  --queries-config '{"Queries":[
      {"Text":"What is the purchase order number?","Alias":"PO_NUMBER"},
      {"Text":"What is the delivery date?","Alias":"DELIVERY_DATE"}
  ]}' 
  --region us-east-1</code></pre>



<p class="wp-block-paragraph">The alias is the entire point. It is a contract between your schema and the extraction layer that survives a vendor redesign. There are ceilings: 15 queries per page synchronously, 30 per page asynchronously, English only. Phrase questions using words that actually appear on the page, and add positional hints when you know the layout, because the model uses them. If Queries underperforms on your corpus, adapters let you tune the feature against your own examples and pass an adapter ID and version at call time.</p>



<h3 class="wp-block-heading">Invoices are their own problem</h3>



<p class="wp-block-paragraph">For invoices and receipts, reach for AnalyzeExpense first. Its response is shaped around the domain rather than around the page: each <code>ExpenseDocument</code> holds <code>SummaryFields</code> for header-level data and <code>LineItemGroups</code> containing the line items. Fields carry a normalized <code>Type</code> such as <code>VENDOR_NAME</code>, plus a <code>ValueDetection</code> and an optional <code>LabelDetection</code>.</p>



<p class="wp-block-paragraph">That optional label is the thing to notice. AnalyzeExpense can identify a vendor name printed only inside a logo, and pull line items out of a grid with no column headers. FORMS structurally cannot do either, because FORMS needs a key to exist. On receipts that is not a small accuracy delta, it is the difference between a usable field and an empty one.</p>



<h2 class="wp-block-heading">Failure family four: confidence is per field and it is not your risk model</h2>



<p class="wp-block-paragraph">Every extracted element carries a confidence score. Two mistakes follow from that.</p>



<p class="wp-block-paragraph">The first is averaging into a document-level score. A document where every field is strong except the invoice total is not a mostly-good document. It is a document with one field that must not be trusted, surrounded by noise. Score fields, not documents.</p>



<p class="wp-block-paragraph">The second is treating one threshold as universal. A misread address on a delivery note is an annoyance. A misread bank account number is an incident. Set thresholds per field from the cost of being wrong, then route into three lanes:</p>



<ol class="wp-block-list">
<li><strong>Auto-accept.</strong> Above threshold and passing a format check. Straight into the target table.</li>

<li><strong>Review.</strong> Below threshold, or above it but failing a validation rule. A human sees the value and the cropped region it came from.</li>

<li><strong>Reject.</strong> Field absent entirely, or the document failed the ingest gate. Back to the sender with a reason.</li>
</ol>



<p class="wp-block-paragraph">Validation rules do more work than thresholds here. A total that does not equal the sum of its line items is wrong at any confidence. Check-digit rules on account and tax identifiers catch character-level errors the model was perfectly confident about. Confidence tells you how sure the model is. Validation tells you whether the answer is even possible.</p>



<p class="wp-block-paragraph">Textract can hand low-confidence results to Amazon Augmented AI through a human loop configuration on the API call, which saves building a review queue. Whether that beats a small internal tool depends on how much domain context your reviewers need on screen. Either way, capture the corrections. Reviewer edits are the only labeled data you get for free, and they are how you calibrate thresholds against your own corpus instead of guessing.</p>



<h2 class="wp-block-heading">Failure family five: reprocessing you didn&#8217;t budget for</h2>



<p class="wp-block-paragraph">Textract bills per page, per feature. Your parser, meanwhile, will change repeatedly in the first months of a pipeline&#8217;s life, because that is what parsers do.</p>



<p class="wp-block-paragraph">So separate extraction from parsing, permanently. Write the raw response JSON to S3 keyed by a hash of the document bytes plus the operation and feature set requested, then parse from that stored artifact. Reprocessing becomes free, deterministic, and diffable: you can replay a parser change across the whole back catalogue and compare outputs before promoting it. Without that cache, every parser bug fix is a bill.</p>



<pre class="wp-block-code"><code>import hashlib

digest = hashlib.sha256(doc_bytes).hexdigest()
features = "-".join(sorted(["FORMS", "TABLES"]))
key = f"textract-raw/{digest}/analyze-document/{features}.json"</code></pre>



<p class="wp-block-paragraph">Hashing content rather than filename also deduplicates the same scan arriving twice under different names, which happens constantly with email-driven intake. For the asynchronous operations, a client request token gives idempotency at the job level; reusing a token with different parameters raises a mismatch error rather than quietly starting a second job.</p>



<p class="wp-block-paragraph">One caveat: source documents often contain personal data, so a cache is a retention decision. Set S3 lifecycle rules on both the originals and the raw JSON, and if working copies land on a workstation or a build box, delete them properly. Tools like O&amp;O SafeErase exist for exactly that gap between &#8220;moved to trash&#8221; and &#8220;actually gone&#8221;.</p>



<h2 class="wp-block-heading">Where Textract stops being the right tool</h2>



<p class="wp-block-paragraph">Being fair about this matters, because AWS now offers overlapping paths to the same outcome. Amazon Bedrock Data Automation is a managed document processing service that handles classification and extraction behind one API, priced per document rather than per page and per feature. AWS positions it as the default starting point for new builds, and for mixed document types with no dominant format that recommendation is sound. You write far less orchestration.</p>



<p class="wp-block-paragraph">Textract keeps the advantage where the corpus is high volume and standardized, where geometry and per-field confidence are needed for auditability, where regional availability constrains you, or where you want extraction to be deterministic and separately testable. Bounding boxes and confidence scores are not a nice-to-have in a regulated workflow, they are the evidence trail.</p>



<p class="wp-block-paragraph">The hybrid is common and sensible: Textract for extraction, a Bedrock model for reasoning over the extracted text. Just do not send a multimodal model a full-page image because OCR felt inconvenient. It costs considerably more per page and gives you no geometry.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Only the first page came through.</strong> A synchronous call on a multi-page PDF or TIFF. Move to the asynchronous operations, and assert the page count Textract reports against the count you expected.</li>

<li><strong>Output truncates partway through a long document.</strong> Missing continuation token handling on the get operation. Loop until the token is absent.</li>

<li><strong>Fields visible on the page never appear in output.</strong> The label is implied rather than printed, or sits inside a logo. Switch that field to Queries, or to AnalyzeExpense if it is an invoice.</li>

<li><strong>Quality collapsed for one sender only.</strong> Template change. Diff the current layout against a stored sample, which is the argument for keeping raw responses.</li>

<li><strong>Throttling during a backfill.</strong> Transactions per second and concurrent job counts are per-account, per-region quotas. Queue submissions rather than fanning out, and request an increase before the migration.</li>

<li><strong>Table cells land in the wrong columns.</strong> Usually a skewed scan. Deskew and raise contrast before submitting, because image preparation is cheaper than post-processing repairs.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Testing on three clean sample documents supplied by the vendor rather than a random sample of real intake.</li>

<li>Requesting every feature type on every page because it is simpler than routing, then being surprised by the bill.</li>

<li>Using FORMS for invoices when AnalyzeExpense exists and understands unlabeled fields.</li>

<li>Treating a confidence score as an accuracy guarantee instead of pairing it with validation rules.</li>

<li>Discarding the raw JSON after parsing, which turns every parser fix into a re-extraction charge.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Classify before you extract. Route each document to the narrowest API that covers it.</li>

<li>Use Queries with explicit aliases for fields your schema depends on, so vendor wording changes cannot break your column names.</li>

<li>Store raw responses immutably, keyed by content hash, and parse from storage.</li>

<li>Set per-field confidence thresholds from the cost of being wrong, and pair every threshold with a format or arithmetic validation.</li>

<li>Instrument it like any other pipeline: extraction latency, straight-through rate, review queue depth, per-field null rate. A slow rise in nulls on one field is the earliest signal of a template change. Prometheus with Grafana, or Grafana Cloud if you would rather not run it yourself, covers this well.</li>

<li>Keep documents and pipeline in the same region, use VPC endpoints for sensitive data, and apply lifecycle rules to originals and extracted output alike.</li>

<li>Keep a golden set of real documents with known correct answers, and run it on every parser change.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can Amazon Textract process multi-page PDFs?</h3>



<p class="wp-block-paragraph">Yes, but only through the asynchronous operations. Synchronous calls accept a single page for PDF and TIFF input. Asynchronous processing handles PDF and TIFF up to 500 MB and 3,000 pages.</p>



<h3 class="wp-block-heading">What is the difference between FORMS and Queries?</h3>



<p class="wp-block-paragraph">FORMS returns whatever key-value pairs it finds from labels printed on the page, so your output keys follow the document. Queries lets you ask for values by natural language question and attach your own alias to each answer, so output keys stay fixed when the layout changes. For a schema you must keep stable, Queries is the safer contract.</p>



<h3 class="wp-block-heading">How much does Amazon Textract cost?</h3>



<p class="wp-block-paragraph">Billing is per page and varies by operation and feature, and requesting several features on one page adds their costs together. Check the current rate card for your region and multiply by pages, not documents. The practical lever is routing: send each document to the narrowest operation that answers your question, and cache raw responses so parser iteration is free.</p>



<h3 class="wp-block-heading">Does Textract handle handwriting?</h3>



<p class="wp-block-paragraph">It detects handwritten text, in English only. Treat handwritten fields as a separate confidence tier with lower auto-accept thresholds, because the variance across writers is much wider than for printed text.</p>



<h3 class="wp-block-heading">Which languages does Amazon Textract support?</h3>



<p class="wp-block-paragraph">Text detection covers English, French, German, Italian, Portuguese, and Spanish. Handwriting, invoices and receipts, identity documents, and Queries are English only. Vertically written text is not supported.</p>



<h3 class="wp-block-heading">Should I use Textract or Bedrock Data Automation?</h3>



<p class="wp-block-paragraph">Bedrock Data Automation is the lower-effort default for mixed document types, with classification and extraction behind a single managed API. Textract fits better at high volume on standardized documents, where per-field confidence and geometry matter for audit, or where extraction needs to be deterministic and independently testable. Combining them is normal.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">If one thing survives this post, make it this: Amazon Textract structured data extraction fails quietly, not loudly. The API returns success while your record is missing its most important field, because a synchronous call read one page of forty, or because a vendor renamed a label, or because a value came back at low confidence and nothing was watching.</p>



<p class="wp-block-paragraph">So build for the silence. Gate at ingest, use asynchronous operations by default, pick the narrowest API per document family, pin your schema with query aliases, store the raw response, and route on per-field confidence backed by validation rules that know what a possible answer looks like. Do that and the OCR really does become the easy part.</p>



<h2 class="wp-block-heading">Need help turning documents into a pipeline you can trust?</h2>



<p class="wp-block-paragraph">Most of the difficulty in document extraction shows up after the proof of concept, once real intake replaces the sample files. That is the part I work on:</p>



<ul class="wp-block-list">
<li>Designing the ingest gate and classification step so each document family reaches the right Textract operation instead of one call doing everything.</li>

<li>Building the asynchronous path properly: S3 events, SNS completion, paginated retrieval, dead-letter handling, idempotent reprocessing.</li>

<li>Replacing brittle FORMS parsing with query aliases or AnalyzeExpense, and validating the change against a golden set of your own documents.</li>

<li>Setting per-field confidence thresholds and validation rules, then wiring a review queue whose corrections feed back into tuning.</li>

<li>Adding observability that catches template drift early: per-field null rates, straight-through rate, and review queue depth on a dashboard.</li>

<li>Getting the cost curve under control through routing, raw-response caching, and lifecycle rules on stored documents and JSON.</li>
</ul>



<p class="wp-block-paragraph">If you have a pipeline misbehaving, send me a sample document with the fields you need and the raw Textract JSON it produced. That is usually enough to see where it is going wrong.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/amazon-textract-structured-data/">Turning External Documents Into Structured Data With Amazon Textract</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>ALB vs API Gateway: Pick the Limits You Can Live With</title>
		<link>https://john-nessime.com/blog/devops/alb-vs-api-gateway/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 25 Aug 2026 09:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Networking]]></category>
		<category><![CDATA[API Gateway]]></category>
		<category><![CDATA[Application Load Balancer]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[JWKS]]></category>
		<category><![CDATA[JWT Validation]]></category>
		<category><![CDATA[Listener Rules]]></category>
		<category><![CDATA[Load Balancing]]></category>
		<category><![CDATA[OIDC]]></category>
		<category><![CDATA[Private Integration]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Web Application Firewall]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=550</guid>

					<description><![CDATA[<p>Choosing between an Application Load Balancer and API Gateway is not a feature comparison. It is a choice of hard limits, a billing curve and an authentication story. Here is how each one bills you, which constraints surface months later, what ALB's native JWT validation changes, and a decision procedure you can run in ten minutes.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/alb-vs-api-gateway/">ALB vs API Gateway: Pick the Limits You Can Live With</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said the reporting endpoint returns a 504 after about thirty seconds. The application log said the job finished in forty-one seconds and wrote a perfectly good response to a client that had already hung up. Nothing crashed, nothing scaled wrong, nobody deployed anything that week.</p>



<p class="wp-block-paragraph">The integration timeout had been sitting in front of that API since the day someone created it. It only became a bug when a customer&#8217;s dataset got big enough to cross it.</p>



<p class="wp-block-paragraph">That is the real shape of the ALB vs API Gateway decision. You&#8217;re not picking a feature list. You&#8217;re picking a set of hard limits, a billing curve and an authentication story, then living inside them. The failure mode that bites isn&#8217;t a bad Friday, it&#8217;s finding the constraint months later, once the thing in front of your app has become load-bearing.</p>



<p class="wp-block-paragraph">Below: how each one bills you, which limits surface late, what changed recently on the load balancer side, and a decision procedure.</p>



<h2 class="wp-block-heading">The old rule of thumb is shakier than it looks</h2>



<p class="wp-block-paragraph">The advice you find everywhere is: containers and instances behind an Application Load Balancer, serverless behind API Gateway. Wrong often enough to cost you. ALB takes Lambda functions as targets. API Gateway reaches containers through private integrations with an NLB, an ALB, or Cloud Map. Compute type barely narrows the field.</p>



<p class="wp-block-paragraph">What separates them is who&#8217;s calling and what needs enforcing before a request reaches your code. A browser session against your own web app is a different problem from a partner&#8217;s server calling a documented endpoint with a key you issued and a quota you promised. One needs routing and a session. The other needs a contract.</p>



<h2 class="wp-block-heading">Application Load Balancer: routing you own the position of</h2>



<p class="wp-block-paragraph">ALB is a Layer 7 router inside your VPC. You give it subnets and security groups, attach listeners, and write rules that match on host, path, header, query string, method or source IP, then forward, redirect, return a fixed response, or authenticate.</p>



<h3 class="wp-block-heading">Where it wins</h3>



<ul class="wp-block-list">
<li><strong>Long requests.</strong> The idle timeout is an attribute you set, defaulting to 60 seconds. Report generation and slow third-party calls become a number in Terraform instead of an architecture problem.</li>

<li><strong>Big bodies and streaming.</strong> Against instance, IP and container targets there&#8217;s no small payload ceiling to design around, and WebSocket upgrades pass through.</li>

<li><strong>Browser authentication.</strong> An HTTPS listener rule can run an <code>authenticate-oidc</code> or <code>authenticate-cognito</code> action before forwarding. The load balancer does the redirect dance, sets a session cookie, and hands your target the verified claims in a header. Your app reads a header instead of implementing OAuth.</li>
</ul>



<h3 class="wp-block-heading">Where it doesn&#8217;t</h3>



<ul class="wp-block-list">
<li><strong>No per-client rate limiting.</strong> No API keys with quotas attached. Rate limiting means AWS WAF rate-based rules in front, counting requests per source over a window. Blunter than a per-consumer plan, and a separate bill.</li>

<li><strong>No request validation or response caching.</strong> A malformed body reaches your handler and burns your compute. A cacheable GET is recomputed unless you put a CDN in front.</li>

<li><strong>Lambda targets are constrained.</strong> Request body capped at 1 MB, response JSON capped at 1 MB, WebSocket upgrades rejected with a 400. If your function returns a large document, that ceiling arrives as a 502 in production, not an error at deploy time.</li>

<li><strong>You pay for it while it sleeps.</strong> An hourly charge per load balancer whether or not a request arrives. Three environments, three floors.</li>
</ul>



<h2 class="wp-block-heading">Amazon API Gateway: the contract layer</h2>



<p class="wp-block-paragraph">API Gateway isn&#8217;t a load balancer with extra features. It&#8217;s a place to define what your API promises: which routes exist, who may call them, how often, and in what shape. That framing explains most of its trade-offs.</p>



<p class="wp-block-paragraph">Choosing it is really two decisions, because REST APIs and HTTP APIs are different products under one name.</p>



<ul class="wp-block-list">
<li><strong>REST APIs</strong> carry the full management surface: API keys, usage plans, per-client throttling, request validation, response caching, AWS WAF integration, the private endpoint type, execution logs and X-Ray tracing.</li>

<li><strong>HTTP APIs</strong> are the stripped-down, cheaper option. They add a built-in JWT authorizer, automatic deployments and Cloud Map private integrations. They offer none of the six management features listed above.</li>
</ul>



<p class="wp-block-paragraph">That second line is the one people get wrong. HTTP APIs aren&#8217;t &#8220;REST APIs but cheaper&#8221;, they&#8217;re a different product that happens to serve HTTP. Pick them to save money, then discover you need metered access, and you&#8217;re rebuilding usage plans in your own code.</p>



<h3 class="wp-block-heading">Where it wins</h3>



<ul class="wp-block-list">
<li><strong>You&#8217;re selling or governing access.</strong> API keys tied to usage plans, on REST APIs, are the only native answer to &#8220;customer B must not burn customer A&#8217;s capacity&#8221;.</li>

<li><strong>Rejecting bad requests before they cost you.</strong> Request validation against a model means a malformed payload never invokes your function. On per-invocation compute that&#8217;s a security boundary and a cost lever at once.</li>

<li><strong>Zero idle cost.</strong> No hourly floor, so a dev stage nobody calls is free. For genuinely spiky traffic this is often the whole argument, and there&#8217;s no network to design either.</li>
</ul>



<h3 class="wp-block-heading">Where it doesn&#8217;t</h3>



<ul class="wp-block-list">
<li><strong>The integration timeout.</strong> The default is 29 seconds. It can be raised, but AWS notes that going beyond 29 seconds may require reducing your Region-level throttle quota. You&#8217;re buying request duration with request rate.</li>

<li><strong>Payload ceilings.</strong> There&#8217;s a documented maximum payload size, well below what a file upload service wants. Large objects want presigned S3 URLs either way, but here the gateway forces that conversation early.</li>

<li><strong>Cost scales with request count.</strong> Per request plus data transfer. A health check hammered by a monitoring tool shows up directly on the line item.</li>

<li><strong>The account throttle is shared.</strong> The default steady-state rate and burst quota applies per Region across your APIs in that account. Raisable, but a shared ceiling by default rather than a per-API one.</li>
</ul>



<h2 class="wp-block-heading">What changed: the load balancer validates JWTs itself now</h2>



<p class="wp-block-paragraph">For years the clean argument for API Gateway on machine-to-machine APIs was token validation. If a partner&#8217;s service calls you with a bearer token, someone has to check signature, issuer and expiry, and you&#8217;d rather that not be a library duplicated in every service.</p>



<p class="wp-block-paragraph">ALB does this natively now. A rule on an HTTPS listener can carry a <code>jwt-validation</code> action that runs before any routing action. The load balancer fetches signing keys from a JWKS endpoint you configure, checks the signature, and requires <code>iss</code> and <code>exp</code>. It also validates <code>nbf</code> and <code>iat</code> when present, and you can configure up to ten additional claims. Valid tokens forward unchanged. Invalid ones never reach your VPC.</p>



<p class="wp-block-paragraph">This isn&#8217;t <code>authenticate-oidc</code>, which is built for humans in browsers and works by redirecting. JWT validation does no redirect. It inspects the token and returns a yes or no, which is what service-to-service traffic needs. Validation action first, routing action second, and order matters:</p>



<pre class="wp-block-code"><code>aws elbv2 create-rule 
  --listener-arn "$LISTENER_ARN" 
  --priority 10 
  --conditions Field=path-pattern,Values="/api/*" 
  --actions '[
    {
      "Type": "jwt-validation",
      "JwtValidationConfig": {
        "JwksEndpoint": "https://issuer.example.com/.well-known/jwks.json",
        "Issuer": "https://issuer.example.com"
      },
      "Order": 1
    },
    {
      "Type": "forward",
      "TargetGroupArn": "$TARGET_GROUP_ARN",
      "Order": 2
    }
  ]'</code></pre>



<p class="wp-block-paragraph">Two constraints first. Only RS256 is supported, and the JWKS endpoint has a maximum response size and key count. A provider publishing a large key set fails that check, and the failure mode is that requests stop reaching your targets. Fetch your provider&#8217;s JWKS document and look at it before committing.</p>



<p class="wp-block-paragraph">So &#8220;we need token validation at the edge&#8221; is no longer by itself a reason to reach for API Gateway. What the gateway still owns is everything downstream of that yes or no: keys, quotas, per-consumer throttles, shape validation, caching.</p>



<h2 class="wp-block-heading">How each one actually bills you</h2>



<p class="wp-block-paragraph">This is where the argument usually goes wrong, because people compare unit prices instead of curve shapes.</p>



<p class="wp-block-paragraph">API Gateway is a slope with no floor: per request, plus data out. Zero traffic costs nothing, ten times the traffic costs roughly ten times as much.</p>



<p class="wp-block-paragraph">ALB is a floor plus a stranger slope. An hourly rate per load balancer, plus capacity units measured across four dimensions at once:</p>



<ul class="wp-block-list">
<li>New connections per second</li>

<li>Active connections per minute</li>

<li>Processed bytes per hour</li>

<li>Rule evaluations per second, where the first ten processed rules per request are free</li>
</ul>



<p class="wp-block-paragraph">You&#8217;re charged on the highest of the four in a given hour, not the sum. That mechanic explains most surprises. Clients that open a fresh connection per call can bill you on new connections while your bandwidth graph looks flat. A listener with dozens of path rules can bill you on rule evaluations while everything else idles.</p>



<p class="wp-block-paragraph">Two details worth knowing: Lambda targets get a smaller processed-bytes allowance per capacity unit than instance, container and IP targets, so identical traffic consumes more units. Enabling mutual TLS reduces the active-connections allowance per unit.</p>



<p class="wp-block-paragraph">Before modelling anything, look at what you already consume:</p>



<pre class="wp-block-code"><code>aws cloudwatch get-metric-statistics 
  --namespace AWS/ApplicationELB 
  --metric-name ConsumedLCUs 
  --dimensions Name=LoadBalancer,Value=app/my-alb/0123456789abcdef 
  --start-time "$(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ)" 
  --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" 
  --period 3600 
  --statistics Sum Maximum</code></pre>



<p class="wp-block-paragraph">Run that against a representative week and you have the real input. For API Gateway, the equivalent is request count per stage. Across several accounts, a cost visibility tool like Vantage or CloudZero attributes this per environment faster than Cost Explorer, though disciplined tagging gets you most of the way for free.</p>



<p class="wp-block-paragraph">Rough shape: low steady traffic favours API Gateway because you skip the floor. High-volume, byte-heavy, long-lived-connection traffic favours ALB because the per-request slope disappears. Chatty traffic with tiny payloads is where the curves cross, and that one is worth calculating.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">A decision procedure for ALB vs API Gateway</h2>



<p class="wp-block-paragraph">Run these in order, stop at the first clear answer.</p>



<ol class="wp-block-list">
<li><strong>Does any request legitimately run past about thirty seconds?</strong> If yes and you can&#8217;t make it asynchronous, ALB. Raising the gateway timeout is possible but trades against your regional throttle quota.</li>

<li><strong>Do you need per-consumer quotas or metered access?</strong> If yes, API Gateway REST APIs. Nothing else here gives you keys and usage plans natively.</li>

<li><strong>Do requests or responses carry large payloads?</strong> If yes, ALB with instance or container targets. Both options constrain size, but the Lambda-target and gateway ceilings arrive sooner.</li>

<li><strong>Is traffic occasional, spiky, or mostly zero?</strong> If yes, API Gateway. An hourly floor for a few thousand requests a day is real cost with no matching benefit.</li>

<li><strong>Is the caller a browser session against your own app?</strong> If yes, ALB, with an authenticate action if you need login. The gateway&#8217;s management features are dead weight here.</li>

<li><strong>Is the caller another service with a bearer token and nothing more?</strong> Either works now. Decide on cost curve and payload shape, because JWT validation stopped being the tiebreaker.</li>
</ol>



<p class="wp-block-paragraph">Still torn? Choose the limits you&#8217;d rather explain to a customer in a year. Usually ALB for applications, API Gateway for products. Whichever you pick, alarm on the constraint rather than the symptom: integration latency approaching the timeout, or consumed capacity units trending up. Grafana Cloud or plain CloudWatch alarms both do the job. Then send a payload just over your ceiling in staging and confirm what the client actually receives.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<h3 class="wp-block-heading">&#8220;It&#8217;s serverless, so it&#8217;s cheaper&#8221;</h3>



<p class="wp-block-paragraph">Per-request billing is cheaper at low volume and more expensive at high volume. Arithmetic, not philosophy. The crossover depends on your request size and connection reuse, which is why the capacity-unit metric matters more than anyone&#8217;s price table.</p>



<h3 class="wp-block-heading">&#8220;We&#8217;ll put API Gateway in front of it later&#8221;</h3>



<p class="wp-block-paragraph">You can, through a private integration, and it&#8217;s a legitimate pattern. But &#8220;later&#8221; means a second hop, a second bill, a second set of timeouts and a second place to look during an incident. Adding it deliberately is fine. Assuming it&#8217;s free is not.</p>



<h3 class="wp-block-heading">&#8220;ALB has no rate limiting, so it&#8217;s insecure&#8221;</h3>



<p class="wp-block-paragraph">ALB has no <em>native</em> rate limiting, but WAF rate-based rules in front of it are a real control, and a source-based limit is often closer to what you wanted than a per-key quota. Cloudflare in front of the load balancer is another common answer when you already want its caching and bot controls. What ALB genuinely lacks is the per-consumer contract, which matters when consumers are customers and doesn&#8217;t when they&#8217;re your own front end.</p>



<p class="wp-block-paragraph">Check the pairing before you rely on it, though: WAF associates with ALB and with API Gateway REST APIs, but not with HTTP APIs. And the amount of request body WAF reads is configurable for some resource types and fixed for the load balancer. Anything past that window isn&#8217;t scanned.</p>



<h3 class="wp-block-heading">&#8220;The front door has to be an AWS service&#8221;</h3>



<p class="wp-block-paragraph">For one service with predictable traffic, a plain VPS running Nginx or Caddy does the same job with far less surface. Providers like Contabo or InterServer are reasonable homes for a small internal API. It won&#8217;t scale into a multi-service platform and it moves patching and certificate renewal onto you, but &#8220;which AWS service&#8221; is sometimes the wrong question.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I use API Gateway and ALB together?</h3>



<p class="wp-block-paragraph">Yes. API Gateway supports private integrations with an Application Load Balancer, so the gateway handles keys, quotas and validation while the load balancer routes inside the VPC. Sound pattern for a public product API in front of an existing internal platform. It also doubles the hops, the bills and the places a request can die, so use it when you need both jobs done.</p>



<h3 class="wp-block-heading">Which is cheaper, ALB or API Gateway?</h3>



<p class="wp-block-paragraph">Neither, universally. API Gateway has no idle cost and charges per request, so it wins at low or intermittent volume. ALB charges an hourly floor plus capacity units billed on the highest of four dimensions, so it wins once the per-request slope dominates. Pull your consumed capacity units and request counts and do the arithmetic on your own traffic.</p>



<h3 class="wp-block-heading">Can ALB do rate limiting?</h3>



<p class="wp-block-paragraph">Not natively. You get it by associating AWS WAF with the load balancer and using rate-based rules, which limit by source over a time window. If you need per-key quotas, that&#8217;s an API Gateway REST API usage plan, or logic you write yourself.</p>



<h3 class="wp-block-heading">Does ALB support JWT authentication now?</h3>



<p class="wp-block-paragraph">Yes, through a <code>jwt-validation</code> action on an HTTPS listener rule. It verifies the signature against a JWKS endpoint and checks issuer and expiry, plus optional additional claims. Note the RS256-only restriction and the JWKS size and key count limits before designing around it.</p>



<h3 class="wp-block-heading">Should I use HTTP APIs or REST APIs?</h3>



<p class="wp-block-paragraph">REST APIs if you need any of: API keys, usage plans, per-client throttling, request validation, response caching, WAF integration, or a private endpoint type. HTTP APIs when you need none of those and want the built-in JWT authorizer at a lower price. Migrating between them isn&#8217;t a config flag, so decide before you publish a URL.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">ALB vs API Gateway isn&#8217;t a question about serverless versus containers, and it stopped being a question about token validation. It&#8217;s a question about whether the thing in front of your application needs to enforce a contract with people outside your organisation.</p>



<p class="wp-block-paragraph">If it does, take the gateway and accept its timeout and payload ceilings as the price of keys, quotas and validation. If it doesn&#8217;t, take the load balancer and spend the saved effort on the network and the health checks. Either way, write down the three numbers that will eventually bite: timeout, payload ceiling, throttle quota. The 504 in that opening ticket wasn&#8217;t a bug. It was a decision nobody had written down.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second opinion on your front door?</h2>



<p class="wp-block-paragraph">Cheap to get right at the start, expensive to revisit. If you&#8217;re staring at this now, or already living with the wrong answer:</p>



<ul class="wp-block-list">
<li>Modelling ALB capacity units against API Gateway request counts on your actual traffic, so the comparison uses your numbers</li>

<li>Auditing an existing gateway or load balancer for the limits you&#8217;re closest to hitting: timeouts, payload ceilings, throttle quotas</li>

<li>Moving browser authentication off your application onto an ALB authenticate rule, or setting up JWT validation at the listener for service-to-service traffic</li>

<li>Designing the gateway plus private integration pattern when you need both layers, including where to terminate TLS and how to preserve client IPs</li>

<li>Building the Terraform or OpenTofu modules for either option, with listener rules, target groups, health checks and alarms in one place</li>

<li>Setting up alarms that fire on approaching limits rather than the 5XX that follows</li>
</ul>



<p class="wp-block-paragraph">Send me a listener rule dump, a stage configuration, or a week of consumed capacity unit metrics and I&#8217;ll tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/alb-vs-api-gateway/">ALB vs API Gateway: Pick the Limits You Can Live With</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Amazon Textract Data Extraction: What Breaks on Real Contracts and Reports</title>
		<link>https://john-nessime.com/blog/devops/amazon-textract-data-extraction/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 20 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon SNS]]></category>
		<category><![CDATA[Amazon SQS]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Batch Processing]]></category>
		<category><![CDATA[Boto3]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Legal Tech]]></category>
		<category><![CDATA[OCR]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Serverless]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=270</guid>

					<description><![CDATA[<p>Textract rarely fails loudly. It returns a plausible result that is quietly incomplete: a truncated result set, a tick box read as an empty string, a clause split across a page break. A practitioner's guide to the failure modes that actually bite when you point Amazon Textract at contracts, technical reports and correspondence, plus how to choose between sync and async, which feature types are worth paying for, and where Textract stops being the right tool.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/amazon-textract-data-extraction/">Amazon Textract Data Extraction: What Breaks on Real Contracts and Reports</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The pipeline passed every test. Three sample agreements, all the right fields, clean JSON out the other side. Then it ran against the real archive and somebody in legal noticed that every contract longer than about forty pages was missing its termination clause. No errors. No failed jobs. The dashboard was green the whole time.</p>



<p class="wp-block-paragraph">The bug was a missing loop. The job had finished, the results were there, and the code had read the first chunk of them and stopped.</p>



<p class="wp-block-paragraph">That is the shape of most problems with Amazon Textract data extraction on real documents. The service rarely fails loudly. It returns a plausible-looking result that is quietly incomplete, and you find out weeks later when somebody asks a question the data cannot answer. This post covers the failures that actually bite on contracts, technical reports and correspondence: sync versus async, reading the block graph without losing fields, which feature types are worth paying for, and where Textract stops being the right tool.</p>



<h2 class="wp-block-heading">Why Amazon Textract data extraction is not just OCR</h2>



<p class="wp-block-paragraph">Plain OCR gives you words and their positions. Textract gives you a graph. Every response is a flat array of <code>Block</code> objects, and each block carries an ID plus relationships to other block IDs. A page relates to its lines, a line to its words, a key to its value, a table to its cells.</p>



<p class="wp-block-paragraph">That structure is the whole point, and it is where the pain lives. Nothing is nested for you. To read one form field you find the <code>KEY_VALUE_SET</code> block with entity type <code>KEY</code>, follow its <code>VALUE</code> relationship to another block, then follow that block&#8217;s <code>CHILD</code> relationships to the words. Three hops, and getting a hop wrong returns an empty string rather than an exception.</p>



<p class="wp-block-paragraph">Contracts and reports make this harder than invoices do. An invoice has a total. A master services agreement has a liability cap buried in a numbered sub-clause that spans a page break.</p>



<h2 class="wp-block-heading">Synchronous or asynchronous: the choice is made for you</h2>



<p class="wp-block-paragraph">People reach for <code>AnalyzeDocument</code> first because it returns results in the same call and is easy to test in a notebook. Then they hit the quotas, which AWS documents as hard limits you cannot raise. Synchronous operations cap JPEG, PNG, PDF and TIFF at 10 MB in memory, and cap PDF and TIFF at <strong>one page</strong>. Asynchronous operations keep JPEG and PNG at 10 MB but take PDF and TIFF up to 500 MB and 3,000 pages.</p>



<p class="wp-block-paragraph">One page. That single constraint decides your architecture. Any real contract goes through <code>StartDocumentAnalysis</code> and <code>GetDocumentAnalysis</code>, which means it goes through S3, which means you need somewhere to stage documents and a way to learn when the job is done.</p>



<p class="wp-block-paragraph">Three other constraints from the same quota page are worth knowing before you promise anything to a client:</p>



<ul class="wp-block-list">
<li>Text detection covers English, French, German, Italian, Portuguese and Spanish. Query detection is English only.</li>

<li>Vertical text is not supported. Rotation is fine, including odd in-plane angles, but vertically written scripts are not.</li>

<li>Password-protected PDFs are rejected and XFA-based PDFs are unsupported. Both turn up in legal archives more often than you would expect.</li>
</ul>



<p class="wp-block-paragraph">Pass a <code>ClientRequestToken</code> on submission. It is an idempotency token: the same token returns the same <code>JobId</code> instead of starting a duplicate job. If your submitter is a Lambda function behind an S3 event, and S3 events can be delivered more than once, that is the difference between paying once and paying twice.</p>



<pre class="wp-block-code"><code>import boto3

textract = boto3.client("textract")

job = textract.start_document_analysis(
    DocumentLocation={
        "S3Object": {"Bucket": "contracts-intake", "Name": "msa/acme-2.pdf"}
    },
    FeatureTypes=["FORMS", "TABLES"],
    ClientRequestToken="msa-acme-2-v1",
    NotificationChannel={
        "SNSTopicArn": "arn:aws:sns:eu-west-1:111122223333:textract-done",
        "RoleArn": "arn:aws:iam::111122223333:role/TextractSnsPublish",
    },
)</code></pre>



<p class="wp-block-paragraph">The notification channel is optional but you want it. Polling in a loop inside a Lambda function burns billed duration doing nothing, and Lambda&#8217;s execution ceiling will cut you off on long documents anyway. Publish to SNS, fan out to SQS, let a second function do the reading, and you get a natural place to hang a dead letter queue.</p>



<h2 class="wp-block-heading">The result set you never finished reading</h2>



<p class="wp-block-paragraph">This is the one that cost the termination clauses. <code>GetDocumentAnalysis</code> returns results in pages. When there are more blocks than fit in one response, the response carries a <code>NextToken</code> and you have to call again with it. If you do not, you get the beginning of the document and nothing tells you so. <code>JobStatus</code> still reports success, because the job did succeed. Your code just stopped reading.</p>



<p class="wp-block-paragraph">Short test documents fit in a single response. That is exactly why this survives testing and dies in production.</p>



<pre class="wp-block-code"><code>def fetch_all_blocks(textract, job_id):
    blocks, next_token = [], None

    while True:
        kwargs = {"JobId": job_id}
        if next_token:
            kwargs["NextToken"] = next_token

        response = textract.get_document_analysis(**kwargs)

        if response["JobStatus"] == "FAILED":
            raise RuntimeError(response.get("StatusMessage", "job failed"))

        blocks.extend(response["Blocks"])
        next_token = response.get("NextToken")

        if not next_token:
            return blocks</code></pre>



<p class="wp-block-paragraph">Then make the failure detectable. Treat the highest page number present in the blocks as a checksum against the page count of the source file. Submit a 60-page PDF, get blocks that stop at page 12, and something is wrong no matter what the job status says.</p>



<h2 class="wp-block-heading">Choosing feature types without paying for all of them</h2>



<p class="wp-block-paragraph"><code>AnalyzeDocument</code> and <code>StartDocumentAnalysis</code> take a <code>FeatureTypes</code> list: <code>FORMS</code>, <code>TABLES</code>, <code>QUERIES</code>, <code>SIGNATURES</code> and <code>LAYOUT</code>. This is not cosmetic. Textract bills per page and the per-page rate depends on which features you enabled, so turning all five on because you might need them later multiplies your bill across the whole archive. Rates change and vary by region, so check the current pricing page rather than trusting a number in a blog post. The mechanism does not change: each feature adds to the per-page cost, and you pay it on every page you submit, including blank separator sheets.</p>



<ul class="wp-block-list">
<li><strong>FORMS</strong> finds key-value pairs where the document has an explicit label. Good for cover sheets, signature blocks, report headers. Useless for prose clauses with no label.</li>

<li><strong>TABLES</strong> reconstructs rows, columns and cells, and also identifies table titles, footers and merged cells. This is what you want for rate cards and appendix tables.</li>

<li><strong>QUERIES</strong> lets you ask natural-language questions and get answers back under an alias you chose. This is the feature that makes contracts tractable.</li>

<li><strong>SIGNATURES</strong> returns the location and confidence of handwritten signatures, electronic signatures and initials. It tells you something is signed at a coordinate, not whose signature it is.</li>

<li><strong>LAYOUT</strong> groups text into titles, headers, footers, section headers, paragraphs and lists in reading order. Skippable on single-column documents. On two-column reports it is the difference between coherent text and interleaved nonsense.</li>
</ul>



<p class="wp-block-paragraph">A practical pattern: run a cheap text-detection pass to classify each document, then apply the expensive feature set only to the pages that matter. A 200-page appendix of scanned site photographs does not need FORMS and TABLES.</p>



<h2 class="wp-block-heading">Reading the block graph without losing fields</h2>



<p class="wp-block-paragraph">Here is the failure that produces silent empty strings. A key-value pair&#8217;s value is not always text. If the value is a tick box, the value block&#8217;s children include a <code>SELECTION_ELEMENT</code> block, and that block has no text at all. It has a <code>SelectionStatus</code> of <code>SELECTED</code> or <code>NOT_SELECTED</code>. Concatenate only <code>WORD</code> children and a ticked box comes back as an empty string, which looks exactly like a field nobody filled in.</p>



<pre class="wp-block-code"><code>def block_text(block, block_map):
    parts = []
    for rel in block.get("Relationships", []):
        if rel["Type"] != "CHILD":
            continue
        for child_id in rel["Ids"]:
            child = block_map[child_id]
            if child["BlockType"] == "WORD":
                parts.append(child["Text"])
            elif child["BlockType"] == "SELECTION_ELEMENT":
                parts.append(child["SelectionStatus"])
    return " ".join(parts)</code></pre>



<p class="wp-block-paragraph">Build <code>block_map</code> once as a dictionary keyed on block ID before you traverse anything. Scanning the block list linearly for each lookup turns a three-hop traversal into an accidental quadratic, which you will notice the first time a 3,000-page bundle goes through.</p>



<h2 class="wp-block-heading">Queries: asking for clauses that have no label</h2>



<p class="wp-block-paragraph">FORMS works when the document says &#8220;Effective Date:&#8221; next to the date. Contracts frequently do not. The governing law sits in a paragraph of prose halfway down page nine. Queries handles that. You pass questions in plain English, each with an <code>Alias</code> you choose, and the response pairs <code>QUERY</code> blocks with <code>QUERY_RESULT</code> blocks through an <code>ANSWER</code> relationship. The alias is what makes this usable downstream: you match on <code>governing_law</code>, not on the exact wording of the question, so you can reword questions without breaking your schema.</p>



<pre class="wp-block-code"><code>QueriesConfig = {
    "Queries": [
        {"Text": "What is the governing law?",
         "Alias": "governing_law", "Pages": ["*"]},
        {"Text": "What is the termination notice period?",
         "Alias": "termination_notice", "Pages": ["9-*"]},
    ]
}</code></pre>



<p class="wp-block-paragraph">The limits matter. AWS documents a maximum of 15 queries per page for synchronous operations and 30 per page for asynchronous ones. A fifty-field schema cannot be asked in one pass. You either split it across multiple calls, which multiplies per-page spend, or you narrow <code>Pages</code> so each query runs only where the answer plausibly lives. Signature blocks are at the end, definitions near the front. Running every query against every page is the most common way people accidentally triple their bill.</p>



<p class="wp-block-paragraph">Queries are pre-trained on a spread of business documents including paystubs, bank statements, loan applications and mortgage notes. Your niche contract template was not in that set. Expect good results on common concepts and mediocre ones on house-specific terminology.</p>



<h2 class="wp-block-heading">Documents that fight back</h2>



<h3 class="wp-block-heading">Clauses that span a page break</h3>



<p class="wp-block-paragraph">Textract analyses pages. A clause starting at the bottom of page 11 and finishing at the top of page 12 is two disconnected fragments as far as the block graph is concerned. Queries will often return the fragment on one page and miss the qualifier on the other, which is the worst outcome available because the answer looks complete. The fix is not clever: reassemble full text in reading order using LAYOUT, then run clause-level logic over that rather than over per-page answers. Use Queries to <em>locate</em> a clause and the reassembled text to <em>read</em> it.</p>



<h3 class="wp-block-heading">Two-column reports and email threads</h3>



<p class="wp-block-paragraph">Without LAYOUT, a two-column technical report reads as alternating lines from both columns. Everything downstream inherits that corruption and it passes a spot check, because individual lines look fine. LAYOUT sequences elements in reading order and returns block types for titles, headers, footers and section headers, which is also what you need to chunk a report by section.</p>



<p class="wp-block-paragraph">Printed email chains are quoted text inside quoted text, newest message at the top, the same signature block repeated five times. Textract extracts all of it faithfully, which is the problem. Splitting a thread into individual messages is a text-processing job you do afterwards, not something a feature type solves.</p>



<h3 class="wp-block-heading">PDFs that were never scanned</h3>



<p class="wp-block-paragraph">This is the cheapest win available and almost everybody misses it. A large share of contract archives are born-digital PDFs exported from a word processor, and they already contain a text layer. Running OCR over them pays a service to guess at text you could have read directly. Put a triage step in front of the pipeline; <code>pdftotext</code> from poppler-utils pulls the embedded layer if there is one:</p>



<pre class="wp-block-code"><code>pdftotext -layout contract.pdf - | head -c 2000</code></pre>



<p class="wp-block-paragraph">Substantial text back means born-digital: route it down a cheaper path and reserve Textract for genuinely scanned material. Triage across a large archive is an embarrassingly parallel batch job that sits better on a plain VPS from somewhere like Contabo or InterServer than on per-invocation serverless billing, since you are CPU-bound for minutes at a time rather than reacting to events. One caveat: born-digital does not mean clean. Some exporters produce a text layer with broken word spacing or mangled ligatures, so sample before you trust it.</p>



<h2 class="wp-block-heading">Confidence scores you can actually act on</h2>



<p class="wp-block-paragraph">Every block carries a confidence score and the instinct is to threshold on it. That is usually wrong. Confidence tells you how sure the model is about <em>which characters are on the page</em>, not whether the field is semantically right. Textract can read a date with total certainty and hand you the wrong one, because it picked up the printing date in the footer instead of the effective date in the recitals. High confidence, completely wrong answer.</p>



<p class="wp-block-paragraph">It is also per block. A key-value pair has a score on the key, another on the value and separate ones on each word, so there is no single number to gate on. Validate the extracted value instead. Does the date parse and fall in a plausible range? Does the total equal the sum of the line items? Does the party name appear in the signature block as well as the preamble? Route to human review on failed validation and treat low confidence as one input among several.</p>



<h2 class="wp-block-heading">When Custom Queries adapters are worth the effort</h2>



<p class="wp-block-paragraph">If pre-trained Queries keeps getting your house document type wrong, train an adapter. Create it with <code>CreateAdapter</code>, annotate sample documents, train a version with <code>CreateAdapterVersion</code>, then reference it at inference time.</p>



<pre class="wp-block-code"><code>AdaptersConfig = {
    "Adapters": [
        {"AdapterId": ADAPTER_ID, "Version": "1", "Pages": ["1-5"]},
        {"AdapterId": ADAPTER_ID, "Version": "1", "Pages": ["6-*"]},
    ]
}</code></pre>



<p class="wp-block-paragraph">That <code>Pages</code> string takes digits, hyphens and an asterisk with no blank spaces, a page can only have one adapter applied to it, and an asterisk meaning all pages must be the only element in the list.</p>



<p class="wp-block-paragraph">Two things to weigh. AWS sets a minimum of five samples per query level for training or testing and says plainly that more is better, so treat five as a demo floor rather than a number that survives layout variation. And successful trainings per month are capped per account, so the tight annotate-train-evaluate loop you want is not available. Adapters suit one high-volume, house-specific document type with a stable schema. They are a poor fit for a long tail of one-off templates, which is what most legal archives actually are.</p>



<h2 class="wp-block-heading">Where Textract stops and a language model starts</h2>



<p class="wp-block-paragraph">This is a real decision now, so both sides deserve a hearing. Textract&#8217;s case is strong on standardised, high-volume documents. It is deterministic in a way generative models are not, it returns bounding-box geometry so you can point at exactly where a value came from, it gives per-element confidence, and it bills predictably per page. When an auditor asks why a field has the value it does, geometry beats a model&#8217;s reasoning.</p>



<p class="wp-block-paragraph">The generative side, whether that is Amazon Bedrock Data Automation as a managed document processing service or a model called directly through Amazon Bedrock, wins where the task needs reasoning rather than pattern matching. &#8220;Does this agreement contain an assignment restriction, and if so summarise it&#8221; is a comprehension question, and Textract has no answer for it. Generative approaches also cope better with layouts that were in nobody&#8217;s training set.</p>



<p class="wp-block-paragraph">Most production pipelines use both, which is what AWS&#8217;s own reference architectures push: Textract for faithful extraction with geometry and confidence, then a model over the extracted text for classification and comprehension. Choosing from scratch, ask whether your documents are standardised and high-volume. If yes, Textract plus a thin post-processing layer is cheaper and more auditable. If they are varied and volume is modest, a managed generative service gets you working faster. Google Document AI and Azure AI Document Intelligence cover similar ground if you are not committed to AWS.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Extraction stops partway through long documents.</strong> You are not following <code>NextToken</code>. Compare the highest page number in your blocks against the source page count.</li>

<li><strong>Fields come back empty on forms you know were filled in.</strong> The value is a <code>SELECTION_ELEMENT</code> and you are only reading <code>WORD</code> children.</li>

<li><strong>The document is rejected as too large.</strong> You are on the synchronous API. Multi-page PDFs must go through <code>StartDocumentAnalysis</code>, and the 10 MB cap is on in-memory size, not file size on disk.</li>

<li><strong>Textract cannot read the document at all.</strong> Check for password protection and XFA-based forms. Both are documented as unsupported.</li>

<li><strong>The service cannot access the S3 object.</strong> Usually the execution role missing <code>s3:GetObject</code>, or a bucket in a different region from the Textract endpoint you called.</li>

<li><strong>Throttling under batch load.</strong> Transactions-per-second quotas are per account per region. Smooth spiky traffic through a queue and retry with exponential backoff and jitter rather than raising concurrency.</li>

<li><strong>Garbled text on reports that look fine to you.</strong> Multi-column layout without LAYOUT enabled, or a source scan below roughly 150 DPI. Re-scanning beats any amount of post-processing.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Testing only on short documents, which hides the pagination bug entirely.</li>

<li>Enabling every feature type on every page because it is easier than deciding.</li>

<li>Running OCR over born-digital PDFs that already carry a text layer.</li>

<li>Treating a confidence score as a correctness score.</li>

<li>Scanning the block array linearly instead of building an ID map.</li>

<li>Polling for job completion inside a Lambda function instead of using SNS.</li>

<li>Storing only extracted values and discarding the raw JSON, so reprocessing means paying again.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Persist the full Textract JSON to S3 keyed on a hash of the source document. Reprocessing then costs storage, not extraction.</li>

<li>Classify first, extract second. A cheap text-detection pass tells you which expensive feature set each document needs.</li>

<li>Validate on the value, not the score. Parse dates, check ranges, cross-reference names against other parts of the document.</li>

<li>Use aliases on every query and treat them as your schema contract.</li>

<li>Set a per-job page-count assertion and alert on mismatches. That is your canary for silent truncation.</li>

<li>Keep source documents in an encrypted bucket with a lifecycle policy. Contracts and correspondence should not accumulate indefinitely by accident.</li>

<li>Track cost per document rather than per API call. Feature stacking makes the per-call figure meaningless.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can Amazon Textract handle multi-page PDF contracts?</h3>



<p class="wp-block-paragraph">Yes, through the asynchronous API only. Synchronous operations cap PDF and TIFF at one page. Asynchronous operations accept PDF and TIFF up to 500 MB and 3,000 pages, staged through S3.</p>



<h3 class="wp-block-heading">Why is my Textract output missing the end of the document?</h3>



<p class="wp-block-paragraph">Almost certainly because you read the first response from <code>GetDocumentAnalysis</code> and stopped. Results are paginated with a <code>NextToken</code> and the job status still reports success, so nothing signals the truncation. Loop until <code>NextToken</code> is absent.</p>



<h3 class="wp-block-heading">How many Textract Queries can I ask per page?</h3>



<p class="wp-block-paragraph">AWS documents 15 per page for synchronous operations and 30 per page for asynchronous ones. Larger schemas need multiple passes, which costs more, so narrowing each query&#8217;s page range is worth the effort.</p>



<h3 class="wp-block-heading">Does Textract work on handwriting and signatures?</h3>



<p class="wp-block-paragraph">It detects handwriting, and the SIGNATURES feature returns the location and confidence of handwritten signatures, electronic signatures and initials. It does not verify identity: it tells you something was signed and where, not by whom.</p>



<h3 class="wp-block-heading">What languages does Amazon Textract support?</h3>



<p class="wp-block-paragraph">Text detection covers English, French, German, Italian, Portuguese and Spanish. Queries detection is English only, vertically written text is unsupported, and Textract does not return the detected language in its output.</p>



<h3 class="wp-block-heading">Should I use Textract or a generative model?</h3>



<p class="wp-block-paragraph">Textract when documents are standardised, volume is high, and you need auditable geometry and per-field confidence. A generative service when layouts vary widely, volume is modest, or the task needs comprehension rather than extraction. Most production pipelines use both.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Amazon Textract data extraction fails quietly far more often than it fails loudly. A truncated result set, a tick box read as an empty string, a clause split across a page break, a confidently extracted wrong date: none of these throw an exception and none show up on a dashboard.</p>



<p class="wp-block-paragraph">So build the assertions that make silence detectable. Compare expected page counts against extracted ones, validate values rather than trusting scores, and keep the raw JSON so you can prove what the service actually returned. Textract is genuinely good at reading documents. Your job is noticing when it did not read all of them.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a document extraction pipeline that does not lose fields?</h2>



<p class="wp-block-paragraph">I build and fix document processing pipelines on AWS. Typical work looks like this:</p>



<ul class="wp-block-list">
<li>Auditing an existing Textract pipeline for silent truncation, dropped selection elements and mis-scoped queries</li>

<li>Designing the async architecture end to end: S3 intake, SNS and SQS fan-out, Lambda or container workers, dead letter handling and retries</li>

<li>Cutting per-page cost by classifying first, triaging born-digital PDFs out of the OCR path and narrowing feature types per page range</li>

<li>Turning block-graph JSON into a clean schema in CSV, a relational database or a data lake, with validation rules and a review queue for exceptions</li>

<li>Building the hybrid path where Textract handles extraction and a model on Amazon Bedrock handles classification and comprehension</li>

<li>Adding the observability that makes quiet failures loud: page-count assertions, per-document cost tracking, confidence distribution monitoring</li>
</ul>



<p class="wp-block-paragraph">If you have a redacted sample document, a chunk of Textract JSON or an extraction coming back half empty, send it over and I will tell you what is going wrong with it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/amazon-textract-data-extraction/">Amazon Textract Data Extraction: What Breaks on Real Contracts and Reports</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building an Insurance Claims Processing Pipeline on AWS That Fails Loudly</title>
		<link>https://john-nessime.com/blog/cloud-computing/insurance-claims-processing-pipeline-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 20 Aug 2026 09:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Insurance Technology]]></category>
		<category><![CDATA[Workflow Automation]]></category>
		<category><![CDATA[Amazon SNS]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Bedrock Data Automation]]></category>
		<category><![CDATA[Claims Automation]]></category>
		<category><![CDATA[Confidence Scoring]]></category>
		<category><![CDATA[Data Validation]]></category>
		<category><![CDATA[Dead Letter Queue]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Event-Driven Architecture]]></category>
		<category><![CDATA[HIPAA]]></category>
		<category><![CDATA[Human In The Loop]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Insurance Claims]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Step Functions]]></category>
		<category><![CDATA[Straight-Through Processing]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=464</guid>

					<description><![CDATA[<p>Claims pipelines rarely crash. They succeed, emit clean JSON, and hand a wrong number to a payment system. Six failure families in an insurance claims processing pipeline on AWS, with the Textract, Bedrock Data Automation and Step Functions details that decide whether a bad extraction is visible or silent.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/insurance-claims-processing-pipeline-aws/">Building an Insurance Claims Processing Pipeline on AWS That Fails Loudly</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The worst ticket on a claims pipeline is never the one that says the pipeline is down. A stuck queue is loud. It pages somebody, somebody restarts something, and it gets fixed before lunch. The bad ticket arrives three weeks later from finance: a run of claims was auto-approved at amounts nobody can reconcile, and every single execution in the Step Functions console is green.</p>



<p class="wp-block-paragraph">That is the failure mode that defines this problem. An insurance claims processing pipeline on AWS almost never falls over in the way you designed it to fall over. It succeeds. It emits well-formed JSON. It hands a number to a payment system, and the number is wrong, and nothing in the pipeline had any reason to think otherwise.</p>



<p class="wp-block-paragraph">This post is organized by failure family rather than by service. I&#8217;ll walk through the six ways these pipelines go quietly wrong, what each one costs, and what the fix actually looks like in Textract, Bedrock, Step Functions and S3. There&#8217;s a troubleshooting section, the mistakes I see repeated, and an FAQ at the end.</p>



<h2 class="wp-block-heading">The shape most claims pipelines end up with</h2>



<p class="wp-block-paragraph">Before the failure families make sense, the skeleton. Almost every serverless claims pipeline lands on roughly the same set of stages, whatever the vendor deck calls them:</p>



<ol class="wp-block-list">
<li><strong>Intake.</strong> A document lands in S3 from a portal upload, an SFTP drop, or a mail scanning vendor. An S3 event or EventBridge rule starts an execution.</li>

<li><strong>Classification.</strong> Work out what the packet actually contains. A first notice of loss, a CMS-style claim form, a police report, an itemized bill, forty pages of photographs.</li>

<li><strong>Extraction.</strong> Pull the fields you need. Amazon Textract for OCR, forms and tables, or Amazon Bedrock Data Automation with a blueprint that names the fields directly.</li>

<li><strong>Validation.</strong> Check the extracted values against business rules, policy data, and each other.</li>

<li><strong>Routing.</strong> Straight-through processing, human review, or rejection with a reason.</li>

<li><strong>Persistence and audit.</strong> The claim record, the extraction artifacts, and enough evidence to explain a decision months later.</li>
</ol>



<p class="wp-block-paragraph">Nothing controversial there. AWS publishes an open-source GenAI IDP Accelerator that implements exactly this shape, with a Bedrock Data Automation mode and a Textract-plus-foundation-model pipeline mode, and it&#8217;s a reasonable place to start reading. The interesting part is not the boxes. It&#8217;s what happens between them.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: the field that was never there</h2>



<p class="wp-block-paragraph">This is the one that pays out the wrong number, and it is worth more attention than everything else in this post combined.</p>



<p class="wp-block-paragraph">Every extraction service gives you confidence scores. So the obvious design is a gate: if every field scores above some threshold, approve automatically; if anything falls below, send it to a human. That gate is sound reasoning applied to the wrong population.</p>



<p class="wp-block-paragraph">A confidence score only exists for a value that came back. When the extractor doesn&#8217;t find a field at all, there is no low score to catch, because there&#8217;s nothing to score. The gate iterates over four returned fields, finds all four above threshold, and reports a clean pass. The fifth field, the one that determines coordination of benefits or the deductible offset, is simply absent from the response. Downstream code treats absent as zero, or as null, or as &#8220;not applicable,&#8221; and the claim goes through.</p>



<p class="wp-block-paragraph">The fix is structural, not statistical. Validate <em>presence against a schema</em> before you validate confidence, and treat the two as separate gates with separate outcomes:</p>



<pre class="wp-block-code"><code># Two gates, not one. Presence first, then confidence.
# 'extracted' is the flattened field map from Textract Queries
# or a Bedrock Data Automation blueprint result.

REQUIRED = {
    "claim_number",
    "date_of_service",
    "billed_amount",
    "member_id",
    "secondary_payer_indicator",
}

def gate(extracted, scores, threshold=0.95):
    missing = REQUIRED - set(extracted)
    if missing:
        # Never silently default. This is a routing decision.
        return "HUMAN_REVIEW", {"reason": "missing_fields",
                                "fields": sorted(missing)}

    weak = [f for f in REQUIRED if scores.get(f, 0.0) &lt; threshold]
    if weak:
        return "HUMAN_REVIEW", {"reason": "low_confidence",
                                "fields": sorted(weak)}

    return "STRAIGHT_THROUGH", {}
</code></pre>



<p class="wp-block-paragraph">Two details matter here. The set difference is computed against a declared schema, not against whatever keys happen to be in the response, so an absent field becomes a first-class routing reason. And <code>scores.get(f, 0.0)</code> defaults to zero rather than to a passing value, so a field that arrives without a score fails closed.</p>



<p class="wp-block-paragraph">If you&#8217;re on Textract Queries, there&#8217;s a second reason to be explicit: Queries let you attach an alias to each question, which means your schema keys are yours rather than whatever label happened to be printed on the form. That&#8217;s the difference between &#8220;the field is missing&#8221; and &#8220;the field moved and we didn&#8217;t notice.&#8221;</p>



<h2 class="wp-block-heading">Failure family two: confidence scores that answer a different question</h2>



<p class="wp-block-paragraph">Assume you&#8217;ve fixed presence. The next trap is what the confidence number is measuring.</p>



<p class="wp-block-paragraph">OCR confidence is a statement about characters. It says the model is highly sure those pixels read <code>1,240.00</code>. It is not a statement that <code>1,240.00</code> is the billed amount rather than the allowed amount from the box directly above it, or the prior balance from a remittance summary that happened to be stapled into the same packet. Read it as a legibility score, because that&#8217;s closer to what it is.</p>



<p class="wp-block-paragraph">Bedrock Data Automation narrows this gap: blueprints define fields semantically, confidence scores come with bounding boxes, and the visual grounding lets you point at the region a value came from. Textract Queries narrow it too, by asking a question rather than harvesting a label. Neither eliminates the problem, because a high-confidence read of the wrong region still scores high.</p>



<p class="wp-block-paragraph">What actually catches this is cross-field invariants. They cost almost nothing and they fail for reasons a human can read:</p>



<ul class="wp-block-list">
<li><strong>Arithmetic.</strong> Line items sum to the claimed total. If they don&#8217;t, one of the two is wrong and you don&#8217;t yet know which.</li>

<li><strong>Temporal.</strong> Date of service falls inside the policy period and before the date of submission. A service date after the submission date is a parsing error nine times out of ten.</li>

<li><strong>Referential.</strong> The member or policy identifier resolves against your system of record. An identifier that matches the format but not a real record is a strong signal you read the wrong box.</li>

<li><strong>Range.</strong> Amounts within a plausible band for the claim type. Not a fraud model, just a tripwire for a decimal point that moved.</li>

<li><strong>Page provenance.</strong> Fields that must come from the same page or the same document within the packet. Bounding box data makes this checkable rather than assumed.</li>
</ul>



<p class="wp-block-paragraph">An invariant failure is more useful than a low score, because it names a contradiction. &#8220;Line items sum to 1,180 but the claimed total reads 1,240&#8221; is something a reviewer resolves in seconds. &#8220;Confidence 0.91&#8221; is something a reviewer stares at.</p>



<h2 class="wp-block-heading">Failure family three: the claim that stops halfway</h2>



<p class="wp-block-paragraph">Claims documents are multi-page packets, so you&#8217;ll be using Textract&#8217;s asynchronous operations. That means jobs, notifications, and a whole class of orchestration bugs that only show up under load or after a weekend.</p>



<p class="wp-block-paragraph">The asynchronous pattern is: call <code>StartDocumentAnalysis</code>, get a <code>JobId</code> back, and let Textract publish completion to an SNS topic you nominate. A request looks like this:</p>



<pre class="wp-block-code"><code>{
  "DocumentLocation": {
    "S3Object": { "Bucket": "claims-intake", "Name": "packets/abc123.pdf" }
  },
  "FeatureTypes": ["FORMS", "TABLES"],
  "ClientRequestToken": "abc123-v1",
  "JobTag": "fnol-packet",
  "NotificationChannel": {
    "SNSTopicArn": "arn:aws:sns:REGION:ACCOUNT:textract-complete",
    "RoleArn": "arn:aws:iam::ACCOUNT:role/TextractPublishRole"
  },
  "OutputConfig": {
    "S3Bucket": "claims-extraction",
    "S3Prefix": "raw/"
  },
  "KMSKeyId": "alias/claims-cmk"
}
</code></pre>



<p class="wp-block-paragraph">Three of those parameters are doing load-bearing work that is easy to skip.</p>



<p class="wp-block-paragraph"><code>ClientRequestToken</code> is the idempotency token. Reuse the same token and you get the same <code>JobId</code> back instead of a second job. Derive it from the document, not from the invocation, and a Lambda retry or a duplicated S3 event stops turning into a duplicate charge and a duplicate claim record.</p>



<p class="wp-block-paragraph"><code>OutputConfig</code> writes results into a bucket you control. Without it, results stay internal to Textract and the only way to read them is the <code>Get</code> operations, which have their own throttling limits. Under concurrency those limits become the bottleneck: you end up polling more jobs than you&#8217;re allowed to poll, backing off, and watching end-to-end latency climb for reasons that have nothing to do with the documents. Writing to S3 sidesteps the whole path.</p>



<p class="wp-block-paragraph"><code>JobTag</code> shows up in the completion notification. In a mixed pipeline where the same topic carries first notice of loss packets, itemized bills and ID documents, that tag is what lets the notification handler route without a lookup.</p>



<p class="wp-block-paragraph">One expiry to plan around: a Textract <code>JobId</code> is only valid for seven days. If your retry story is &#8220;requeue it and someone will look on Monday,&#8221; a bad weekend turns recoverable failures into full reprocessing. Persist the S3 output location, not the job identifier.</p>



<h3 class="wp-block-heading">Callbacks that never come back</h3>



<p class="wp-block-paragraph">Human review means pausing a workflow for hours or days, which in Step Functions means the callback pattern. You append <code>.waitForTaskToken</code> to the resource ARN, pass <code>$$.Task.Token</code> into the payload, and the execution parks until something calls <code>SendTaskSuccess</code> or <code>SendTaskFailure</code> with that token.</p>



<p class="wp-block-paragraph">The trap is that a callback task with no timeout waits until the execution itself hits its quota, and Standard workflow executions can run for up to a year. A reviewer who leaves, a review UI that drops the token, a queue consumer that crashes after reading the message and before writing it to the review table: all of these produce an execution that is neither failed nor finished. It just sits there. Nobody alerts on it because nothing broke.</p>



<pre class="wp-block-code"><code>"AwaitAdjusterDecision": {
  "Type": "Task",
  "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
  "Parameters": {
    "FunctionName": "enqueue-review-task",
    "Payload": {
      "claimId.$": "$.claimId",
      "taskToken.$": "$$.Task.Token"
    }
  },
  "TimeoutSeconds": 259200,
  "HeartbeatSeconds": 3600,
  "Catch": [{
    "ErrorEquals": ["States.Timeout"],
    "Next": "EscalateStaleReview"
  }],
  "Next": "ApplyDecision"
}
</code></pre>



<p class="wp-block-paragraph"><code>TimeoutSeconds</code> is the maximum total lifetime of the task regardless of heartbeats. <code>HeartbeatSeconds</code> is the maximum gap between <code>SendTaskHeartbeat</code> calls, so a review app that periodically confirms the item is still in someone&#8217;s queue will fail fast when that app dies, rather than at the outer limit. AWS&#8217;s own guidance is to set the heartbeat below the task timeout for exactly this reason: a heartbeat failure tells you the worker died, a timeout tells you the work took too long, and those are different incidents. Catch <code>States.Timeout</code> and route to a real state. An unhandled timeout is just a differently-shaped silence.</p>



<p class="wp-block-paragraph">One constraint worth knowing before you design around it: the callback pattern requires Standard workflows. Express workflows support request-response integrations only, so no <code>.waitForTaskToken</code> and no <code>.sync</code>. If you split your pipeline into a fast Express path and a Standard review path, the boundary between them is where the token has to live.</p>



<h2 class="wp-block-heading">Failure family four: the human review service you can no longer sign up for</h2>



<p class="wp-block-paragraph">This one catches people copying a reference architecture, and it&#8217;s the reason to read publication dates on IDP blog posts.</p>



<p class="wp-block-paragraph">Amazon Augmented AI, known as A2I, was the managed answer to human-in-the-loop review. It plugged directly into Textract&#8217;s <code>AnalyzeDocument</code>, watched confidence conditions, and spun up review tasks with a worker UI for you. It appears in a great many architecture diagrams for claims and lending workflows.</p>



<p class="wp-block-paragraph">Per the AWS documentation, SageMaker A2I is no longer open to new customers. Existing customers can keep using it, and AWS continues security and availability work, but no new features are planned. If you&#8217;re standing up a new account today, that diagram does not deploy.</p>



<p class="wp-block-paragraph">Be fair about what that costs you, because A2I genuinely removed real work: the task assignment logic, the worker UI, the private workforce plumbing through Cognito, result consolidation. Rebuilding it means owning all of that. What you get back is that the review queue becomes yours, which in practice means you can put claim-specific context on the screen instead of a generic key-value editor. For adjusters that difference is not cosmetic.</p>



<p class="wp-block-paragraph">A minimal replacement is not exotic:</p>



<ul class="wp-block-list">
<li>A DynamoDB table of review items, each holding the claim identifier, the extracted values, the bounding boxes, and the Step Functions task token.</li>

<li>A small web app for reviewers that renders the page image with the boxes overlaid, so a reviewer confirms placement rather than retyping values.</li>

<li>An API that writes the corrected values and calls <code>SendTaskSuccess</code> with the stored token.</li>

<li>Authentication in front of it. Amazon Cognito if you want to stay inside AWS, or an identity-aware proxy such as Cloudflare Access if your reviewers are external adjusters you&#8217;d rather not create AWS identities for.</li>

<li>A sweeper that finds review items older than your heartbeat window and escalates them.</li>
</ul>



<p class="wp-block-paragraph">If the review app is a small internal tool with no data residency requirement of its own, it doesn&#8217;t have to live in the same account or even the same provider. A modest VPS from a host like Contabo or InterServer running behind a Cloudflare Tunnel is a legitimate answer for a reviewer console that talks to AWS over scoped API credentials. Just be honest about what crosses that boundary, which brings us to the next family.</p>



<h2 class="wp-block-heading">Failure family five: claim data in places nobody decided to put it</h2>



<p class="wp-block-paragraph">Claims documents carry protected health information, financial identifiers, and often photographs of people and property. The pipeline you drew has three or four places that data lives. The pipeline you deployed has a dozen.</p>



<p class="wp-block-paragraph">The ones that get missed:</p>



<ul class="wp-block-list">
<li><strong>Lambda logs.</strong> One <code>print</code> of an event payload during a debugging session, and CloudWatch Logs is now a claims repository with a different retention policy and a different access model.</li>

<li><strong>Dead letter queues.</strong> A DLQ holds the full failed message. If that message carries extracted values, your DLQ is regulated data, and it is usually the least governed thing in the account.</li>

<li><strong>Step Functions execution history.</strong> State input and output are visible in the console and the history API. Passing extracted fields between states puts them there.</li>

<li><strong>Intermediate extraction output.</strong> The bucket you pointed <code>OutputConfig</code> at holds raw OCR of the whole packet, often with a lifecycle policy nobody wrote.</li>

<li><strong>Model invocation logging.</strong> Bedrock can log inputs and outputs to S3 or CloudWatch. Useful for debugging, and another copy of everything.</li>
</ul>



<p class="wp-block-paragraph">The pattern that keeps this manageable is passing pointers, not payloads. States carry an S3 key and a claim identifier; the values themselves stay in one encrypted bucket with one lifecycle policy and one access policy. It makes debugging marginally more annoying and it makes the data map fit on a page.</p>



<p class="wp-block-paragraph">On regulated workloads, check the current AWS HIPAA-eligible services list and your executed BAA for every service in the path, in the specific region you&#8217;re deploying to. Eligibility is per service and it changes. Textract has long been used for claims workflows on that basis, and Bedrock is listed as HIPAA eligible, but &#8220;I read a blog post&#8221; is not a control. Pull the list yourself before PHI touches anything.</p>



<p class="wp-block-paragraph">Two smaller things worth deciding early. Reviewers working from home should reach the console over something better than the open internet, whether that&#8217;s a corporate tunnel, a business VPN account from a provider like NordVPN or Surfshark, or an identity-aware proxy. And if reviewers ever download claim documents locally, agree what happens to those files afterward, because a deleted file is not an erased file. Tools such as O&amp;O SafeErase exist for exactly that gap on Windows endpoints.</p>



<h2 class="wp-block-heading">Failure family six: paying twice for the same page</h2>



<p class="wp-block-paragraph">Document AI services bill per page. That single fact reshapes how you think about retries, because in most pipelines a retry is free and here it isn&#8217;t.</p>



<p class="wp-block-paragraph">The expensive patterns are all shaped the same way. A poison document fails a downstream parser, gets requeued, and is re-extracted on every attempt. A batch job re-runs over an entire prefix instead of a delta. A misconfigured S3 event delivers twice. An operator reprocesses a day&#8217;s intake to fix a mapping bug in the transform stage, when the extraction stage was fine all along.</p>



<p class="wp-block-paragraph">The structural fix is separating extraction from interpretation. Extract once, write the raw result to S3 keyed by a content hash of the document, and let every downstream stage read from that. When the mapping bug shows up, you re-run interpretation over stored output and pay nothing. Combined with <code>ClientRequestToken</code>, most accidental double-charges disappear.</p>



<p class="wp-block-paragraph">Also route the packet before you extract it. Forty pages of accident photographs do not need forms and tables analysis. Classification is cheaper than extraction, and page-level routing is often the single largest lever on the bill.</p>



<p class="wp-block-paragraph">For attributing that spend, cost allocation tags on the buckets and functions give you the AWS-native view, and platforms like Vantage or CloudZero are worth a look if you need per-claim or per-client unit costs rather than per-service totals. Whatever you use, the metric that matters is cost per claim processed, split by straight-through versus reviewed. Those two numbers tell you whether the automation is earning its keep.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting an insurance claims processing pipeline on AWS</h2>



<p class="wp-block-paragraph">Symptoms you&#8217;ll actually see, and where to look first.</p>



<ul class="wp-block-list">
<li><strong>Executions succeed but downstream amounts are wrong.</strong> Check whether required fields are present, not just confident. Diff the schema against the response keys for a sample of recent claims. This is failure family one until proven otherwise.</li>

<li><strong>Executions stuck in Running for days.</strong> A callback task with no timeout. List running executions ordered by start time and look for the state name of your review task.</li>

<li><strong>Throttling on the extraction stage under load.</strong> If you&#8217;re polling <code>Get</code> operations, move to <code>OutputConfig</code> and SNS notification and stop polling. If you&#8217;re already there, check the start-operation limits rather than assuming the whole service is slow.</li>

<li><strong>The same claim appearing twice.</strong> Look for a missing or per-invocation <code>ClientRequestToken</code>, and check whether your S3 event handler is idempotent. Delivery is at-least-once.</li>

<li><strong>Extraction quality dropped for one document type.</strong> Usually the form changed, not the model. Compare bounding boxes for the affected field against an older sample. If the box moved, that&#8217;s a layout change, and query aliases or a blueprint update is the fix.</li>

<li><strong>Review queue growing faster than reviewers clear it.</strong> Break the routing reasons apart. If most items are low confidence on one field, that&#8217;s an extraction problem wearing a staffing problem&#8217;s clothes.</li>

<li><strong>SNS notification arrives, handler can&#8217;t find the results.</strong> Confirm the notification role has permission to publish and the handler is reading the S3 prefix rather than calling <code>Get</code> with an expired job identifier.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Gating only on confidence, so a missing field is indistinguishable from a clean extraction.</li>

<li>Defaulting absent values to zero or null in the transform layer instead of raising a routing decision.</li>

<li>Copying an architecture diagram that includes A2I into a new AWS account.</li>

<li>Callback tasks with no <code>TimeoutSeconds</code> and no heartbeat.</li>

<li>Passing extracted claim values through Step Functions state rather than passing an S3 pointer.</li>

<li>Treating a single global confidence threshold as adequate for every field. A name and a dollar amount do not carry the same downstream risk.</li>

<li>Running forms and tables analysis over every page of a packet including the photographs.</li>

<li>No metric for straight-through rate, so nobody notices when it quietly drops.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li><strong>Declare the schema, then validate presence, then confidence, then invariants.</strong> Four gates, four distinct rejection reasons, four things a reviewer can act on.</li>

<li><strong>Set per-field thresholds by consequence.</strong> Get the payable amount wrong and money moves. Get a street suffix wrong and a letter is slightly odd.</li>

<li><strong>Keep a labeled regression set.</strong> A few dozen real packets with known-correct values, run on every blueprint or query change. Bedrock Data Automation can use ground-truth examples to refine blueprint instructions, which only works if you maintain the ground truth.</li>

<li><strong>Store bounding boxes alongside values.</strong> They make review faster and they turn &#8220;quality dropped&#8221; from a guess into a comparison.</li>

<li><strong>Alarm on rates, not just errors.</strong> Straight-through rate, review-queue age, and cost per claim. A pipeline that stops approving anything is broken even though nothing threw.</li>

<li><strong>Make every stage idempotent on a content hash.</strong> Reprocessing is normal. It should be safe and cheap.</li>

<li><strong>Instrument the pipeline like a pipeline.</strong> CloudWatch covers the AWS surface; if you&#8217;re consolidating with on-premises claims systems, a platform like Grafana Cloud gives you one place to correlate both sides.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Should I use Amazon Textract or Bedrock Data Automation for claims extraction?</h3>



<p class="wp-block-paragraph">Textract is the sharper tool when your documents are standardized forms and you want deterministic OCR with forms, tables and targeted queries. Bedrock Data Automation is stronger on mixed packets, because it splits along logical document boundaries, classifies each part, and applies a blueprint per document type, with confidence scores and visual grounding on the output. Claims intake is usually mixed packets, which tilts toward Data Automation, but the honest answer is to run both against a sample of your real documents. The evaluation costs a day and it decides your architecture.</p>



<h3 class="wp-block-heading">What replaces Amazon A2I for human review?</h3>



<p class="wp-block-paragraph">For new AWS accounts, a custom review path: a queue or table of review items, a reviewer UI, and the Step Functions callback pattern to resume the workflow. It&#8217;s more code than A2I but not a large amount, and it gives you a review screen designed around claims rather than around generic key-value pairs. Existing A2I customers can continue as they are, though building on a service with no planned features is a decision to make deliberately rather than by default.</p>



<h3 class="wp-block-heading">What straight-through processing rate should I expect?</h3>



<p class="wp-block-paragraph">Anyone quoting you a number without seeing your documents is guessing. It depends almost entirely on document quality and how many fields you require. What&#8217;s reliable is the method: measure your current rate, split failures by reason, and fix the largest reason. Requiring one rarely-present field can dominate everything else, and that&#8217;s a policy decision as much as an engineering one.</p>



<h3 class="wp-block-heading">How do I keep an insurance claims processing pipeline on AWS HIPAA-aligned?</h3>



<p class="wp-block-paragraph">Start from the AWS HIPAA-eligible services list and an executed BAA, and confirm eligibility for each service in your specific region. Then do the unglamorous work: customer-managed KMS keys, no PHI in logs or state payloads, scoped IAM roles per stage, VPC endpoints where the service supports them, retention policies on every bucket and queue including dead letter queues, and CloudTrail configured so you can answer who accessed which claim. Eligibility is permission to build; the controls are yours.</p>



<h3 class="wp-block-heading">Can I run the whole pipeline with Step Functions Express workflows?</h3>



<p class="wp-block-paragraph">Not the part that waits for a human. Express workflows support request-response integrations only, so the callback pattern requires Standard. A common split is Express for the high-volume deterministic stages and Standard for anything holding a task token, with the two connected by an event or a queue.</p>



<h3 class="wp-block-heading">How do I stop duplicate claims from duplicate events?</h3>



<p class="wp-block-paragraph">Treat every trigger as at-least-once. Derive an idempotency key from the document itself, usually a content hash plus a version marker, pass it as <code>ClientRequestToken</code> to the extraction call, and use it as the conditional write key when you create the claim record. Then a duplicate event is a no-op rather than a second claim.</p>



<h3 class="wp-block-heading">Is it worth starting from the AWS GenAI IDP Accelerator?</h3>



<p class="wp-block-paragraph">As a reference for structure and as a way to get a working pipeline in front of stakeholders quickly, yes. As a production system you inherit wholesale, be careful: you&#8217;re adopting someone else&#8217;s opinions about classification, review and storage, and you&#8217;ll be reading that code anyway the first time something behaves oddly. Read it, borrow the patterns, own what you deploy.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">An insurance claims processing pipeline on AWS is not hard to build. Textract, Bedrock Data Automation, Step Functions and S3 will get you a working pipeline in a couple of weeks. What&#8217;s hard is making it fail in ways you can see.</p>



<p class="wp-block-paragraph">Every expensive failure in this space shares one shape: the pipeline had no opinion about what it did not receive. A field that didn&#8217;t come back scored nothing, a callback that never fired errored nothing, a duplicate event failed nothing. If you take one design rule from this, take that one. Declare what a complete claim looks like, check for its absence explicitly, and route anything incomplete to a human with a reason attached.</p>



<p class="wp-block-paragraph">Green executions are not evidence. Reconciled numbers are.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with your claims pipeline?</h2>



<p class="wp-block-paragraph">I work with teams building document-heavy workflows on AWS, and claims pipelines are one of the places where a small amount of design care prevents a large amount of reconciliation work. Things I can help with:</p>



<ul class="wp-block-list">
<li>Reviewing an existing extraction pipeline for silent-failure paths, particularly missing-field handling and default values in the transform layer</li>

<li>Designing the routing logic: schema gates, per-field thresholds, cross-field invariants, and the escalation rules that sit behind them</li>

<li>Building a human review path with the Step Functions callback pattern, including timeouts, heartbeats and a sweeper for stale tasks</li>

<li>Running a structured evaluation of Amazon Textract against Bedrock Data Automation on your actual documents, with a labeled regression set you keep afterward</li>

<li>Tracing where claim data actually lands across logs, queues, execution history and intermediate buckets, then shrinking that footprint</li>

<li>Instrumenting straight-through rate, review-queue age and cost per claim so regressions surface before finance finds them</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;d like a second opinion, send me a state machine definition, a sample extraction response with the values redacted, or the routing code that decides what goes to review. That&#8217;s usually enough to spot the gap.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/insurance-claims-processing-pipeline-aws/">Building an Insurance Claims Processing Pipeline on AWS That Fails Loudly</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Salesforce AWS Integration Architecture: The Four Paths and What Breaks Silently</title>
		<link>https://john-nessime.com/blog/devops/salesforce-aws-integration-architecture/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 19 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AppFlow]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Change Data Capture]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Event Relay]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[Named Credentials]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Platform Events]]></category>
		<category><![CDATA[PrivateLink]]></category>
		<category><![CDATA[Pub/Sub API]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[Serverless]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=209</guid>

					<description><![CDATA[<p>Most Salesforce AWS integrations don't fail loudly, they drift. This guide covers the four integration paths between Salesforce and AWS, when to use each, the silent failure mode hiding in every one of them, and the reconciliation job you have to build yourself.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-aws-integration-architecture/">Salesforce AWS Integration Architecture: The Four Paths and What Breaks Silently</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone in the ops channel asks why the opportunity count on the QuickSight dashboard doesn&#8217;t match the count in the Salesforce report. It&#8217;s off by a few hundred. Nobody can answer that, and worse, nobody can say when the two numbers stopped agreeing. Every pipeline is green. The scheduled flow reports successful runs. The EventBridge rule shows invocations. Everything is healthy except the data.</p>



<p class="wp-block-paragraph">That&#8217;s the shape of most Salesforce AWS integration problems. They&#8217;re rarely outages. They&#8217;re divergence, accumulating quietly while every status page stays green, and you find out about it when someone in finance runs a number past a customer.</p>



<p class="wp-block-paragraph">This guide covers the four integration paths between Salesforce and AWS, which one to reach for and when, the specific failure each one hides, and the reconciliation layer you&#8217;ll have to build yourself because neither vendor ships it.</p>



<h2 class="wp-block-heading">Why a Salesforce AWS integration fails silently</h2>



<p class="wp-block-paragraph">Start here, because it changes how you design everything downstream. There are three mechanisms that lose data without raising an error anywhere you&#8217;re looking.</p>



<p class="wp-block-paragraph"><strong>Events sent to an unassociated partner event bus are dropped on the floor.</strong> When you create an Event Relay, Salesforce creates a partner event source in your AWS account. Until you associate that source with an event bus, EventBridge discards anything the partner publishes to it. AWS is explicit that those events are not persisted at rest. There&#8217;s no queue holding them for you, no backlog to drain once you finish the setup. If the relay is running and the bus isn&#8217;t associated, you are streaming into a void.</p>



<p class="wp-block-paragraph"><strong>The Salesforce event bus retains high-volume events for 72 hours.</strong> Platform events and change data capture events sit on the event bus for three days, and each one carries a replay ID so a durable subscriber can resume from where it left off. Past that window, they&#8217;re purged. Salesforce doesn&#8217;t guarantee availability beyond it. So a relay that goes into an error state on a Friday afternoon is a data loss event by Monday morning, and the only symptom during those three days is an absence of messages, which looks identical to a quiet weekend.</p>



<p class="wp-block-paragraph"><strong>Deletes don&#8217;t show up in query-based syncs.</strong> A scheduled pull selects records that exist. Records that stopped existing between runs are simply not in the result set, so your S3 prefix or Redshift table keeps a row that Salesforce no longer has. Nothing errors. The row count on the AWS side just creeps upward relative to the CRM, forever, until someone notices.</p>



<p class="wp-block-paragraph">Every architecture decision below is really a decision about which of these three you&#8217;re accepting and how you&#8217;re going to detect it.</p>



<h2 class="wp-block-heading">A Salesforce AWS integration starts with a direction, not a service</h2>



<p class="wp-block-paragraph">The most common mistake I see is picking a service first, usually whichever one appeared in a conference demo, and then bending the requirement to fit it. Two questions decide the path, and they take about a minute to answer.</p>



<ol class="wp-block-list">
<li><strong>Which way does the data move?</strong> Salesforce to AWS, or AWS to Salesforce. If the answer is &#8220;both&#8221;, you have two integrations, not one, and they will have different failure modes and different owners.</li>

<li><strong>What shape is the workload?</strong> Batch, where you want a consistent set of records on a schedule, or event-driven, where you want a reaction within seconds of a change.</li>
</ol>



<p class="wp-block-paragraph">That gives you four cells, and each cell has a default answer.</p>



<h3 class="wp-block-heading">Salesforce to AWS, batch: Amazon AppFlow</h3>



<p class="wp-block-paragraph">Amazon AppFlow is the managed answer for pulling Salesforce objects into S3, Redshift, or the Glue Data Catalog on a schedule or on demand. It handles the OAuth connection, field mapping, and incremental pulls without you writing or hosting anything. For analytics loads it&#8217;s usually the right default, and it&#8217;s the path of least resistance if the destination is a data lake you&#8217;re already cataloguing with Glue and querying with Athena.</p>



<p class="wp-block-paragraph">The trade-off is that it&#8217;s a polling client, and it spends your Salesforce API allocation to do it. AWS documents this plainly with an example: an hourly flow pulling five pages of data makes 120 API calls a day, and all of them count against the org&#8217;s 24-hour API request limit. Multiply that by every object you&#8217;re syncing and every environment you&#8217;re syncing from, and AppFlow can become the largest single consumer of an allocation that Apex callouts, integration users, and third-party tools are also drawing on.</p>



<p class="wp-block-paragraph">The failure mode that catches people: if a field referenced in the flow&#8217;s mapping is deleted in Salesforce, the flow run fails. Someone cleans up an unused custom field on a Tuesday and your nightly load breaks on Wednesday, in a service that nobody on the Salesforce side knows exists. AppFlow will automatically pick up newly created Salesforce fields when the destination is S3, but the reverse case, a removed field, needs a human to edit the mapping. Put the flow names somewhere the Salesforce admins will see them before they run a field cleanup.</p>



<h3 class="wp-block-heading">Salesforce to AWS, event-driven: Event Relay and Amazon EventBridge</h3>



<p class="wp-block-paragraph">Event Relay subscribes to a channel on the Salesforce event bus and streams platform events and change data capture events into Amazon EventBridge as a partner event source. No middleware, no subscriber you have to keep alive, no polling and therefore no API allocation burn on the read path. From EventBridge you fan out to Lambda, Step Functions, SQS, Firehose, whatever the downstream is.</p>



<p class="wp-block-paragraph">Setup runs across both consoles, and the order matters. On the Salesforce side you create a named credential that points at your AWS account and region. It uses no authentication protocol, because trust is established by the partner event source relationship rather than by a credential in the request. The region goes in uppercase, which is the single most common reason a first attempt doesn&#8217;t work.</p>



<pre class="wp-block-code"><code>POST /services/data/vXX.X/tooling/sobjects/NamedCredential/

{
  "FullName": "AwsRelayCredential",
  "Metadata": {
    "label": "AwsRelayCredential",
    "endpoint": "arn:aws:US-EAST-1:111122223333",
    "principalType": "NamedUser",
    "protocol": "NoAuthentication",
    "generateAuthorizationHeader": true
  }
}</code></pre>



<p class="wp-block-paragraph">The endpoint is not a URL. It&#8217;s an ARN-shaped string carrying the AWS region and the twelve-digit account ID, and that&#8217;s the whole addressing scheme. Get the account wrong and the partner event source appears in a stranger&#8217;s console, not yours.</p>



<p class="wp-block-paragraph">You then create an event channel, add the platform events or change events you want as channel members, and create the relay configuration against that channel. A relay is created in a stopped state deliberately, so you get a chance to verify the AWS account and region before anything flows. That pause is the useful part of the design, not an inconvenience to click through.</p>



<p class="wp-block-paragraph">Now switch to AWS and associate the source. This is the step that, skipped, produces the silent drop described earlier.</p>



<pre class="wp-block-code"><code># Find the partner event source Salesforce created in your account
aws events list-event-sources --region us-east-1

# Create the matching bus. Both values must be the source name, character for character.
aws events create-event-bus 
  --name "aws.partner/salesforce.com/&lt;org-and-channel-suffix&gt;" 
  --event-source-name "aws.partner/salesforce.com/&lt;org-and-channel-suffix&gt;" 
  --region us-east-1

# Confirm the source moved from PENDING to ACTIVE
aws events describe-event-source 
  --name "aws.partner/salesforce.com/&lt;org-and-channel-suffix&gt;" 
  --region us-east-1</code></pre>



<p class="wp-block-paragraph"><code>describe-event-source</code> returning <code>ACTIVE</code> is your proof that a matching bus exists and is enabled. <code>PENDING</code> means either you haven&#8217;t created the bus or it&#8217;s deactivated, and every event published in that state is gone. This is the single check worth wiring into a synthetic monitor, because it&#8217;s cheap to run and it catches the most expensive mistake.</p>



<p class="wp-block-paragraph">Only once the source is active do you set the relay state to run. From the Salesforce side, the feedback object is where relay errors surface rather than in any log you&#8217;d normally tail:</p>



<pre class="wp-block-code"><code>SELECT Id, RemoteResource, Status, ErrorMessage, ErrorTime, ErrorIdentifier
FROM EventRelayFeedback
WHERE EventRelayConfigId = '7k2XXXXXXXXXXXXXXX'</code></pre>



<p class="wp-block-paragraph">If you&#8217;d rather not use Event Relay at all, the Pub/Sub API is the direct route: a gRPC interface to the same event bus, delivering Avro-encoded payloads, which you subscribe to from your own consumer. That&#8217;s more code and a process to keep running, on ECS or a VPS from a provider like InterServer or Hetzner if you&#8217;re keeping it outside AWS, but you get explicit control over replay ID checkpointing. Teams already running Kafka often go this way and land the stream through a managed connector from Confluent instead. Event Relay is the lower-operational-cost option; the Pub/Sub API is the one to reach for when you need the checkpoint under your own control.</p>



<h3 class="wp-block-heading">AWS to Salesforce, event-driven: EventBridge API Destinations</h3>



<p class="wp-block-paragraph">Going the other way, an API destination lets EventBridge call the Salesforce REST API directly, with no Lambda in the middle. You create a connection holding OAuth client credentials from a Salesforce connected app, point the destination at your org&#8217;s My Domain endpoint, and route matched events to it. Publishing back into an inbound platform event is the common pattern, so the write lands on the Salesforce event bus and Apex or Flow picks it up from there.</p>



<p class="wp-block-paragraph">Two constraints shape the design, and both are documented:</p>



<ul class="wp-block-list">
<li><strong>Five second client execution timeout.</strong> If Salesforce takes longer than that to respond, EventBridge times out the request and treats it as a retryable failure. Anything you make synchronous behind that endpoint, a trigger doing a callout, a validation chain, a busy org at peak, will eat into it.</li>

<li><strong>The default retry policy is generous, which is the problem.</strong> For event buses, EventBridge will retry a failed delivery for up to 24 hours and up to 185 times by default. If the endpoint is genuinely broken, that&#8217;s a day of retries against your org, consuming API calls, before the event is discarded with nowhere to go.</li>
</ul>



<p class="wp-block-paragraph">Tighten the policy and always attach a dead letter queue. Failed events land somewhere you can inspect and replay rather than evaporating at the end of the retry window:</p>



<pre class="wp-block-code"><code>{
  "Id": "salesforce-writeback",
  "Arn": "arn:aws:events:us-east-1:111122223333:api-destination/sf-platform-event/&lt;id&gt;",
  "RoleArn": "arn:aws:iam::111122223333:role/EventBridgeApiDestinationRole",
  "RetryPolicy": {
    "MaximumRetryAttempts": 20,
    "MaximumEventAgeInSeconds": 3600
  },
  "DeadLetterConfig": {
    "Arn": "arn:aws:sqs:us-east-1:111122223333:salesforce-writeback-dlq"
  }
}</code></pre>



<p class="wp-block-paragraph">Set the invocation rate limit on the destination to something your org can absorb. It exists specifically so you can align delivery with a third-party API&#8217;s limits, and Salesforce is very much a third-party API with limits.</p>



<p class="wp-block-paragraph">One thing worth knowing before you design around it: API destinations don&#8217;t support mutual TLS. If your security review requires mTLS on outbound calls, you&#8217;re putting a Lambda or a proxy in the path regardless.</p>



<h3 class="wp-block-heading">AWS to Salesforce, batch: Lambda and the Bulk API</h3>



<p class="wp-block-paragraph">For loading volume back into the org, nothing managed does this well and you write it yourself. A Lambda or a container job authenticates with the JWT bearer flow against a connected app, submits a Bulk API job, and polls for results. The REST API is fine for small writes; past a few thousand records the per-call overhead and the API allocation make the Bulk API the only sensible choice.</p>



<p class="wp-block-paragraph">This is the cell where you own everything: retries, idempotency, partial-failure handling, and backoff when the org throttles you. Use external ID fields and upsert semantics rather than insert, so a retried batch converges instead of duplicating. Store the job ID before you submit, not after, so a Lambda timeout mid-flight leaves you something to reconcile against. And keep the Salesforce credential in Secrets Manager with rotation wired up, because a hardcoded connected app secret in an environment variable is the finding that shows up in every audit.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The network layer: public internet, PrivateLink, or Private Connect</h2>



<p class="wp-block-paragraph">All four paths above work over the public internet with TLS, and for most workloads that&#8217;s genuinely fine. The question is whether your compliance posture or your risk appetite says otherwise.</p>



<p class="wp-block-paragraph">Salesforce Private Connect is the managed private path. Salesforce operates a transit VPC in each supported AWS region, and you create a PrivateLink connection between that transit VPC and yours. Traffic between the two never traverses the public internet. The terminology is worth getting right because it trips people up in design reviews:</p>



<ul class="wp-block-list">
<li><strong>Inbound</strong> means traffic flowing <em>into</em> Salesforce. AWS to Salesforce, or on-premises to Salesforce.</li>

<li><strong>Outbound</strong> means traffic flowing <em>out of</em> Salesforce. Salesforce calling a service in your VPC.</li>
</ul>



<p class="wp-block-paragraph">The direction is named from Salesforce&#8217;s point of view, not yours, and getting it backwards means provisioning the wrong connection.</p>



<p class="wp-block-paragraph">AppFlow can ride on this too. Choosing the PrivateLink option when you create the Salesforce connection has AppFlow route through Private Connect and manage the endpoint lifecycle for you, which is a meaningful simplification over building it by hand.</p>



<p class="wp-block-paragraph">Where I&#8217;d push back: Private Connect is a licensed add-on, and each VPC needs its own connection, so a multi-account AWS estate with separate VPCs per environment multiplies the cost. If the driver is a checkbox on a questionnaire rather than a real threat model, TLS with strict IP allowlisting on the Salesforce side and a well-scoped connected app gets you most of the way for nothing. If the driver is a regulator who wants to see that customer data never touches a public network path, Private Connect is the clean answer and the audit conversation ends quickly. Decide which one you&#8217;re in before you start pricing it.</p>



<h2 class="wp-block-heading">The reconciliation job nobody builds</h2>



<p class="wp-block-paragraph">This is the part that separates an integration that survives from one that quietly rots, and it&#8217;s about thirty lines of code plus a schedule.</p>



<p class="wp-block-paragraph">Run a count comparison on a window that both sides can express. Nightly is plenty:</p>



<pre class="wp-block-code"><code>-- Salesforce side, via the REST query endpoint
SELECT COUNT(Id) FROM Opportunity
WHERE LastModifiedDate = LAST_N_DAYS:1

-- AWS side, same window, in Athena
SELECT COUNT(DISTINCT id) FROM crm.opportunity
WHERE last_modified_date &gt;= date_add('day', -1, current_date)</code></pre>



<p class="wp-block-paragraph">If the two numbers differ by more than a small tolerance for clock skew at the window edges, alert. That single check catches a stopped relay, a failed flow, a mapping that silently dropped a field, and a partner event source sitting in <code>PENDING</code>. It catches all of them without you having to instrument each path separately, which is exactly why it&#8217;s worth building first.</p>



<p class="wp-block-paragraph">For the delete problem, Salesforce exposes a dedicated endpoint. Ask it for records removed in a window and apply those deletions on the AWS side:</p>



<pre class="wp-block-code"><code>GET /services/data/vXX.X/sobjects/Opportunity/deleted/
    ?start=&lt;ISO-8601 timestamp, URL encoded&gt;
    &amp;end=&lt;ISO-8601 timestamp, URL encoded&gt;</code></pre>



<p class="wp-block-paragraph">The window you can ask about is bounded by how long deleted records stay recoverable, so run it more often than that bound, not less. Change data capture delivers delete events too, which is the better answer when it&#8217;s available, but the endpoint is what you use to backfill after a gap, and after a 72-hour gap it&#8217;s the <em>only</em> thing that will tell you what you missed.</p>



<p class="wp-block-paragraph">Put both checks on one dashboard alongside your AppFlow run status and EventBridge failed-invocation metrics. CloudWatch will do it, and if you&#8217;re already running Grafana Cloud or Datadog for the rest of the estate, pull the Salesforce-side numbers in as a custom metric so one panel answers &#8220;are the two systems agreeing right now&#8221;.</p>



<h2 class="wp-block-heading">Troubleshooting a Salesforce AWS integration</h2>



<ul class="wp-block-list">
<li><strong>Relay is running, nothing arrives in EventBridge.</strong> Run <code>describe-event-source</code>. If the state is <code>PENDING</code>, the bus was never created or has been deactivated, and everything published so far is gone. Create the bus, then treat the gap as a backfill job.</li>

<li><strong>Relay won&#8217;t start, or flips to an error state.</strong> Query <code>EventRelayFeedback</code> for the config. The usual causes are a wrong AWS account ID in the named credential, a region in lowercase, or the channel having no members.</li>

<li><strong>Events arrive but no rule matches.</strong> Check the rule is attached to the partner event bus, not the default bus. This one costs people an afternoon. Match on the source prefix and the detail type, and test with an event pattern before wiring a real target.</li>

<li><strong>An AppFlow flow that ran for months suddenly fails.</strong> Look for a field in the mapping that no longer exists in the source object. Also check for a bad gateway error, which points at a connection configured with the wrong instance URL and is fixed by deleting the connection and recreating it rather than editing it.</li>

<li><strong>API destination returns 401 intermittently.</strong> The connection refreshes its OAuth token when it sees a 401 or 407. Persistent 401s usually mean the connected app has no usable scopes for the client credentials flow, or the run-as user was changed. Check the connected app&#8217;s scopes before you touch anything in AWS.</li>

<li><strong>Everything times out at almost exactly five seconds.</strong> That&#8217;s the API destination timeout, not Salesforce being slow. Move the work behind an inbound platform event so the HTTP call returns immediately and the processing happens asynchronously in the org.</li>

<li><strong>Unexplained API limit exhaustion.</strong> Sum the scheduled AppFlow runs against page counts before blaming a rogue integration user. A frequent flow across several objects adds up faster than most people expect.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Starting the relay before associating the partner event bus, and assuming the events queued somewhere.</li>

<li>Treating the 72-hour retention as a comfortable buffer. It&#8217;s three days including a weekend, and an unmonitored relay burns through it without a single alert.</li>

<li>Using change data capture as a replacement for an initial load. It streams changes from the moment you enable it; it doesn&#8217;t backfill history.</li>

<li>Building the whole integration bidirectionally through one mechanism, then discovering you&#8217;ve created an echo loop where a write from AWS fires a change event that triggers another write.</li>

<li>Sizing the AppFlow schedule for freshness without checking what it does to the org&#8217;s API allocation.</li>

<li>Leaving the default 24-hour EventBridge retry policy on an API destination with no dead letter queue.</li>

<li>Provisioning Private Connect for every VPC because it&#8217;s the secure-sounding option, without modelling the per-connection cost across environments.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Build the reconciliation check before you build the second integration path. It&#8217;s the only thing that tells you the whole system is working.</li>

<li>Monitor the partner event source state, not just rule invocations. Invocations go to zero on a quiet day and on a broken day, and the state field distinguishes them.</li>

<li>Give every integration its own connected app and its own integration user. Shared credentials make it impossible to attribute API consumption or revoke one path without breaking three.</li>

<li>Make writes idempotent with external IDs and upsert. Every path here retries, and some retry aggressively.</li>

<li>Version your event payloads. Platform events are immutable once published, and adding a required field later means a new event definition, not an edit.</li>

<li>Keep the AWS-side resource names discoverable from the Salesforce side. Put the flow name or event bus name in the named credential label so a Salesforce admin can see what&#8217;s attached before they change something.</li>

<li>Define the whole thing in Terraform or CloudFormation on the AWS side and in metadata on the Salesforce side, so a sandbox refresh doesn&#8217;t leave you clicking through two consoles from memory.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need MuleSoft to integrate Salesforce with AWS?</h3>



<p class="wp-block-paragraph">No. For the four patterns above, the native services cover it without an integration platform. MuleSoft, Boomi, and similar tools earn their place when you&#8217;re orchestrating across many systems, need transformation logic that lives outside both platforms, or have a governance requirement for a single integration layer. For a two-system integration, they&#8217;re usually more machinery than the problem needs.</p>



<h3 class="wp-block-heading">Does Event Relay use my Salesforce API allocation?</h3>



<p class="wp-block-paragraph">Event delivery counts against your platform event delivery allocation, which is a separate budget from the 24-hour API request limit that REST calls and AppFlow polling draw on. That separation is one of the strongest arguments for the event-driven path when API consumption is already tight in the org.</p>



<h3 class="wp-block-heading">Can I replay Salesforce events that AWS missed?</h3>



<p class="wp-block-paragraph">Within the 72-hour retention window, yes, using replay IDs from a durable subscriber. Beyond it, no. The events are purged and there&#8217;s no recovery path through the event bus. Backfilling from that point means querying the objects directly and using the deleted-records endpoint to catch removals, which is exactly why the reconciliation job matters.</p>



<h3 class="wp-block-heading">Should I use AppFlow or write my own Lambda extractor?</h3>



<p class="wp-block-paragraph">Start with AppFlow. It removes the auth handling, the pagination, and the scheduling, and for standard object pulls into S3 or Redshift it&#8217;s hard to justify writing that yourself. Move to a custom extractor when you need transformation logic AppFlow can&#8217;t express, when you&#8217;re hitting connector limitations on a specific object, or when you want tighter control over API consumption than a fixed schedule allows.</p>



<h3 class="wp-block-heading">How do I send data from AWS back into Salesforce without a Lambda?</h3>



<p class="wp-block-paragraph">EventBridge API destinations call the Salesforce REST API directly using OAuth client credentials from a connected app. Point them at an inbound platform event endpoint so the write returns fast and processing happens asynchronously in the org. You still need a Lambda if you require mTLS or payload transformation beyond what the input transformer handles.</p>



<h3 class="wp-block-heading">Is Private Connect required for a secure Salesforce AWS integration?</h3>



<p class="wp-block-paragraph">Not required. Everything runs over TLS by default. Private Connect removes the public internet from the path entirely, which matters when a regulator or a security review asks you to demonstrate it. It&#8217;s a licensed add-on with a per-connection cost, so model it against your actual VPC count before committing.</p>



<h3 class="wp-block-heading">Why does my EventBridge rule never fire even though events are arriving?</h3>



<p class="wp-block-paragraph">Almost always because the rule is on the default event bus rather than the partner event bus. Partner events land on their own bus and rules are bus-scoped, so a correct pattern on the wrong bus matches nothing and reports nothing.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A Salesforce AWS integration doesn&#8217;t usually break loudly. It drifts. A relay stops, a mapping goes stale, a delete never propagates, and the two systems keep answering questions confidently with different numbers. The services themselves are well built and the setup is genuinely not hard once you know the region goes in uppercase and the bus has to be associated first.</p>



<p class="wp-block-paragraph">So pick the path by direction and shape, wire the failure modes you&#8217;ve accepted into alerts, and build the nightly count comparison before you build anything clever. That one job is worth more than any amount of pipeline polish, because it&#8217;s the only thing in the architecture that can tell you the whole thing is still true.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your Salesforce and AWS setup?</h2>



<p class="wp-block-paragraph">I work with teams on the infrastructure side of CRM integrations, usually where the data is disagreeing and nobody can prove why. Things I can help with:</p>



<ul class="wp-block-list">
<li>Reviewing an existing Salesforce to AWS pipeline and finding where records are being lost between the two.</li>

<li>Standing up Event Relay into EventBridge end to end, including the partner bus association and the monitoring that proves it&#8217;s live.</li>

<li>Designing the AppFlow schedule and object set so it doesn&#8217;t eat the org&#8217;s API allocation.</li>

<li>Building the reconciliation and delete-propagation jobs, with alerting in CloudWatch, Grafana, or whatever you already run.</li>

<li>Writing the write-back path properly: Bulk API jobs, idempotent upserts, dead letter queues, and credential rotation through Secrets Manager.</li>

<li>Working out whether Private Connect is worth the licence in your specific environment, or whether TLS and tighter scoping gets you there.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;ve got a flow that&#8217;s failing, a relay stuck in an error state, or two record counts that won&#8217;t agree, send me the config or the error and I&#8217;ll tell you what I&#8217;d check first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-aws-integration-architecture/">Salesforce AWS Integration Architecture: The Four Paths and What Breaks Silently</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</title>
		<link>https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 17 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Cursor Pagination]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[GraphQL]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Orchestration]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[Step Functions]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=215</guid>

					<description><![CDATA[<p>A GraphQL source can hand you a 200 OK, a populated data block, and a quietly broken column in the same response. Here is how to build a GraphQL data ingestion pipeline on AWS that catches partial errors, respects cost-based rate limits, resumes cleanly from a cursor, and notices when the schema moves under you.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/">Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The row counts looked fine. That was the problem.</p>



<p class="wp-block-paragraph">A nightly sync from a partner&#8217;s GraphQL API had run for weeks without a single failed execution. Green across the board in the console. Then someone asked why one column had been empty since the previous Tuesday. Not sparse. Empty. Every row, every night, for six nights.</p>



<p class="wp-block-paragraph">The API had started returning a field-level error on that one field. Status code 200. A populated <code>data</code> block. An <code>errors</code> array sitting right next to it. The fetcher parsed <code>data</code>, wrote the records to S3, exited zero, and the orchestrator logged a success. Nothing in the stack was wrong, exactly. Everything in the stack was checking the wrong thing.</p>



<p class="wp-block-paragraph">This post covers how to build a GraphQL data ingestion pipeline on AWS that fails honestly. It is organised by failure family rather than by service, because the AWS wiring is the easy part. The hard part is that GraphQL breaks four assumptions that REST ingestion quietly relies on, and every one of them fails silently by default.</p>



<h2 class="wp-block-heading">Why GraphQL ingestion breaks the habits REST taught you</h2>



<p class="wp-block-paragraph">Most ETL code carries three inherited reflexes from years of REST work: the status code tells you whether it worked, the response shape is the server&#8217;s business, and rate limits are counted in requests. GraphQL invalidates all three.</p>



<ul class="wp-block-list">
<li><strong>The status code is not the contract.</strong> The GraphQL over HTTP specification says a server should return 200 even when field errors are raised during execution, because a partial response is still a successful execution.</li>

<li><strong>You own the response shape.</strong> The selection set is written by you, which means schema drift shows up as a query that stops being valid, not as a payload that changes underneath you.</li>

<li><strong>Cost is query-shaped, not endpoint-shaped.</strong> One endpoint can cost almost nothing or blow your entire budget depending on how deeply you nested it.</li>
</ul>



<p class="wp-block-paragraph">Each of those becomes a distinct failure family. Let&#8217;s take them in the order they bite.</p>



<h2 class="wp-block-heading">Failure one: the 200 OK that quietly ate a column</h2>



<p class="wp-block-paragraph">This is the one that costs the most and shows up the latest. GraphQL splits errors into two categories, and they behave completely differently on the wire.</p>



<ul class="wp-block-list">
<li><strong>Request errors</strong> happen before execution starts: syntax errors, validation failures, bad variable coercion. No <code>data</code> key is returned at all. Nothing executed.</li>

<li><strong>Field errors</strong> happen during execution: a resolver threw, a downstream service timed out, a permission check failed. Execution continues, the failed field goes null, and you get <code>data</code> and <code>errors</code> together.</li>
</ul>



<p class="wp-block-paragraph">Field errors are the dangerous ones for ingestion, because they look exactly like success to anything that only inspects the transport layer. Worse, GraphQL&#8217;s null propagation rules mean a single failing non-null field bubbles up to the nearest nullable ancestor, so one broken resolver can hollow out an entire nested object while the surrounding rows land normally.</p>



<p class="wp-block-paragraph">The fix is to make the response body the contract. Classify before you commit anything:</p>



<pre class="wp-block-code"><code>import json, urllib.request

class RequestError(Exception): pass
class FieldError(Exception): pass

def graphql(endpoint, token, query, variables):
    body = json.dumps({"query": query, "variables": variables}).encode()
    req = urllib.request.Request(
        endpoint,
        data=body,
        headers={
            "Content-Type": "application/json",
            "Accept": "application/graphql-response+json, application/json",
            "Authorization": f"Bearer {token}",
        },
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        payload = json.loads(resp.read())

    # The status code told us nothing. The body is the contract.
    if "data" not in payload:
        # Nothing executed. Query or auth problem. Do not retry blindly.
        raise RequestError(payload.get("errors"))

    if payload.get("errors"):
        # Partial response. data is populated but incomplete.
        raise FieldError(payload["errors"])

    return payload["data"]</code></pre>



<p class="wp-block-paragraph">Two things worth explaining. The <code>Accept</code> header asks for <code>application/graphql-response+json</code>, the media type the GraphQL over HTTP spec recommends, which lets a compliant server return a real 4xx for pre-execution failures instead of burying them in a 200. Servers that don&#8217;t support it fall back to <code>application/json</code>, so including both is free.</p>



<p class="wp-block-paragraph">The second is the split between the two exception types. They need different handling. A request error is a bug in your query or a credential problem, and retrying it just burns budget on the same failure. A field error is transient often enough to be worth retrying, but never worth committing.</p>



<h3 class="wp-block-heading">Deciding what &#8220;partial&#8221; means for your data</h3>



<p class="wp-block-paragraph">Failing the whole page on any error is the safe default and the one I reach for first. But it&#8217;s not always right. If you are pulling a hundred fields and one optional enrichment field is flaky, refusing every page forever is its own outage.</p>



<p class="wp-block-paragraph">The workable middle ground is to classify by path. Every error object carries a <code>path</code> array pointing at the exact field that failed. Maintain a small allowlist of paths that are permitted to be null, fail the page on anything outside it, and emit the error count as a CloudWatch metric so a rising trend is visible before someone in finance notices.</p>



<h2 class="wp-block-heading">Failure two: rate limits that never send a 429</h2>



<p class="wp-block-paragraph">Your retry logic almost certainly triggers on 429 and 5xx. Several major GraphQL APIs will throttle you without ever sending one.</p>



<p class="wp-block-paragraph">GitHub&#8217;s GraphQL API is explicit about this: exceeding the primary rate limit returns a 200 response with an error message in the body and <code>x-ratelimit-remaining</code> set to zero. Your Step Functions retrier sees a success. Your Lambda sees no data. The run completes, having ingested nothing, and reports green.</p>



<p class="wp-block-paragraph">The other half of the problem is that request counting is the wrong unit. Shopify&#8217;s GraphQL Admin API and several others price queries by <em>calculated cost</em>, using a leaky bucket. The server statically analyses your query before executing it, charges the bucket the worst-case <em>requested</em> cost, executes, then refunds the difference between requested and <em>actual</em> cost. That has a consequence people miss: a query asking for the maximum connection size gets charged as if every page were full, even when the last page returns three records.</p>



<p class="wp-block-paragraph">The good news is that these APIs tell you the state of the bucket on every response, under the <code>extensions</code> key:</p>



<pre class="wp-block-code"><code>{
  "extensions": {
    "cost": {
      "requestedQueryCost": 72,
      "actualQueryCost": 38,
      "throttleStatus": {
        "maximumAvailable": 1000.0,
        "currentlyAvailable": 962.0,
        "restoreRate": 100.0
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Read those numbers, never hardcode them. Bucket size and restore rate vary by plan tier, and a query that works fine against a large merchant&#8217;s store will fail instantly against a small one. The pattern that holds up is a pre-flight check: before dispatching, compare your estimated cost against <code>currentlyAvailable</code>, and if you&#8217;re short, sleep for the shortfall divided by <code>restoreRate</code>.</p>



<p class="wp-block-paragraph">On APIs that expose their own budget as a queryable field, ask for it in the same round trip rather than spending a second call to find out:</p>



<pre class="wp-block-code"><code>query BudgetCheck {
  rateLimit {
    limit
    cost
    remaining
    resetAt
  }
}</code></pre>



<p class="wp-block-paragraph">Two more things about concurrency. First, if you fan out across Lambda invocations, each worker&#8217;s local view of the bucket is wrong, because the bucket is shared. Multiple workers independently deciding they have headroom is the classic way to get throttled while every individual worker believes it&#8217;s behaving. Either centralise the token accounting or cap concurrency low enough that the arithmetic can&#8217;t go wrong.</p>



<p class="wp-block-paragraph">Second, if the source offers a bulk or async export operation, use it for backfills. Paginating fifty thousand records through the interactive endpoint burns the same budget your incremental syncs need, and it competes with itself.</p>



<h2 class="wp-block-heading">Failure three: cursors that don&#8217;t survive a restart</h2>



<p class="wp-block-paragraph">Most production GraphQL schemas follow the Relay cursor connections convention: <code>first</code> and <code>after</code> arguments, a <code>pageInfo</code> object with <code>hasNextPage</code> and <code>endCursor</code>, and edges carrying opaque per-item cursors.</p>



<pre class="wp-block-code"><code>query Orders($cursor: String) {
  orders(first: 100, after: $cursor) {
    pageInfo {
      hasNextPage
      endCursor
    }
    edges {
      cursor
      node {
        id
        updatedAt
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">The filter argument for incremental sync is not standardised. Some schemas take a query-string filter, some take a typed timestamp argument, some expose nothing at all. Run introspection against the schema and find out rather than guessing, because a wrong argument name is a request error and your run will fail before it starts.</p>



<p class="wp-block-paragraph">The failure here is subtle. Cursors are opaque and their validity is entirely up to the server. Some are stable positional pointers. Some encode a sort key and a snapshot identifier and expire. If a page-47 fetch fails and you retry an hour later with the stored cursor, you may get an error, or you may get results from a shifted window with a hole in the middle. The second outcome is worse, because it looks like it worked.</p>



<p class="wp-block-paragraph">Three habits make restarts safe:</p>



<ol class="wp-block-list">
<li><strong>Checkpoint after the write, not after the fetch.</strong> A cursor recorded before the page lands in S3 is a promise you can&#8217;t keep.</li>

<li><strong>Store a high watermark alongside the cursor.</strong> The cursor is your fast path. The watermark, taken from the newest record you actually committed, is your fallback when the cursor is rejected.</li>

<li><strong>Overlap the watermark deliberately.</strong> Restart a few minutes before the last committed timestamp and deduplicate on the primary key downstream. Re-reading a handful of rows is cheap. Missing rows that arrived during a clock skew is not.</li>
</ol>



<p class="wp-block-paragraph">A DynamoDB item per source stream is enough state for all of this:</p>



<pre class="wp-block-code"><code>{
  "source": "orders",
  "run_id": "01hq7k...",
  "cursor": "eyJsYXN0X2lkIjoxMjM0fQ==",
  "high_watermark": "&lt;updatedAt of newest committed record&gt;",
  "pages_committed": 47,
  "status": "IN_PROGRESS"
}</code></pre>



<p class="wp-block-paragraph">Overlap only works if downstream deduplication is real. Write raw pages with an ingest-time partition and a run identifier in the object key, then let the curated layer resolve duplicates by primary key and latest <code>updatedAt</code>. If you write straight into a merged table with no dedupe step, overlapping restarts will double-count and you&#8217;ll have traded silent loss for silent inflation.</p>



<h2 class="wp-block-heading">Failure four: the schema moved and nobody told you</h2>



<p class="wp-block-paragraph">GraphQL&#8217;s deprecation story is genuinely better than REST&#8217;s. Fields carry a <code>@deprecated</code> directive with a reason string, and introspection exposes it. The problem is that nothing forces you to look.</p>



<p class="wp-block-paragraph">The failure mode is a field that gets deprecated, stops being populated, then eventually gets removed. Stage one is invisible: your query still validates, the field returns null, and your warehouse fills with nulls. Stage two is loud but late: the query becomes invalid and the whole ingest breaks at 3am.</p>



<p class="wp-block-paragraph">Put an introspection diff in CI. Fetch the schema on a schedule, store it as an artifact, and diff against the last known good version. Fail the build on removals or type changes to fields your queries actually select, and warn on new deprecations. Tools in the Apollo and Hasura ecosystems ship schema-diff tooling for this, and rolling your own against the introspection result is not much work either. Either way, the point is that the check runs on a schedule rather than when the pager goes off.</p>



<p class="wp-block-paragraph">The complementary habit is to keep queries narrow. Every field you select is a field that can break you, a field that costs budget, and a field that widens your Glue table. Selecting a whole object because it was convenient during development is a liability you carry indefinitely.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Wiring the GraphQL data ingestion pipeline on AWS</h2>



<p class="wp-block-paragraph">With the failure families understood, the AWS assembly is mostly a set of trade-offs about where the fetch loop lives.</p>



<h3 class="wp-block-heading">Where the fetch loop runs</h3>



<p class="wp-block-paragraph">Step Functions can call an HTTPS endpoint directly with an HTTP Task, no Lambda involved. It uses an EventBridge connection to hold the credentials, which means your token lives in a Secrets Manager secret managed by EventBridge rather than in an environment variable:</p>



<pre class="wp-block-code"><code>"FetchPage": {
  "Type": "Task",
  "Resource": "arn:aws:states:::http:invoke",
  "Parameters": {
    "ApiEndpoint": "https://api.example.com/graphql",
    "Method": "POST",
    "InvocationConfig": {
      "ConnectionArn": "arn:aws:events:region:account-id:connection/name/id"
    },
    "Headers": { "Content-Type": "application/json" },
    "RequestBody": {
      "query.$": "$.query",
      "variables.$": "$.variables"
    }
  },
  "Retry": [{
    "ErrorEquals": [
      "States.Http.StatusCode.429",
      "States.Http.StatusCode.502",
      "States.Http.StatusCode.503",
      "States.Http.StatusCode.504"
    ],
    "IntervalSeconds": 1,
    "BackoffRate": 2,
    "MaxAttempts": 3,
    "JitterStrategy": "FULL"
  }],
  "Next": "InspectBody"
}</code></pre>



<p class="wp-block-paragraph">The execution role needs four permissions: <code>states:InvokeHTTPEndpoint</code>, <code>events:RetrieveConnectionCredentials</code>, and both <code>secretsmanager:GetSecretValue</code> and <code>secretsmanager:DescribeSecret</code> scoped to the EventBridge-managed secret.</p>



<p class="wp-block-paragraph">It&#8217;s a clean pattern, and I like it for small, high-value syncs. Two constraints decide whether it fits. HTTP Task requests time out after 60 seconds, which a deeply nested GraphQL query can exceed without being pathological. And Step Functions reads the response within its standard state payload ceiling of roughly 256 KB, so a page of a hundred rich records will not fit. Also note the retrier above: it fires on HTTP status codes, so it will not catch a throttle that arrives as a 200 with an error in the body. That check has to happen in the state after it.</p>



<p class="wp-block-paragraph">The alternative is a Lambda driver that paginates internally and writes pages straight to S3, returning only the cursor and a count to the state machine. This dodges both limits, gives you a real HTTP client with real backoff, and keeps large payloads out of the workflow&#8217;s state. It costs you per-page retry granularity, which you buy back with the checkpoint record.</p>



<p class="wp-block-paragraph">Distributed Map is worth mentioning for the fan-out case, such as one child execution per tenant or per shard. It gives you high concurrency and a per-item execution history, which is genuinely useful for debugging. Be aware of two things before committing: express child workflows cap at five minutes rather than Lambda&#8217;s fifteen, and practitioners have published benchmarks showing throughput well below what the headline concurrency numbers suggest for small, fast items. For a lot of ingestion work, SQS plus a DynamoDB job table remains the simpler answer.</p>



<p class="wp-block-paragraph">One case genuinely falls outside serverless: sources that require a long-lived session, a persistent websocket subscription, or a synchronisation window longer than fifteen minutes with no resumable cursor. A small always-on worker on a modest VPS from a provider like InterServer or Hetzner, feeding the same S3 landing zone, is less clever and more honest than contorting Lambda around the constraint.</p>



<h3 class="wp-block-heading">The landing zone</h3>



<p class="wp-block-paragraph">Land raw responses first, unmodified, including the <code>errors</code> and <code>extensions</code> keys. Storage is the cheapest thing in this architecture and the raw layer is what lets you reprocess after you discover a parsing bug six weeks later.</p>



<ul class="wp-block-list">
<li><strong>Partition by ingest time, not event time.</strong> Event-time partitioning requires reading the record before you know where to put it, which means you can&#8217;t write until you&#8217;ve parsed. Ingest-time keys let the raw write be dumb and fast.</li>

<li><strong>Put the run identifier in the key.</strong> When you need to unwind one bad run, being able to delete or ignore by prefix beats writing a surgical delete.</li>

<li><strong>Define Glue tables explicitly rather than crawling.</strong> You wrote the selection set, so you already know the schema. A crawler inferring types from JSON will surprise you the first time an optional field is absent from an entire page.</li>

<li><strong>Convert to columnar in the curated layer.</strong> Athena over raw JSON works and is fine for debugging. Parquet is what you want for anything anyone queries repeatedly.</li>
</ul>



<p class="wp-block-paragraph">If your source pushes rather than being polled, such as webhook deliveries carrying GraphQL payloads, Amazon Data Firehose (renamed from Kinesis Data Firehose, with APIs and IAM policies unchanged) will buffer, convert to Parquet and partition on the way to S3 without you running a consumer. For scheduled pull-based syncs it adds a hop you don&#8217;t need.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Runs succeed but a column is all null.</strong> Field errors. Pull a raw response from the landing zone and look at the <code>errors</code> array. The <code>path</code> tells you exactly which field.</li>

<li><strong>Runs succeed but ingest zero rows.</strong> Likely a throttle returned as a 200. Log the response body and the rate-limit headers on empty results, always.</li>

<li><strong>Intermittent failures partway through pagination.</strong> Cursor expiry. Fall back to the watermark and check whether the source documents a cursor lifetime.</li>

<li><strong>Query fails immediately with no <code>data</code> key.</strong> Request error. Validate against current introspection; something in the schema moved.</li>

<li><strong>Throttled despite staying under the documented limit.</strong> Either concurrent workers are sharing a bucket they each think they own, or you&#8217;re being charged requested cost on near-empty pages.</li>

<li><strong>HTTP Task fails on large pages.</strong> The response exceeds the state payload ceiling. Reduce page size or move the fetch into Lambda.</li>

<li><strong>Row counts drift upward over time.</strong> Overlap without deduplication. Check that the curated layer actually resolves on the primary key.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating a 200 as success and never reading the body.</li>

<li>Retrying request errors. They are deterministic; retrying just spends budget to fail identically.</li>

<li>Hardcoding bucket sizes and restore rates that differ per plan tier and change without notice.</li>

<li>Checkpointing the cursor before the data is durable.</li>

<li>Requesting the maximum page size on every call while being charged the worst-case cost for it.</li>

<li>Selecting entire objects because it was easier during development.</li>

<li>Alerting only on execution failure, so a job that succeeds while ingesting nothing never pages anyone.</li>

<li>Overlapping restart windows with no downstream deduplication.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Classify every response into clean, partial, or request-failed before anything downstream sees it.</li>

<li>Persist raw responses with <code>errors</code> and <code>extensions</code> intact, then parse from the raw layer.</li>

<li>Alarm on row-count deltas and partial-error rate, not just on execution status.</li>

<li>Read the throttle state the API hands you and back off before you hit the wall, not after.</li>

<li>Keep both a cursor and a watermark, and know which one is authoritative on restart.</li>

<li>Diff the schema on a schedule so deprecations reach you before they reach your data.</li>

<li>Hold credentials in Secrets Manager or an EventBridge connection, never in a Lambda environment variable.</li>

<li>Trace the pipeline end to end. CloudWatch covers the basics; Grafana or Datadog earn their keep once you&#8217;re correlating child executions across a fan-out.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can Step Functions call a GraphQL API without a Lambda function?</h3>



<p class="wp-block-paragraph">Yes. An HTTP Task using the <code>arn:aws:states:::http:invoke</code> resource posts your query directly, with an EventBridge connection handling authentication. It works well for small pages. The 60-second request timeout and the state payload size ceiling are what push most real ingestion work back into Lambda.</p>



<h3 class="wp-block-heading">How do I detect GraphQL errors when the status code is always 200?</h3>



<p class="wp-block-paragraph">Inspect the body. If there&#8217;s no <code>data</code> key, nothing executed and you have a request error. If <code>data</code> is present alongside a non-empty <code>errors</code> array, you have a partial response and the data is incomplete. Only a response with <code>data</code> and no errors is unambiguously clean.</p>



<h3 class="wp-block-heading">Does AWS AppSync fit into this?</h3>



<p class="wp-block-paragraph">Only if you&#8217;re on the serving side. AppSync is a managed GraphQL API for exposing your own data, so it&#8217;s what you&#8217;d reach for if you wanted clients to push events in via mutations. For pulling data out of somebody else&#8217;s GraphQL API, AppSync isn&#8217;t part of the picture.</p>



<h3 class="wp-block-heading">Should I store raw GraphQL responses or parsed records in S3?</h3>



<p class="wp-block-paragraph">Both, in separate layers. Raw responses are your audit trail and your reprocessing path, and they&#8217;re the only way to answer &#8220;was this field null in the source, or did my parser drop it?&#8221; weeks after the fact. Parsed and columnar output goes in a curated prefix that analysts query.</p>



<h3 class="wp-block-heading">How do I run an incremental sync from a GraphQL API?</h3>



<p class="wp-block-paragraph">Find the schema&#8217;s filter argument for a modification timestamp using introspection, pass your stored watermark into it, and page forward with cursors. Overlap the window slightly on each run and deduplicate on the primary key downstream. If the schema exposes no such filter, you&#8217;re limited to full pulls or webhook-driven change capture.</p>



<h3 class="wp-block-heading">Why am I throttled when I&#8217;m making very few requests?</h3>



<p class="wp-block-paragraph">Cost-based APIs charge for query complexity, not call count. A single deeply nested query pulling a connection inside a connection can consume a large fraction of the bucket by itself. Read the cost figures the API returns and flatten the query if the requested cost is consistently much higher than the actual cost.</p>



<h3 class="wp-block-heading">Is GraphQL a worse ingestion source than REST?</h3>



<p class="wp-block-paragraph">Different, not worse. You control the payload, which cuts transfer and lets you avoid the N+1 fan-out REST forces on nested data. Deprecation metadata is machine-readable, which REST rarely offers. The cost is that error signalling and rate limiting both moved into the body, so naive clients fail silently instead of loudly.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A GraphQL data ingestion pipeline on AWS fails differently from a REST one, and the difference is almost entirely about where the truth lives. In REST, the transport tells you what happened. In GraphQL, the transport tells you the request arrived, and the body tells you what happened. Every failure family in this post traces back to that: partial errors hidden behind a 200, throttles hidden behind a 200, cursors that fail by returning the wrong rows rather than an error.</p>



<p class="wp-block-paragraph">So build the pipeline so that a green run means something. Parse before you commit, alarm on row deltas rather than exit codes, and keep the raw responses so you can prove what the source actually said. Everything else is ordinary AWS plumbing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with a GraphQL ingestion pipeline?</h2>



<p class="wp-block-paragraph">This is the kind of work I take on regularly. Specifically:</p>



<ul class="wp-block-list">
<li>Auditing an existing GraphQL sync for silent partial-error loss and telling you which columns are already affected</li>

<li>Building the fetch, checkpoint and land loop in Step Functions and Lambda with cursors that survive restarts</li>

<li>Fitting a cost-aware throttle to whatever budget model your source actually uses, including multi-worker coordination</li>

<li>Designing the raw and curated S3 layout, Glue table definitions and Athena partitioning so queries stay cheap</li>

<li>Adding introspection diffing to CI so schema deprecations surface in a pull request, not in a dashboard</li>

<li>Wiring CloudWatch metrics and alarms on row-count deltas and partial-error rates, so a green run is actually green</li>
</ul>



<p class="wp-block-paragraph">Send me a run log, a state machine definition, or a raw response with an <code>errors</code> array in it, and I&#8217;ll tell you what it&#8217;s costing you.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/">Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Streaming Shopify Events into AWS Without Losing Orders</title>
		<link>https://john-nessime.com/blog/devops/streaming-shopify-events-into-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 16 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon SQS]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Dead Letter Queue]]></category>
		<category><![CDATA[DynamoDB]]></category>
		<category><![CDATA[Ecommerce Analytics]]></category>
		<category><![CDATA[Event-Driven Architecture]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Partner Event Source]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Reliability Engineering]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[Terraform]]></category>
		<category><![CDATA[Webhooks]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=221</guid>

					<description><![CDATA[<p>Wiring Shopify webhooks into Amazon EventBridge takes an afternoon. Keeping every order is the hard part. A walk through the five failure families that actually bite when streaming Shopify events into AWS: the partner source that silently drops everything, duplicate and out-of-order deliveries, rule patterns that match nothing, targets that fail without a dead-letter queue, and the 64 KB metering rule that quietly inflates the bill.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/streaming-shopify-events-into-aws/">Streaming Shopify Events into AWS Without Losing Orders</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The partner event source in the EventBridge console said <code>Pending</code>. It had said <code>Pending</code> for six days.</p>



<p class="wp-block-paragraph">Nobody noticed, because nothing errored. No 5xx in a log. No failed delivery in Shopify&#8217;s dashboard. No alarm. Shopify had been publishing order events the entire time, and AWS had been throwing every single one of them on the floor.</p>



<p class="wp-block-paragraph">That behaviour is documented, in one short note in the AWS docs: events published to a partner event source that has not been associated with an event bus are dropped immediately and are not persisted at rest. There is no retry for that. There is no buffer. The events are gone, and the only way to get the data back is to go ask the Shopify Admin API for it after the fact.</p>



<p class="wp-block-paragraph">That is the shape of most of the pain in this integration. Streaming Shopify events into AWS is easy to stand up and easy to get quietly wrong, and every one of the quiet failures looks identical from the outside: everything is green, and some of your data isn&#8217;t there.</p>



<p class="wp-block-paragraph">This post walks the five failure families that actually cost you records, plus the reconciliation layer that most teams only build after the first incident. It assumes you can read a rule pattern and an IAM policy. It does not assume you have shipped this before.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">What the pipe actually looks like</h2>



<p class="wp-block-paragraph">Four moving parts, and only two of them live in your account.</p>



<ol class="wp-block-list"><li>A Shopify app holds the webhook subscriptions. Each subscription has a topic and a delivery method. For this path the delivery method is EventBridge and the address is an ARN, not a URL.</li><li>Shopify creates a <strong>partner event source</strong> inside your AWS account, in the region you nominated.</li><li>You associate that source with a <strong>partner event bus</strong>. This is the step everyone forgets.</li><li>Rules on that bus match events and push them at targets: Lambda, SQS, Step Functions, Firehose, whatever fits.</li></ol>



<p class="wp-block-paragraph">The ARN trips people up more than anything else in the setup. Shopify wants the <em>event source</em> ARN, not the event bus ARN. They look similar and only one of them works:</p>



<pre class="wp-block-code"><code># Correct - the event source ARN. Note the empty account field.
arn:aws:events:eu-west-1::event-source/aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;

# Wrong - this is the bus, and Shopify will reject it
arn:aws:events:eu-west-1:123456789012:event-bus/aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;</code></pre>



<p class="wp-block-paragraph">Associating the source is a single call, and both the name and the source name are the same string:</p>



<pre class="wp-block-code"><code># Create the partner event bus that accepts the source
aws events create-event-bus 
  --name "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;" 
  --event-source-name "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;" 
  --region eu-west-1

# Confirm it flipped from PENDING to ACTIVE
aws events describe-event-source 
  --name "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;" 
  --region eu-west-1</code></pre>



<p class="wp-block-paragraph">The same architecture applies if you are not on Shopify. BigCommerce and commercetools both publish to EventBridge as partner sources, and the failure families below are identical because they come from EventBridge&#8217;s semantics, not the store&#8217;s.</p>



<h2 class="wp-block-heading">Failure family one: events that never existed</h2>



<p class="wp-block-paragraph">This is the one from the opening, and it is the most expensive because it is completely silent on both sides.</p>



<p class="wp-block-paragraph">Shopify considers the delivery successful. It handed the event to the partner source, which is its contract. AWS considers nothing to have happened, because an unassociated source has no bus to write to, and EventBridge does not persist events at rest before a bus exists. Your CloudWatch metrics show nothing, because metrics are emitted per bus and per rule, and you have neither.</p>



<p class="wp-block-paragraph">The same class of hole opens up in two other ways:</p>



<ul class="wp-block-list"><li><strong>Region mismatch.</strong> The source is created in the region you gave Shopify. Your bus, your rules, your targets and your dead-letter queues all have to be in that region. A rule in the right account but the wrong region matches nothing, forever, without complaint.</li><li><strong>Environment drift.</strong> A staging store pointed at a production source, or a source created against an account ID that belonged to an old sandbox. Nothing errors. Events just land somewhere you are not looking.</li></ul>



<p class="wp-block-paragraph">The fix is boring and it works: treat the source state as a monitored asset. A scheduled job that calls <code>describe-event-source</code> and alarms if <code>State</code> is anything other than <code>ACTIVE</code> costs you twenty minutes and covers the entire failure family. Put it next to your other synthetic checks, not inside the pipeline it is watching.</p>



<p class="wp-block-paragraph">The second half of that check is a heartbeat on volume. If a bus that normally sees a few thousand events a day sees zero for an hour, that is an incident even when every component reports healthy. Alarm on <code>MatchedEvents</code> hitting zero, not just on errors.</p>



<h2 class="wp-block-heading">Failure family two: events that arrive twice, or backwards</h2>



<p class="wp-block-paragraph">EventBridge is at-least-once. Shopify&#8217;s webhooks are at-least-once. Neither one promises ordering. Put those together and you get two distinct bugs that people usually try to fix with one patch.</p>



<p class="wp-block-paragraph">The duplicate is the obvious one. The same <code>orders/create</code> arrives twice, and if your handler posts to a fulfilment provider or sends a customer email, you have just done it twice. The dedupe key is sitting in the envelope: Shopify puts <code>X-Shopify-Webhook-Id</code> into <code>detail.metadata</code>, and it identifies the delivery. Write it into DynamoDB with a conditional put and a TTL of a few days, and drop the event if the write fails.</p>



<p class="wp-block-paragraph">The out-of-order case is the one that costs you money quietly. An <code>orders/updated</code> carrying a cancelled status arrives before the <code>orders/updated</code> carrying the address change, and your database ends up holding the older state because it was written last. Nothing failed. The row is just wrong, and it will stay wrong until someone complains.</p>



<p class="wp-block-paragraph">The envelope carries what you need for this too. <code>detail.metadata</code> includes <code>X-Shopify-Triggered-At</code>, and the resource in <code>detail.payload</code> carries its own <code>updated_at</code>. Compare before you write, and refuse to apply an update whose timestamp is older than the one already stored.</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow"><p>The dedupe key stops you from doing the work twice. The version check stops you from doing the work backwards. They solve different problems and you need both.</p></blockquote>



<p class="wp-block-paragraph">One thing you can skip on this path: HMAC verification. On the HTTPS delivery method you must verify the signature, because anyone can POST to your endpoint. On the EventBridge path, only the partner account behind the event source is permitted to publish to that bus, and the AWS docs are explicit that adding your own resource policy to a partner bus is rejected. The signature header still rides along in the metadata, but the trust boundary is enforced by AWS rather than by your code.</p>



<h2 class="wp-block-heading">Failure family three: rules that match nothing</h2>



<p class="wp-block-paragraph">Every Shopify event, regardless of topic, arrives with the same <code>detail-type</code>. That single fact invalidates the routing instinct most people bring from AWS service events.</p>



<pre class="wp-block-code"><code>{
  "version": "0",
  "id": "1b8e2e75-b771-e964-f0e6-fbca6a21dad8",
  "detail-type": "shopifyWebhook",
  "source": "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;",
  "account": "123456789012",
  "time": "2022-07-02T12:47:58Z",
  "region": "eu-west-1",
  "resources": [],
  "detail": {
    "payload": {
      "id": 1234567890,
      "title": "Columbia Las Hermosas"
    },
    "metadata": {
      "Content-Type": "application/json",
      "X-Shopify-Topic": "products/update",
      "X-Shopify-Shop-Domain": "example.myshopify.com",
      "X-Shopify-Hmac-SHA256": "...",
      "X-Shopify-Webhook-Id": "...",
      "X-Shopify-API-Version": "...",
      "X-Shopify-Triggered-At": "2022-07-02T12:47:57.989779121Z"
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Two things to take from that envelope. The resource body is nested under <code>detail.payload</code>, not at the top of <code>detail</code>, so a pattern copied from an HTTPS handler will match nothing. And the topic lives in <code>detail.metadata</code>, which is where all your routing has to happen.</p>



<pre class="wp-block-code"><code>// Exact topic match
{
  "detail-type": ["shopifyWebhook"],
  "detail": {
    "metadata": {
      "X-Shopify-Topic": ["orders/create"]
    }
  }
}

// Every orders topic, one rule
{
  "detail-type": ["shopifyWebhook"],
  "detail": {
    "metadata": {
      "X-Shopify-Topic": [{ "prefix": "orders/" }]
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Do not deploy a pattern you have not tested against a real envelope. <code>test-event-pattern</code> answers in a second and saves an afternoon:</p>



<pre class="wp-block-code"><code>aws events test-event-pattern 
  --event-pattern file://pattern.json 
  --event file://sample-event.json</code></pre>



<h3 class="wp-block-heading">One rule or thirty?</h3>



<p class="wp-block-paragraph">There is a real argument for a single catch-all rule that pushes everything into one queue and lets your consumer branch on the topic. It is less infrastructure, it deploys faster, and adding a topic does not require a Terraform run.</p>



<p class="wp-block-paragraph">What you give up is per-topic visibility. <code>MatchedEvents</code>, <code>FailedInvocations</code> and the dead-letter queue are all scoped to the rule. Collapse thirty topics into one rule and you can no longer tell that inventory events stopped three days ago, because the aggregate number still looks fine.</p>



<p class="wp-block-paragraph">The split I reach for first: a dedicated rule for each topic that touches money or fulfilment, and one catch-all for everything else. You get precise alarms where the cost of being wrong is high and low overhead everywhere else.</p>



<h2 class="wp-block-heading">Failure family four: events that arrive and die at the target</h2>



<p class="wp-block-paragraph">By default EventBridge keeps retrying a failed target invocation for up to a day, with exponential backoff and jitter. That is generous, and it is also the reason people assume they do not need a dead-letter queue. They do, for two reasons.</p>



<p class="wp-block-paragraph">First, a whole class of errors gets <em>no</em> retries at all. Missing permissions on the target, a target that no longer exists, an address that will not resolve. EventBridge does not retry those, because retrying cannot help. It sends them straight to the DLQ if one is configured, and drops them if one is not.</p>



<p class="wp-block-paragraph">Second, a day of retries is not much when the failure is a bad deploy discovered on a Friday evening.</p>



<pre class="wp-block-code"><code>aws events put-targets 
  --rule shopify-orders-create 
  --event-bus-name "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;" 
  --targets '[{
    "Id": "order-processor",
    "Arn": "arn:aws:lambda:eu-west-1:123456789012:function:order-processor",
    "RetryPolicy": {
      "MaximumRetryAttempts": 20,
      "MaximumEventAgeInSeconds": 3600
    },
    "DeadLetterConfig": {
      "Arn": "arn:aws:sqs:eu-west-1:123456789012:shopify-orders-dlq"
    }
  }]'</code></pre>



<p class="wp-block-paragraph">Lowering the retry window is deliberate here. Twenty-four hours of retries against a genuinely broken consumer buys you nothing and hides the problem; a shorter window pushes failures into the DLQ where they are visible and countable.</p>



<h3 class="wp-block-heading">The DLQ permission trap</h3>



<p class="wp-block-paragraph">This one catches almost everyone who manages infrastructure as code. Configure a DLQ through the console and AWS attaches the queue policy for you. Configure it through <code>PutTargets</code> — which is what Terraform, CloudFormation and the CLI all do — and you must attach it yourself. Miss it, and you have a dead-letter queue that cannot receive dead letters.</p>



<pre class="wp-block-code"><code>{
  "Sid": "Dead-letter queue permissions",
  "Effect": "Allow",
  "Principal": { "Service": "events.amazonaws.com" },
  "Action": "sqs:SendMessage",
  "Resource": "arn:aws:sqs:eu-west-1:123456789012:shopify-orders-dlq",
  "Condition": {
    "ArnEquals": {
      "aws:SourceArn": "arn:aws:events:eu-west-1:123456789012:rule/shopify-orders-create"
    }
  }
}</code></pre>



<p class="wp-block-paragraph">The metric that catches this is <code>InvocationsFailedToBeSentToDlq</code>. If it is ever non-zero, your safety net has a hole in it and events are being lost at the exact moment you were counting on it. Alarm on it at a threshold of one. It only reports when it is non-zero, so it costs nothing the rest of the time.</p>



<p class="wp-block-paragraph">Two more constraints worth knowing before you design around a DLQ: it must be a standard SQS queue, not FIFO, and it must live in the same region as the rule. Each message carries the error code, the exhausted retry condition, the retry count and both ARNs as message attributes, which is usually enough to triage without opening the payload.</p>



<h2 class="wp-block-heading">Failure family five: the bill</h2>



<p class="wp-block-paragraph">EventBridge does not meter one event as one event. It meters in 64 KB chunks, so an event larger than that bills as multiple events. Rates change and vary by region, so check the current pricing page rather than trusting any number you read in a blog post, but the mechanism is stable and it is what determines your bill.</p>



<p class="wp-block-paragraph">This matters more for commerce than for most event sources. A product update is small. An order with thirty line items, per-item discount allocations, tax lines, shipping lines, note attributes and a stack of metafields is not. Wholesale and subscription stores routinely produce order payloads that cross the chunk boundary, and the same order updated eight times through its lifecycle multiplies that.</p>



<p class="wp-block-paragraph">Three levers, roughly in order of how much they return:</p>



<ul class="wp-block-list"><li><strong>Trim at the subscription.</strong> Shopify&#8217;s webhook subscription API lets you restrict which fields are included in the payload and which metafield namespaces come along. Fields you never read cost you at ingestion, at archive and again at replay. This is the only lever that stops paying for the data before it enters AWS.</li><li><strong>Subscribe to fewer topics.</strong> Broad topics like <code>orders/updated</code> fire on changes you do not care about. If you only act on fulfilment state, subscribe to the fulfilment topics instead of filtering a firehose after you have paid for it.</li><li><strong>Archive selectively, and set retention.</strong> Archives bill for processing, for storage and again for replay. An archive with no retention period grows forever. Archive the topics you would genuinely replay and let the rest go.</li></ul>



<p class="wp-block-paragraph">One structural limit to design around: EventBridge caps the total size of a single event. A payload that exceeds it does not get truncated in a helpful way — the publish fails. Trimming at the subscription protects you here as well as on cost.</p>



<h2 class="wp-block-heading">The layer nobody builds until they need it</h2>



<p class="wp-block-paragraph">Archive and replay is genuinely useful, and it is also routinely misunderstood. Replay re-delivers events that <em>reached the bus</em>. It does nothing at all for the failure family at the top of this post, where the events never reached the bus in the first place. Replay fixes bugs in your consumer. It does not fix gaps in your ingestion.</p>



<p class="wp-block-paragraph">For that you need reconciliation: a scheduled job that queries the Shopify Admin API for resources changed since a stored watermark and compares them against what you hold. It is unglamorous, it is the thing that catches the outage you did not know about, and it is worth building before you need it rather than during the incident.</p>



<ul class="wp-block-list"><li>Run it hourly for orders and fulfilments, daily for products and customers. The cadence should track how expensive being wrong is, not how much data there is.</li><li>Store a watermark per topic and advance it only after a successful full pass. A partial pass that advances the watermark creates the exact gap you built the job to find.</li><li>Compare counts first, records second. A count mismatch is cheap to compute and tells you whether to bother with the expensive comparison.</li><li>Emit the drift as a metric, not just a log line. &#8220;Orders in Shopify but not in our store, last hour&#8221; is a graph worth putting on a dashboard, and it should normally read zero.</li></ul>



<p class="wp-block-paragraph">The reconciliation worker does not need to live in Lambda. It is a long, paginated, rate-limited crawl, which is an awkward fit for a function timeout and a comfortable fit for a small VPS you already run. If you have a box at InterServer or Hetzner sitting there for other jobs, a cron entry and a script is a perfectly respectable answer.</p>



<h2 class="wp-block-heading">Troubleshooting by symptom</h2>



<p class="wp-block-paragraph">Work these in order. Each one is cheap and rules out a whole branch.</p>



<h3 class="wp-block-heading">Nothing is arriving at all</h3>



<ol class="wp-block-list"><li>Run <code>describe-event-source</code>. If <code>State</code> is not <code>ACTIVE</code>, stop here. Everything published so far is gone and you need the reconciliation path.</li><li>Confirm the region of the bus matches the region in the source ARN.</li><li>List your webhook subscriptions through the Admin API and confirm the address is the event-source ARN, not the bus ARN.</li><li>Confirm the subscriptions belong to the app whose access token you are using. Registering with a token from a different app is a common and confusing dead end.</li><li>Check <code>MatchedEvents</code> on the bus with no rule dimension. Non-zero means events are landing and your rules are the problem, not the plumbing.</li></ol>



<h3 class="wp-block-heading">Some topics arrive, order or customer topics do not</h3>



<p class="wp-block-paragraph">This is almost always scopes rather than infrastructure. Order and customer topics sit behind protected customer data access, which is a separate approval in the app configuration on top of the read scopes. Without it, product events flow perfectly and order events silently do not — which looks exactly like a broken rule and is not.</p>



<h3 class="wp-block-heading">The rule matches but the target does nothing</h3>



<p class="wp-block-paragraph">Compare <code>MatchedEvents</code> against <code>SuccessfulInvocationAttempts</code> on the rule. A gap sends you to <code>FailedInvocations</code> and to the DLQ. Check the target&#8217;s resource policy, and check <code>InvocationsFailedToBeSentToDlq</code> before you trust that the DLQ is catching anything.</p>



<h3 class="wp-block-heading">Events arrive, but late</h3>



<p class="wp-block-paragraph">Look at <code>ThrottledRules</code> and at <code>IngestionToInvocationSuccessLatency</code>. Sustained throttling usually means an invocation quota rather than a rule problem, and it shows up first during flash sales, which is the worst possible time to discover it. Load-test the path before a peak event, not after.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list"><li>Creating the partner event source and never associating it with a bus. Silent, total, unrecoverable data loss for the whole window.</li><li>Registering the event bus ARN instead of the event source ARN, then debugging Shopify&#8217;s rejection for an hour.</li><li>Writing rule patterns against the resource shape from an HTTPS webhook, forgetting that the body sits under <code>detail.payload</code>.</li><li>Routing on <code>detail-type</code>. Every Shopify event carries the same one, so a pattern that matches on it alone matches everything.</li><li>Configuring a DLQ through Terraform without the queue policy, and only finding out when you needed it.</li><li>Assuming replay covers ingestion gaps. It replays what reached the bus and nothing else.</li><li>Deduplicating on the resource ID instead of the webhook ID, so legitimate subsequent updates get discarded as duplicates.</li><li>Skipping reconciliation because the pipeline &#8220;works&#8221;. It works right up until it doesn&#8217;t, and that is precisely when you need the other path.</li></ul>



<h2 class="wp-block-heading">Best practices for streaming Shopify events into AWS</h2>



<ul class="wp-block-list"><li>Alarm on the event source state and on <code>MatchedEvents</code> reaching zero. Absence of events is a signal, and it is the only signal you get for the worst failure.</li><li>Dedupe on <code>X-Shopify-Webhook-Id</code> and version-check on <code>X-Shopify-Triggered-At</code>. Two mechanisms, two problems.</li><li>Give every target a DLQ and a retry window you chose deliberately, rather than inheriting the default.</li><li>Keep dedicated rules for money and fulfilment topics so their metrics stay legible; batch the rest behind a catch-all.</li><li>Trim payloads at the Shopify subscription rather than in a Lambda. Filtering after ingestion means you already paid for the bytes.</li><li>Define the whole thing in Terraform or CloudFormation, including the queue policies. This stack has too many one-time console clicks to survive being hand-built twice.</li><li>Point your observability platform at the same bus. Datadog and New Relic are both EventBridge partners, so business events and infrastructure telemetry can share one pipeline instead of two.</li><li>Build reconciliation before your first peak trading period, not after your first missing-order ticket.</li></ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I still need to verify the HMAC signature on the EventBridge path?</h3>



<p class="wp-block-paragraph">No. Only the partner account behind the event source can publish to a partner event bus, and AWS actively rejects attempts to add your own resource policy granting anyone else access. The signature header is still present in the metadata, but the trust boundary is enforced by AWS rather than by your handler. On the HTTPS delivery method, verification remains mandatory.</p>



<h3 class="wp-block-heading">Should I use EventBridge or plain HTTPS webhooks?</h3>



<p class="wp-block-paragraph">HTTPS is simpler, works with any host, and is easier to debug because you can curl your own endpoint. It also puts you on the hook for absorbing burst traffic within a short response deadline, and for keeping the endpoint up well enough that Shopify does not remove the subscription after persistent failures. EventBridge moves that burst absorption to AWS and gives you native fan-out. If your consumers already live in AWS, the operational maths favours EventBridge. If they do not, a dedicated reliability layer such as Hookdeck in front of an HTTPS endpoint is a reasonable alternative and a much smaller change.</p>



<h3 class="wp-block-heading">Can I use one partner event source for multiple stores?</h3>



<p class="wp-block-paragraph">Events from every shop that installed your app flow through the source associated with that app, and the shop is identified by <code>X-Shopify-Shop-Domain</code> in the metadata. You can route per-shop with rule patterns matching that field. For genuine tenant isolation — separate accounts, separate blast radius — you want separate apps and separate sources, because a single bus is a single failure domain.</p>



<h3 class="wp-block-heading">Why do I get duplicate order events even though nothing failed?</h3>



<p class="wp-block-paragraph">Because at-least-once means exactly that. Duplicates are normal operation, not a fault to be investigated. Separately, an order genuinely does change several times shortly after creation — payment capture, risk assessment, post-purchase upsells — so several <code>orders/updated</code> events for one order are expected and are not duplicates at all. Deduplicate on the webhook ID to tell the two apart.</p>



<h3 class="wp-block-heading">What happens to events published while my consumer is broken?</h3>



<p class="wp-block-paragraph">They reach the bus, match your rules, and EventBridge retries the target within your configured window. Once that window is exhausted they go to the DLQ if you have one and are discarded if you do not. The events themselves are not lost at the bus level as long as the source is associated — this is the failure family you can actually engineer your way out of.</p>



<h3 class="wp-block-heading">Can I archive and replay Shopify events?</h3>



<p class="wp-block-paragraph">Yes, with an archive on the partner event bus and an event pattern controlling what gets archived. Budget for three separate charges — processing into the archive, storage while it sits there, and the replay itself — and always set an explicit retention period, because an archive without one grows indefinitely.</p>



<h3 class="wp-block-heading">Does this work the same way for BigCommerce or commercetools?</h3>



<p class="wp-block-paragraph">The AWS half is identical: partner source, association step, bus, rules, targets, and every failure family in this post. What differs is the envelope shape and how you register subscriptions on the vendor side. The association gap in particular bites the same way regardless of which platform is publishing.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Almost everything about streaming Shopify events into AWS degrades loudly. Targets throw errors, retries show up as metrics, dead letters pile up in a queue you can see. Those are the failures you will handle correctly, because they announce themselves.</p>



<p class="wp-block-paragraph">The one that will actually hurt you is the one that reports success on both sides while dropping every event on the floor. Association state and event volume are the two signals that catch it, and neither one appears on any dashboard by default. Add them on day one, before you write the first rule. Everything else in this post can be fixed after the fact; that one cannot.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a hand with your event pipeline?</h2>



<p class="wp-block-paragraph">Most of my work on this stack is either standing it up properly the first time or working out where records went after someone else stood it up. Things I can help with:</p>



<ul class="wp-block-list"><li>Building the Shopify-to-EventBridge path end to end in Terraform, including the queue policies and retry configuration that the console quietly does for you.</li><li>Auditing an existing pipeline for silent loss: source association, region drift, missing DLQ permissions, rules that have been matching nothing since the day they shipped.</li><li>Designing the idempotency and ordering layer — dedupe store, TTLs, version checks — so replays and duplicates stop corrupting downstream state.</li><li>Writing the reconciliation job against the Admin API, with watermarks, drift metrics and alarms that fire before a customer does.</li><li>Cutting EventBridge spend by trimming payloads at the subscription and rationalising archive retention, without losing anything you actually query.</li><li>Load-testing the whole path ahead of a peak trading period so throttling shows up in a test window rather than on the day.</li></ul>



<p class="wp-block-paragraph">If you have a rule pattern that isn&#8217;t matching, a DLQ that&#8217;s mysteriously empty, or a bill that grew faster than your order volume, send me the pattern, the metric graph or the line item and I&#8217;ll tell you what I&#8217;d look at first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/streaming-shopify-events-into-aws/">Streaming Shopify Events into AWS Without Losing Orders</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>AWS Amplify vs Firebase: Choosing a Backend You Can Still Leave</title>
		<link>https://john-nessime.com/blog/devops/aws-amplify-vs-firebase/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 10 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[AppSync]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Amplify]]></category>
		<category><![CDATA[BaaS]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Cognito]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Deployment]]></category>
		<category><![CDATA[DynamoDB]]></category>
		<category><![CDATA[Firebase]]></category>
		<category><![CDATA[Firestore]]></category>
		<category><![CDATA[GraphQL]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Vendor Lock-In]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=177</guid>

					<description><![CDATA[<p>Both platforms look identical on day one: a schema file, a typed client, a deploy that works. The difference shows up months later, in the queries you cannot afford and the layer you cannot leave. A practical comparison of data models, billing shape, auth stickiness and exit cost.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-amplify-vs-firebase/">AWS Amplify vs Firebase: Choosing a Backend You Can Still Leave</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket that ends the honeymoon usually looks harmless. Something like: <em>show me last quarter&#8217;s orders, grouped by region, filtered to three statuses, sorted by value</em>. Someone picks it up, opens the backend, and works out that there is no way to answer it without either pulling the whole collection down to the client or standing up a second database next to the first one.</p>



<p class="wp-block-paragraph">That is the moment the <strong>AWS Amplify vs Firebase</strong> decision actually gets made. Not on day one, when both platforms look like a schema file, a typed client and a deploy that just works. On day two hundred, when the read path you committed to without noticing starts charging rent.</p>



<p class="wp-block-paragraph">This post is a comparison, but not a feature tour. Both vendors publish good feature tables and neither of them is lying. What they do not tell you is which parts of the platform you can walk away from cheaply and which parts you are stuck with. So this is organised around the four things that decide that: the data model, the billing shape, the auth layer, and the exit cost.</p>



<h2 class="wp-block-heading">The failure mode that bites: your read path is decided on day one</h2>



<p class="wp-block-paragraph">Both platforms hand you a pleasant abstraction over a NoSQL store. Firebase gives you Firestore, a document database. Amplify&#8217;s data layer, by default, gives you AppSync sitting in front of DynamoDB. In both cases you write a schema, get a typed client, and start shipping.</p>



<p class="wp-block-paragraph">Here is what neither schema file makes obvious: it is not a description of your data. It is a bet on the queries you will need. Change the bet later and you are not editing a schema, you are backfilling.</p>



<p class="wp-block-paragraph">In Firestore, every query shape needs an index that supports it. Filter on two fields and sort on a third and you need a composite index. The console will helpfully offer to create it for you, which is why most teams never notice they are accumulating a per-query-shape index budget. There are no joins. If an order needs its customer&#8217;s name, either you denormalise the name into the order document or you do a second fetch per row.</p>



<p class="wp-block-paragraph">In Amplify, relationships between models are resolved by AppSync, which usually means an extra round trip per relationship. Secondary indexes have to be declared in the schema so the underlying table can be built to serve them. Same constraint, different vocabulary.</p>



<p class="wp-block-paragraph">The good news is that both vendors have finally admitted this is a problem. Firebase Data Connect puts a managed Cloud SQL for PostgreSQL instance behind a GraphQL schema and generates typed SDKs against it. On the AWS side, Amplify&#8217;s data layer can be pointed at a SQL database rather than DynamoDB, and the newer Blocks approach lets you compose Postgres and other capabilities into an existing Amplify backend. If your application has anything resembling reporting, joins or ad-hoc filtering in its future, start there rather than starting on documents and migrating under pressure.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">AWS Amplify: where it wins and where it doesn&#8217;t</h2>



<p class="wp-block-paragraph">Amplify Gen 2 defines the whole backend in TypeScript. Auth, data, storage and functions all live as code in an <code>amplify/</code> directory, get reviewed in pull requests, and compile down through CDK to CloudFormation.</p>



<pre class="wp-block-code"><code>// amplify/data/resource.ts
import { type ClientSchema, a, defineData } from '@aws-amplify/backend';

const schema = a.schema({
  Order: a
    .model({
      region: a.string().required(),
      status: a.enum(['PENDING', 'SHIPPED', 'CANCELLED']),
      total: a.float(),
    })
    .authorization((allow) =&gt; [allow.owner()]),
});

export type Schema = ClientSchema&lt;typeof schema&gt;;

export const data = defineData({ schema });</code></pre>



<p class="wp-block-paragraph">Two things in that snippet matter more than they look. The <code>authorization</code> call is not a comment about intent, it generates the actual resolver-level access rules, so an access change is a code change with a diff. And <code>ClientSchema</code> flows the backend types into the frontend, which means removing a field breaks the build rather than breaking production at 3am.</p>



<p class="wp-block-paragraph">Development happens in a per-developer cloud sandbox. It watches the <code>amplify/</code> folder and redeploys on save:</p>



<pre class="wp-block-code"><code># Deploy and watch your own isolated backend stack
npx ampx sandbox

# Tear it down when you're finished - it is real infrastructure
npx ampx sandbox delete</code></pre>



<h3 class="wp-block-heading">Where Amplify wins</h3>



<ul class="wp-block-list">
<li><strong>You are already on AWS.</strong> If your organisation has accounts, IAM boundaries, a VPC design and a compliance story, Amplify slots into all of it. Firebase would mean a second cloud to audit.</li>

<li><strong>The escape hatch is real.</strong> Because the backend compiles to CDK, you can drop into raw constructs and attach any AWS service without leaving the framework. That ceiling is much higher than Firebase&#8217;s.</li>

<li><strong>Environments are branches.</strong> Git branch to deployed environment is the native model, so staging is not a thing you build, it is a thing you push.</li>

<li><strong>Type safety end to end.</strong> This genuinely catches a class of bug that Firebase projects tend to find in production.</li>
</ul>



<h3 class="wp-block-heading">Where Amplify doesn&#8217;t</h3>



<ul class="wp-block-list">
<li><strong>Gen 1 is on a clock.</strong> Amplify Gen 1 is in maintenance mode and reaches end of life on 1 May 2027. This matters because a lot of the tutorials, Stack Overflow answers and agency codebases you will encounter are Gen 1, and Gen 1 to Gen 2 is a migration, not an upgrade. AWS ships tooling for it, but the guide describes a blue/green run with an irreversible refactor step. Budget for it properly.</li>

<li><strong>&#8220;Local&#8221; development is not local.</strong> The sandbox is a real CloudFormation stack in a real account. It is high fidelity, which is the point, but your inner loop now includes deploy time and your dev environment costs money.</li>

<li><strong>Errors surface as AWS errors.</strong> A mistake in a TypeScript schema can come back as a CloudFormation rollback referencing a resource you never named. Debugging Amplify means being comfortable reading AppSync, DynamoDB and CloudWatch.</li>

<li><strong>Smaller community.</strong> Fewer answers exist, and a meaningful share of the ones that do exist are for the previous generation.</li>
</ul>



<h2 class="wp-block-heading">Google Firebase: where it wins and where it doesn&#8217;t</h2>



<p class="wp-block-paragraph">Firebase&#8217;s advantage is not that it is simpler. It is that the whole loop, from writing a rule to seeing a client behave, is shorter. The emulator suite runs the pieces on your machine:</p>



<pre class="wp-block-code"><code># Run auth, Firestore and functions locally, no cloud project touched
firebase emulators:start --only auth,firestore,functions</code></pre>



<p class="wp-block-paragraph">The thing you will spend the most time on is security rules, and it is worth understanding why they are shaped the way they are:</p>



<pre class="wp-block-code"><code>rules_version = '2';
service cloud.firestore {
  match /databases/{database}/documents {
    match /orders/{orderId} {
      allow read: if request.auth != null
                  &amp;&amp; request.auth.uid == resource.data.ownerUid;

      allow create: if request.auth != null
                    &amp;&amp; request.auth.uid == request.resource.data.ownerUid;

      allow update, delete: if false;
    }
  }
}</code></pre>



<p class="wp-block-paragraph">The distinction people get wrong is <code>resource.data</code> versus <code>request.resource.data</code>. The first is the document as it exists in the database, so it protects the read path. The second is the document the client is trying to write, so it protects the write path. Use the wrong one in a create rule and you are checking ownership on a document that does not exist yet, which evaluates to nothing and quietly lets the write through. That is the kind of bug that ships.</p>



<h3 class="wp-block-heading">Where Firebase wins</h3>



<ul class="wp-block-list">
<li><strong>Mobile.</strong> Crashlytics, Cloud Messaging, Remote Config, A/B testing and analytics are one SDK away and genuinely good. AWS has equivalents, but not assembled like this.</li>

<li><strong>Offline and realtime.</strong> Firestore&#8217;s offline persistence and listener model handle flaky connectivity without you designing for it. If your users are on trains or in warehouses, this is a serious argument.</li>

<li><strong>Emulators.</strong> Being able to run the backend on a laptop, in CI, with no cloud project, is a real productivity difference against Amplify&#8217;s cloud sandbox.</li>

<li><strong>Time to first screen.</strong> For a prototype or an MVP that needs to exist by Friday, Firebase usually wins on speed alone.</li>
</ul>



<h3 class="wp-block-heading">Where Firebase doesn&#8217;t</h3>



<ul class="wp-block-list">
<li><strong>Configuration drifts out of git.</strong> The console is convenient and it is also a second source of truth. Indexes get created from an error link, a provider gets enabled by hand, and six months later nobody can rebuild the project from the repository. Terraform and the Firebase CLI both help, but you have to decide to use them.</li>

<li><strong>Security rules are a language, not a checkbox.</strong> They are also evaluated on the server for every access, which has cost implications when rules perform document lookups.</li>

<li><strong>Features you assume are included are an upgrade.</strong> Multi-factor auth, SAML and OIDC, blocking functions, audit logging and multi-tenancy live behind the Identity Platform upgrade, which comes with a different pricing model. On the free Spark plan, upgrading also introduces a daily active user cap. Find this out during design, not during an enterprise sales cycle.</li>

<li><strong>The ceiling is lower.</strong> When you outgrow Firebase you are usually moving to Google Cloud proper, which is a different set of tools and a different mental model.</li>
</ul>



<h2 class="wp-block-heading">What actually generates the bill</h2>



<p class="wp-block-paragraph">Rates change, so treat published numbers as current-at-time-of-reading and go to the vendor pricing pages for the arithmetic. What is stable, and what you should actually reason about, is the <em>shape</em> of the meters.</p>



<p class="wp-block-paragraph">Firestore charges per document, and the details are where teams get caught:</p>



<ul class="wp-block-list">
<li>A query that returns 200 documents is 200 reads, whether you use two fields from each one or forty.</li>

<li>A realtime listener is charged a read each time a document in its result set is added or updated. Attach a listener to a busy collection and you have built a meter that runs while nobody is looking at the screen.</li>

<li>Where your SDK exposes query offsets, skipped documents are still billed. Paginate with cursors, not offsets.</li>

<li>Aggregations such as <code>count()</code> are billed per batch of index entries read, with a minimum of one document read. Cheap, but not free, and not zero when you run them per page load.</li>

<li>Stored bytes include indexes and metadata, so index sprawl shows up twice: once in write amplification, once in storage.</li>
</ul>



<p class="wp-block-paragraph">Amplify has no single meter, because Amplify is not a service. The bill is assembled from AppSync (query and mutation operations, realtime update messages, and connection-minutes for open WebSockets), DynamoDB, Lambda invocations and duration, CloudWatch log ingestion and retention, Cognito monthly active users, and Amplify Hosting build minutes plus data served.</p>



<p class="wp-block-paragraph">That difference is the practical one. Firebase gives you a bill you can read in the console in thirty seconds. AWS gives you eight line items that will not tell you which feature caused them unless you tagged resources properly on the way in. If you go the AWS route, set up cost allocation tags and a budget alarm before launch, not after the first invoice. Pushing CloudWatch metrics into <a href="https://grafana.com/" target="_blank" rel="noreferrer noopener">Grafana</a> alongside your application metrics is worth the afternoon it takes, because cost anomalies and traffic anomalies are usually the same incident viewed twice.</p>



<h2 class="wp-block-heading">Auth is the component you cannot cheaply replace</h2>



<p class="wp-block-paragraph">Firebase Authentication and Amazon Cognito are both fine. Neither is the reason to pick a platform. But auth is the layer where switching hurts most, because you are not migrating data, you are migrating <em>sessions and identities</em>, and every user notices.</p>



<p class="wp-block-paragraph">Both let you export users, including password hashes, so a migration is technically possible. What is hard is everything hanging off the user ID: rules that compare against it, resolver authorisation that assumes it, third-party integrations keyed on it, and support tooling that looks users up by it.</p>



<p class="wp-block-paragraph">One habit removes most of that pain regardless of which platform you pick: store your own internal user record from the first commit, with your own primary key, and treat the provider&#8217;s UID as an external identifier on that record. It costs you one table and one lookup. It converts an auth migration from a rewrite into a mapping exercise.</p>



<h2 class="wp-block-heading">Hosting is the layer nobody should agonise over</h2>



<p class="wp-block-paragraph">Amplify Hosting deploys from a git branch and supports server-rendered frameworks. Firebase now has two: the original Hosting for static sites, and App Hosting for modern server-rendered apps, which is generally available and runs on Cloud Run with Cloud CDN in front.</p>



<p class="wp-block-paragraph">Both are good. Neither should influence your decision much, because hosting is the one layer with genuine competition and near-zero switching cost. A Next.js or Nuxt app moves to Vercel, Netlify, Cloudflare Pages or a container on a VPS from a provider like <a href="https://www.interserver.net/" target="_blank" rel="noreferrer noopener">InterServer</a> or DigitalOcean in an afternoon. Put a CDN such as <a href="https://www.cloudflare.com/" target="_blank" rel="noreferrer noopener">Cloudflare</a> in front and the origin becomes an implementation detail your users never see.</p>



<p class="wp-block-paragraph">Spend your deliberation budget on the data layer instead. That is where it pays.</p>



<h2 class="wp-block-heading">Exit cost, ranked</h2>



<p class="wp-block-paragraph">If you rank the layers by what it costs to leave, the ordering is the same on both platforms and it is the opposite of the ordering people use when choosing:</p>



<ol class="wp-block-list">
<li><strong>Hosting.</strong> Swap it in a day. Barely counts as lock-in.</li>

<li><strong>Functions.</strong> Cloud Functions and Lambda handlers are mostly your own code with a different signature wrapped around it. A rewrite, but a bounded one.</li>

<li><strong>Auth.</strong> A user migration project with a communications plan attached.</li>

<li><strong>The data model.</strong> You do not migrate this. You rebuild the read path. Firestore&#8217;s managed export produces Firestore-format files in Cloud Storage, and DynamoDB exports to S3, and in both cases what lands is denormalised documents shaped by decisions you made two years ago, not a relational dataset you can point a reporting tool at.</li>
</ol>



<p class="wp-block-paragraph">There is a fifth item with no clean export at all: realtime. Firestore listeners and AppSync subscriptions have no portable equivalent, so if realtime is core to your product, whichever you choose is close to permanent. That is not automatically an argument against either. It is an argument for knowing you are making that commitment when you make it.</p>



<h2 class="wp-block-heading">AWS Amplify vs Firebase: how I&#8217;d actually decide</h2>



<p class="wp-block-paragraph">Not a scorecard. A procedure, in order, stopping at the first clear answer:</p>



<ol class="wp-block-list">
<li><strong>Write down your three hardest read queries.</strong> Not the CRUD. The reporting screen, the search, the admin filter. If you cannot write them, you are not ready to choose a backend.</li>

<li><strong>Do any of them need joins, aggregates or filters you cannot predict?</strong> If yes, start relational: Firebase Data Connect, Amplify over a SQL data source, or plain Postgres behind an API you own. Do not start on documents planning to fix it later.</li>

<li><strong>Is this mobile-first with real offline requirements?</strong> Firebase, and it is not close.</li>

<li><strong>Does your organisation already run on AWS?</strong> Existing accounts, IAM, VPCs, audit posture and a security team who will ask questions about a second cloud all point to Amplify, even if Firebase would be faster this quarter.</li>

<li><strong>Who is maintaining this in two years?</strong> Amplify Gen 2 expects TypeScript fluency and some tolerance for CDK and CloudFormation. Firebase expects less. Match the platform to the team you will actually have.</li>

<li><strong>Any data residency or regulatory constraints?</strong> Check region availability for every service you plan to use, not just the headline one, before you commit.</li>

<li><strong>Still tied?</strong> It is a prototype. Pick the one your team can ship this week, and write down explicitly that the data model is the part you will have to revisit.</li>
</ol>



<h2 class="wp-block-heading">Mistakes I see repeatedly</h2>



<ul class="wp-block-list">
<li><strong>Choosing on SDK ergonomics.</strong> Both SDKs are pleasant. You are picking a data model and a billing model, and the SDK is the layer you would replace most easily.</li>

<li><strong>Building reporting on the transactional store.</strong> Both platforms punish this. Stream to a warehouse or a read replica and let the operational store do one job.</li>

<li><strong>Launching without a budget alarm.</strong> A missing <code>limit()</code>, a listener on a busy collection, or one post that gets traction can turn a hobby bill into a real one overnight. Set the alarm before you need it.</li>

<li><strong>Treating authorisation as a later task.</strong> On both platforms, access control is coupled to the data model. Retrofitting it usually means reshaping documents.</li>

<li><strong>Following Amplify Gen 1 material.</strong> The commands, the directives and the mental model are all different. Check which generation a tutorial targets before you follow it.</li>

<li><strong>Letting console clicks become infrastructure.</strong> Anything you enabled by hand is something you cannot rebuild. Get it into the CLI config or into Terraform.</li>

<li><strong>Assuming the free tier is the plan.</strong> It is a trial of the plan. Model your costs at ten times current traffic and see whether you still like the answer.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is AWS Amplify or Firebase cheaper?</h3>



<p class="wp-block-paragraph">It depends entirely on your access patterns, and anyone who answers this without seeing them is guessing. Firebase tends to be cheaper and far more predictable at small scale, partly because the free allowances are generous and partly because there is one meter to watch. AWS tends to win when you have steady, high-volume traffic and the engineering discipline to tune DynamoDB capacity and Lambda sizing. The bigger practical difference is legibility: a Firebase bill tells you what happened, an AWS bill tells you what was consumed.</p>



<h3 class="wp-block-heading">Can I move from Firebase to AWS Amplify later?</h3>



<p class="wp-block-paragraph">Yes, but price it by layer rather than as one project. Hosting and functions move cheaply. Auth is a real migration with user impact. The data model is usually a rebuild rather than a move, because a Firestore export gives you denormalised documents, not a relational dataset. Teams that manage this well are the ones that kept business logic out of security rules and out of client code from the beginning.</p>



<h3 class="wp-block-heading">Is Amplify Gen 1 still supported?</h3>



<p class="wp-block-paragraph">Gen 1 is in maintenance mode and reaches end of life on 1 May 2027. It receives critical fixes and security patches, not new features. If you are starting something new, use Gen 2. If you have a Gen 1 application in production, schedule the migration deliberately using the official tooling and guide, and test thoroughly before the refactor step, which is difficult to reverse.</p>



<h3 class="wp-block-heading">Do I need Firebase Data Connect, or is Firestore enough?</h3>



<p class="wp-block-paragraph">Firestore is enough when your reads are predictable, mostly key-based, and denormalise cleanly. The moment you need joins, aggregates across collections, or filters that product will keep changing, Data Connect&#8217;s managed Postgres is the better foundation. It is also a lot easier to adopt at the start than to migrate onto after a year of document design.</p>



<h3 class="wp-block-heading">Which one is better for a mobile app?</h3>



<p class="wp-block-paragraph">Firebase, in most cases. The offline persistence model, the crash reporting, the push messaging and the remote configuration are all mature and designed to work together. Amplify has mobile libraries and they work, but you are assembling the surrounding pieces from separate AWS services rather than getting them as a set.</p>



<h3 class="wp-block-heading">What about a self-hosted alternative?</h3>



<p class="wp-block-paragraph">Worth considering if lock-in is your main worry. Supabase and Appwrite both offer a comparable developer experience over Postgres, and either can run on your own infrastructure. Understand what you are taking on, though: backups, restore testing, upgrades, patching and monitoring all become yours. That is a real operational load, and it is the trade most teams are implicitly paying a managed vendor to avoid.</p>



<h3 class="wp-block-heading">Can I use both together?</h3>



<p class="wp-block-paragraph">You can, and some teams do run Firebase for analytics and messaging alongside an AWS backend. Be honest about the cost: two identity models, two billing accounts, two audit trails and two on-call runbooks. Do it because one platform is clearly better at a specific job, not because it was easier than making a decision.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The <strong>AWS Amplify vs Firebase</strong> question is almost never settled by features, and it is never settled by which SDK feels nicer in a tutorial. Both platforms will get you to a working application quickly, and both will hold up in production for the workloads they were designed for.</p>



<p class="wp-block-paragraph">What separates them, months later, is the read path you committed to on day one and the layer you cannot afford to leave. Write down your three hardest queries before you write any schema. Decide whether you need relational storage while it is still a decision rather than a migration. Keep your own user record from the first commit. Set a budget alarm before launch.</p>



<p class="wp-block-paragraph">Do those four things and either platform is a reasonable choice. Skip them and neither one saves you.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second opinion before you commit?</h2>



<p class="wp-block-paragraph">Backend platform decisions are cheap to review and expensive to reverse. Things I help with in this area:</p>



<ul class="wp-block-list">
<li>Reviewing a Firestore or DynamoDB data model against the queries you actually need, before the design hardens</li>

<li>Cost modelling both platforms against your real traffic assumptions, with the meters broken out so you can see what drives the bill</li>

<li>Amplify Gen 1 to Gen 2 migration planning, including the blue/green sequence and what to verify before the irreversible step</li>

<li>Auditing Firestore security rules and Amplify authorisation rules for gaps between the read path and the write path</li>

<li>Setting up cost allocation tags, budget alarms and CloudWatch or Grafana dashboards so a runaway query shows up in hours rather than on the invoice</li>

<li>Exit planning: working out what a migration off your current platform would genuinely cost, layer by layer</li>
</ul>



<p class="wp-block-paragraph">If you want a concrete answer rather than a general one, send me the thing itself: your schema file, your security rules, a screenshot of last month&#8217;s billing breakdown, or the three queries you are worried about. That is usually enough to tell you something useful.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-amplify-vs-firebase/">AWS Amplify vs Firebase: Choosing a Backend You Can Still Leave</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Apache Airflow on AWS: Building SaaS and API Pipelines That Don&#8217;t Lie to You</title>
		<link>https://john-nessime.com/blog/devops/apache-airflow-aws-saas-api-pipelines/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 05 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Airflow]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[MWAA]]></category>
		<category><![CDATA[Orchestration]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=145</guid>

					<description><![CDATA[<p>Most API pipeline failures are green DAGs producing incomplete data. A practical guide to running Apache Airflow on AWS for SaaS and API extraction: choosing between MWAA provisioned, MWAA Serverless and self-managed, the pool setting that silently stops throttling when you go deferrable, retry and pagination design, secrets handling, and the four cost lines that actually move.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/apache-airflow-aws-saas-api-pipelines/">Apache Airflow on AWS: Building SaaS and API Pipelines That Don&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually lands on a Monday: &#8220;the CRM numbers look wrong again.&#8221; Not missing. Wrong. The dashboard populated, every DAG run is green, and somewhere in the middle of last week&#8217;s data there is a hole where a paginated API returned a 429 and the task treated the empty body as a legitimate final page.</p>



<p class="wp-block-paragraph">That is the shape of most API pipeline incidents. Not a crash. A success that isn&#8217;t one.</p>



<p class="wp-block-paragraph">This post covers running Apache Airflow on AWS specifically for SaaS and API workloads: pulling from HubSpot, Salesforce, Stripe, Zendesk, Shopify, an internal partner API, whatever. It is organised by failure family rather than by feature, because the Airflow documentation already explains what an operator is and does a poor job of explaining which of these things will page you at 3am. I will cover choosing a deployment model, the concurrency trap that catches almost everyone, retry design, incremental state, secrets, and where the money actually goes.</p>



<h2 class="wp-block-heading">Why SaaS and API sources break differently</h2>



<p class="wp-block-paragraph">When your source is a database you control, failure is loud: connection refused, deadlock, disk full. When it is somebody else&#8217;s SaaS API, three things change.</p>



<ul class="wp-block-list">
<li><strong>You are a guest.</strong> The vendor decides your rate limit, and they can change it without telling you. Your pipeline&#8217;s correctness now depends on a number in someone else&#8217;s config file.</li>

<li><strong>Errors arrive as valid HTTP.</strong> A 429, a 200 with a truncated page, a 200 with an error object in the body. Your HTTP client is happy. Your data is not.</li>

<li><strong>Tasks spend most of their life waiting.</strong> API extraction is I/O bound almost end to end. That sounds harmless and is the root of the most expensive mistakes.</li>
</ul>



<h2 class="wp-block-heading">Pick the deployment model before you write a DAG</h2>



<p class="wp-block-paragraph">This decision constrains everything after it and is harder to reverse than people expect. Three realistic options.</p>



<h3 class="wp-block-heading">Amazon MWAA, provisioned</h3>



<p class="wp-block-paragraph">AWS runs the scheduler, web server, workers, triggerer and metadata database on Fargate; you drop DAGs into an S3 bucket and they get picked up.</p>



<p class="wp-block-paragraph">Where it wins: real Airflow, custom providers, custom plugins, full control over environment configuration. If your DAGs need arbitrary Python libraries, this option will not fight you.</p>



<p class="wp-block-paragraph">Where it doesn&#8217;t: the environment bills by the hour whether or not anything is running. There is no scale to zero on the base environment. If you sync six APIs once a day and each run takes twenty minutes, you are paying for a mostly idle cluster around the clock. The <code>mw1.micro</code> class exists precisely for the small case, but it collapses the scheduler and worker into a single Fargate task and caps worker autoscale low, so treat it as a dev or isolation tier rather than a cheap production tier.</p>



<h3 class="wp-block-heading">Amazon MWAA Serverless</h3>



<p class="wp-block-paragraph">You submit workflow definitions and AWS runs each task in its own Fargate container, billing per task duration with a one-minute minimum rather than per environment hour.</p>



<p class="wp-block-paragraph">Where it wins: spiky or infrequent schedules. If the workload is &#8220;six syncs a day, nothing overnight,&#8221; the cost profile beats a permanently running environment by a wide margin. Each workflow also gets its own IAM execution role, which is a real security improvement over one shared role per environment.</p>



<p class="wp-block-paragraph">Where it doesn&#8217;t: it leans on declarative YAML workflow definitions based on the DAG Factory format and a curated set of AWS operators. That is a deliberate trade: because the definition is declarative, the service can schedule tasks without executing your DAG code. It also means custom operators, exotic third-party providers and clever Python at parse time are not the sweet spot. It is also available in fewer regions than provisioned MWAA, so check your region before you design around it.</p>



<h3 class="wp-block-heading">Self-managed on ECS, EKS or a VPS</h3>



<p class="wp-block-paragraph">On EKS with the Kubernetes executor you get per-task pods and tight cost control. On a single VPS from a provider like InterServer or Hetzner, a Docker Compose stack with a Postgres metadata database will run a modest set of API syncs for a fraction of any managed price.</p>



<p class="wp-block-paragraph">Where it wins: cost at both extremes, and total control. Where it doesn&#8217;t: you now own metadata database upgrades, major version migrations, log retention and the 2am scheduler restart. Managed Airflow is a bet that your time is worth more than the hourly premium. For a solo engineer with three pipelines that bet often loses; for a data team of eight it usually wins. Astronomer is the main non-AWS managed option worth pricing alongside these.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: the throttle that silently stops throttling</h2>



<p class="wp-block-paragraph">You start with a normal setup: an Airflow pool named <code>crm_api</code> with four slots, and every task that touches the vendor assigned to it. Four concurrent requests, comfortably under the vendor&#8217;s limit. This works.</p>



<p class="wp-block-paragraph">Then you notice those tasks spend nearly all their runtime waiting on HTTP, burning worker slots to sit still. So you switch them to deferrable operators. A deferrable task suspends itself while waiting, releases its worker slot, and hands the waiting to the triggerer, which polls asynchronously. Worker pressure drops. Everything looks better.</p>



<p class="wp-block-paragraph">And your rate limiting quietly stops working.</p>



<p class="wp-block-paragraph">By default, a pool does not count tasks in the <em>deferred</em> state as occupying slots. That was deliberate, and the logic is sound in the abstract: a deferred task is not consuming a worker. But if you were using the pool to protect an external API rather than your own workers, it has just stopped doing the job you gave it. Every task can defer at once, and the vendor sees the full fan-out.</p>



<p class="wp-block-paragraph">The fix is a per-pool flag, <code>include_deferred</code>, which tells the scheduler to count deferred tasks against the slot budget. It is off by default. You can set it when editing the pool in the Airflow UI, or through the API.</p>



<p class="wp-block-paragraph">The failure signature is what makes this nasty. Nothing errors. Your DAG gets faster. The vendor starts returning 429s that your retry logic absorbs, and the only symptom is that runs take a little longer and occasionally a page goes missing. Weeks can pass. Two related traps in the same family:</p>



<ul class="wp-block-list">
<li><code>max_active_tasks</code> at the DAG level has the same blind spot with deferred tasks, and there is no equivalent opt-in flag. If you need a hard external concurrency cap, use a pool with <code>include_deferred</code> enabled, not DAG-level concurrency.</li>

<li>On MWAA, the triggerer runs alongside the scheduler on the same Fargate task, so scheduler count and triggerer capacity are linked. If you go heavily deferrable and your deferred tasks start stalling, scheduler capacity is the thing to look at.</li>
</ul>



<h2 class="wp-block-heading">Failure family two: retries that make the outage worse</h2>



<p class="wp-block-paragraph">The default instinct is to set <code>retries</code> high and move on. Against a rate-limited API, a fixed retry delay across many parallel tasks is just a slower version of the same stampede.</p>



<p class="wp-block-paragraph">What you want is exponential backoff with a ceiling. The shape:</p>



<pre class="wp-block-code"><code>from datetime import timedelta

from airflow.sdk import dag, task

@dag(
    schedule="0 5 * * *",
    catchup=False,
    max_active_runs=1,          # never let two runs of this DAG overlap
    default_args={
        "retries": 5,
        "retry_delay": timedelta(seconds=30),
        "retry_exponential_backoff": True,   # 30s, 60s, 120s, 240s...
        "max_retry_delay": timedelta(minutes=15),  # stop doubling here
        "pool": "crm_api",      # shared budget across every task touching this vendor
    },
    tags=["crm", "extract"],
)
def crm_extract():

    @task(max_active_tis_per_dag=4)
    def fetch_page(page_token: str) -&gt; str:
        ...

crm_extract()</code></pre>



<p class="wp-block-paragraph">The lines that matter:</p>



<ul class="wp-block-list">
<li><code>retry_exponential_backoff</code> turns <code>retry_delay</code> into a base rather than a constant, so repeated failures spread out instead of hammering in lockstep.</li>

<li><code>max_retry_delay</code> caps the doubling. Without it, a task that fails five times can sit idle for hours and blow past the window you actually cared about.</li>

<li><code>max_active_runs=1</code> is the one people skip. If a run overruns its schedule, the next one starts anyway, and now two runs are fetching the same pages from the same vendor with the same credentials. This is a common way to trigger a rate limit you have never hit before.</li>

<li><code>max_active_tis_per_dag</code> limits how many instances of that specific task run concurrently across DAG runs, which is the right knob for dynamically mapped extraction tasks.</li>
</ul>



<p class="wp-block-paragraph">One thing Airflow will not do for you: honour a <code>Retry-After</code> header. Airflow&#8217;s retry timing is computed from your config, not from the vendor&#8217;s response. If the API tells you exactly how long to wait, you have to catch that in your own code and sleep or reschedule accordingly. Ignoring a header the vendor bothered to send is a good way to get your API key throttled harder.</p>



<h2 class="wp-block-heading">Failure family three: pagination, cursors and the empty page</h2>



<p class="wp-block-paragraph">Back to the Monday message. The specific bug behind most &#8220;the numbers are wrong but nothing failed&#8221; incidents is a loop that treats any non-error response as a terminating condition. Three rules prevent it:</p>



<ol class="wp-block-list">
<li><strong>Never infer &#8220;done&#8221; from an empty result.</strong> Terminate on the explicit signal the API gives you: a null <code>next_cursor</code>, a missing <code>Link</code> header, a page count. An empty array with a valid cursor still has more data behind it.</li>

<li><strong>Assert the response shape before you use it.</strong> Check the status code explicitly and validate that the fields you depend on exist. A 200 carrying <code>{"error": "..."}</code> should raise, not return zero rows.</li>

<li><strong>Land raw, transform later.</strong> Write the untouched API response to S3 first, then parse from S3. When the vendor changes a field type, you can replay from raw instead of re-extracting from an API that no longer serves that window.</li>
</ol>



<h3 class="wp-block-heading">Where to keep incremental state</h3>



<p class="wp-block-paragraph">The tempting pattern is to store the last-seen timestamp in an Airflow Variable and update it at the end of a run. Do not make that your source of truth. If a run dies midway, the Variable is in an undefined state, and clearing and re-running the DAG will not restore it. Airflow&#8217;s retry and backfill machinery has no idea it exists.</p>



<p class="wp-block-paragraph">Better: make each run&#8217;s window a function of the run itself, and write output to a deterministic, run-scoped location such as <code>s3://bucket/source=crm/dt=&lt;logical-date&gt;/</code>. Re-running the same interval overwrites the same prefix. That is what makes a task idempotent, and idempotency is the difference between &#8220;clear the task and let it rerun&#8221; and a two-hour manual repair.</p>



<p class="wp-block-paragraph">Then overlap your windows deliberately. Many SaaS APIs order results by <em>modified</em> time with eventual consistency, so a record edited at the boundary can appear after you have already moved on. Query a window slightly wider than your schedule interval and rely on an idempotent upsert downstream to absorb the duplicates. Late-arriving data is not an edge case with SaaS sources. It is the normal case.</p>



<p class="wp-block-paragraph">Airflow&#8217;s asset-based scheduling is the clean way to trigger downstream DAGs from this: the extract DAG produces an asset, and the transform DAG runs when the asset updates, rather than being scheduled at a time you hope is late enough.</p>



<h2 class="wp-block-heading">Failure family four: credentials</h2>



<p class="wp-block-paragraph">API tokens rotate, sometimes on the vendor&#8217;s schedule rather than yours. Storing an API key in an Airflow Connection through the UI works, and is the wrong long-term answer: the value lives in the metadata database and there is no rotation story. On AWS, point Airflow&#8217;s secrets backend at AWS Secrets Manager. On MWAA that is an environment configuration option:</p>



<pre class="wp-block-code"><code>secrets.backend
  airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend

secrets.backend_kwargs
  {"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables"}</code></pre>



<p class="wp-block-paragraph">With that in place, a connection lookup for <code>crm_default</code> resolves against the secret at <code>airflow/connections/crm_default</code>. Rotating the credential is a Secrets Manager operation with no Airflow deployment involved.</p>



<p class="wp-block-paragraph">Two things to know before you turn it on. First, every connection and variable lookup becomes a Secrets Manager API call, and lookups fall through to the backend before hitting the metadata database, so a DAG that reads a Variable at parse time will generate a call on every parse cycle. Move those reads inside tasks. Second, the environment&#8217;s execution role needs explicit read permission on the relevant secret ARNs, and if you use a customer-managed KMS key, decrypt permission on that key too.</p>



<p class="wp-block-paragraph">Worth knowing if you are on Airflow 3: task code can no longer reach the metadata database directly. All runtime interaction goes through the Task Execution API. If you inherited custom operators that open a session and query Airflow&#8217;s own tables, that is a migration blocker, not a warning.</p>



<h2 class="wp-block-heading">Failure family five: the bill</h2>



<p class="wp-block-paragraph">Nobody is surprised by the environment line item. They are surprised by the other four.</p>



<ul class="wp-block-list">
<li><strong>Idle time.</strong> A provisioned MWAA environment bills continuously. Compute the ratio of hours billed to hours doing work. If it is bad, that is the argument for MWAA Serverless or for consolidating several thin pipelines into one environment.</li>

<li><strong>NAT Gateway.</strong> This is the classic one. Private-subnet workers calling public SaaS APIs route through a NAT Gateway, which charges hourly <em>and</em> per gigabyte processed. A high-volume extraction pipeline can spend more on NAT than on Airflow. VPC endpoints remove that cost for AWS service traffic, but they do nothing for calls to a third-party API, which is exactly the traffic an API pipeline generates.</li>

<li><strong>CloudWatch Logs.</strong> Task logs go to CloudWatch, and ingestion is billed per gigabyte. Set the Airflow log level per component rather than globally at DEBUG, and set a retention policy on the log groups. The default is to keep logs forever.</li>

<li><strong>S3 requests.</strong> Landing raw API responses one small object per page generates a lot of PUTs. Batch pages into larger objects where you can.</li>
</ul>



<p class="wp-block-paragraph">Rates and dimensions change, so model your own workload against the current pricing page rather than trusting a number from a blog post. The point is knowing which four lines to look at.</p>



<h2 class="wp-block-heading">Troubleshooting Apache Airflow on AWS when API pipelines misbehave</h2>



<h3 class="wp-block-heading">Tasks sit in &#8220;queued&#8221; and never start</h3>



<p class="wp-block-paragraph">Usually a slot problem, not a broken scheduler. Check, in order: is the pool full; has DAG-level <code>max_active_tasks</code> been hit; is worker autoscaling at its configured maximum. On MWAA, the container and queue utilisation metrics published to CloudWatch tell you which of the three it is far faster than reading scheduler logs.</p>



<h3 class="wp-block-heading">DAG file changes don&#8217;t appear</h3>



<p class="wp-block-paragraph">On MWAA, DAGs sync from S3 on an interval; it is not instant. If a file has been there for several minutes and still hasn&#8217;t appeared, it almost always failed to parse. Check the DAG processing logs in CloudWatch rather than the scheduler logs, because a broken import raises there and never reaches the scheduler.</p>



<h3 class="wp-block-heading">A new provider package won&#8217;t install</h3>



<p class="wp-block-paragraph">MWAA installs from your <code>requirements.txt</code> in the DAGs bucket, and from Airflow 2.7.2 onward that file must include a constraint line. Without one, MWAA picks a constraint for you, and pip is free to resolve a provider version that conflicts with the Airflow build in the image.</p>



<pre class="wp-block-code"><code>--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-&lt;AIRFLOW_VERSION&gt;/constraints-&lt;PYTHON_VERSION&gt;.txt"

apache-airflow-providers-http
apache-airflow-providers-salesforce</code></pre>



<p class="wp-block-paragraph">Substitute the literal Airflow version your environment runs and the Python version bundled with it. MWAA does not expand shell variables in that file. Test the requirements file against a local Airflow image before you upload it, because a failed install on MWAA surfaces as a partially working environment rather than a clean error.</p>



<h3 class="wp-block-heading">A backfill is stuck and you need to clear it</h3>



<p class="wp-block-paragraph">You do not need a web login token for this. MWAA exposes the Airflow REST API through a signed AWS API call, so you can drive it from CI or a runbook with normal IAM credentials:</p>



<pre class="wp-block-code"><code>aws mwaa invoke-rest-api 
  --name MyMWAAEnvironment 
  --path "/dags/crm_extract/clearTaskInstances" 
  --method POST 
  --body '{"dry_run": true}'</code></pre>



<p class="wp-block-paragraph">Start with <code>dry_run</code> set to true so the response tells you which task instances would be cleared before you actually clear them. Note that the resource paths differ between Airflow 2 and Airflow 3 environments, so confirm against the API version your environment exposes.</p>



<h3 class="wp-block-heading">Deferred tasks stall forever</h3>



<p class="wp-block-paragraph">If deferred tasks stop resuming while the environment reports healthy, suspect the triggerer rather than your DAG. A triggerer that has lost its ability to process triggers can keep heartbeating normally, so the scheduler sees nothing wrong while every deferred task drifts toward timeout. This class of bug has been fixed and re-fixed upstream, so check your Airflow version&#8217;s release notes before assuming it is your code.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Switching to deferrable operators without enabling <code>include_deferred</code> on the pools that were protecting the API.</li>

<li>Leaving <code>max_active_runs</code> unset, so a slow run and the next scheduled run compete for the same rate limit budget.</li>

<li>Treating an empty response page as the end of pagination.</li>

<li>Storing the incremental watermark in an Airflow Variable and updating it mid-run.</li>

<li>Calling an API or reading a Variable at DAG parse time, which executes on every parse cycle rather than once per run.</li>

<li>Transforming during extraction, so a vendor schema change means re-pulling data the API may no longer serve.</li>

<li>Sizing the environment for peak concurrency when the actual constraint is the vendor&#8217;s rate limit.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>One pool per vendor, sized to their published limit with headroom, and <code>include_deferred</code> enabled on every one of them.</li>

<li>Land raw responses to S3 before parsing. Extraction and transformation are separate tasks with separate failure modes.</li>

<li>Make every task idempotent and window-scoped, so &#8220;clear and rerun&#8221; is always a safe repair.</li>

<li>Overlap extraction windows and deduplicate downstream rather than trusting a vendor&#8217;s timestamps to be exact.</li>

<li>Secrets Manager for credentials, with the execution role scoped to specific secret ARNs.</li>

<li>Alert on row counts and freshness, not just task state. A green DAG that produced 40% of yesterday&#8217;s rows is the failure you actually care about. Shipping Airflow&#8217;s StatsD metrics into Prometheus, Grafana Cloud or Datadog makes that a dashboard rather than a discovery.</li>

<li>Define the environment in Terraform or OpenTofu. Recreating an MWAA environment by hand after a bad configuration change is a bad afternoon.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is MWAA worth it compared to self-hosting Airflow on EC2?</h3>



<p class="wp-block-paragraph">It depends almost entirely on how many people share the platform. MWAA&#8217;s premium buys you managed metadata database upgrades, patched images and version migration support. If one engineer maintains three DAGs, self-hosting on a modest VPS is cheaper and the operational load is real but small. Once several teams depend on the scheduler being up, the premium is easy to justify.</p>



<h3 class="wp-block-heading">Should I use Step Functions instead of Airflow for API pipelines?</h3>



<p class="wp-block-paragraph">Step Functions is genuinely better for event-driven, AWS-service-centric orchestration with modest branching, and it scales to zero. Airflow wins when you need scheduled batch semantics, backfills over historical windows, dependencies between many pipelines, and a UI that non-platform engineers can use to see why last Tuesday failed. Backfill is usually the deciding feature.</p>



<h3 class="wp-block-heading">Do deferrable operators reduce my AWS bill?</h3>



<p class="wp-block-paragraph">On provisioned MWAA, they reduce worker <em>pressure</em>, which reduces autoscaling into additional worker instances. The base environment cost is unchanged. On a Kubernetes executor setup where each task is a pod, the saving is more direct. Either way, do not adopt them purely for cost without revisiting your pool configuration first.</p>



<h3 class="wp-block-heading">How do I handle a vendor with no documented rate limit?</h3>



<p class="wp-block-paragraph">Start conservative, one or two concurrent requests, and instrument the response status codes. Raise the pool size gradually and watch for 429s or rising latency. Latency creeping up under load is often the earlier signal, because some vendors throttle by slowing you down before they start rejecting.</p>



<h3 class="wp-block-heading">Can Airflow read a Retry-After header automatically?</h3>



<p class="wp-block-paragraph">No. Airflow computes retry timing from <code>retry_delay</code> and the backoff settings on the task. If a vendor sends <code>Retry-After</code>, you need to handle it in your own request code or in a custom operator.</p>



<h3 class="wp-block-heading">What breaks when upgrading to Airflow 3?</h3>



<p class="wp-block-paragraph">The big one for API pipelines is that task code can no longer access the metadata database directly; everything goes through the Task Execution API. Imports also move to the <code>airflow.sdk</code> namespace, and several core operators now live in the standard provider package. Audit custom operators first, since that is where direct database access hides. MWAA requires you to be on the latest Airflow 2 minor version before a major upgrade, so plan two steps.</p>



<h3 class="wp-block-heading">How many DAGs can one MWAA environment handle?</h3>



<p class="wp-block-paragraph">The binding constraint is usually the metadata database and scheduler CPU, not DAG count. Watch metadata database memory and scheduler CPU utilisation; when either saturates, you either move up an environment class or split into multiple environments. Splitting also gives you blast-radius isolation, which matters more than people expect.</p>



<h2 class="wp-block-heading">Wrapping up</h2>



<p class="wp-block-paragraph">Running Apache Airflow on AWS for SaaS and API pipelines is mostly not an Airflow problem. The scheduler works. The operators work. What bites is the gap between &#8220;the task succeeded&#8221; and &#8220;the data is correct,&#8221; and that gap lives in concurrency settings, pagination logic and retry design rather than anywhere Airflow will warn you about.</p>



<p class="wp-block-paragraph">If you take one thing away: <strong>a green DAG is not a signal that your data is complete.</strong> Enable <code>include_deferred</code> on the pools protecting your vendors, terminate pagination on an explicit signal instead of an empty page, make every task idempotent, and alert on row counts. Those four things prevent most of the incidents that never show up as a failed task.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with your Airflow pipelines on AWS?</h2>



<p class="wp-block-paragraph">I work with teams running data and API pipelines on AWS, usually somewhere between &#8220;it works but nobody trusts it&#8221; and &#8220;we need to move off cron.&#8221; Things I can help with:</p>



<ul class="wp-block-list">
<li>Reviewing existing DAGs for silent data loss: pagination logic, retry behaviour, pool and concurrency configuration.</li>

<li>Choosing between MWAA provisioned, MWAA Serverless and self-managed Airflow, with a cost model for your actual schedule rather than a generic comparison.</li>

<li>Building SaaS extraction pipelines that are idempotent and safely re-runnable, landing raw to S3 with incremental windows that survive failure.</li>

<li>Cutting MWAA cost: environment right-sizing, NAT Gateway traffic, CloudWatch log volume and dependency install time.</li>

<li>Airflow 2 to 3 migration audits, focused on custom operators and direct metadata database access.</li>

<li>Data freshness and volume alerting in Grafana or CloudWatch, so you learn about a partial sync before the business does.</li>
</ul>



<p class="wp-block-paragraph">If something specific is broken, send me the DAG file, the task log, or the CloudWatch metrics for the run that went wrong. It is usually faster to look at the real thing than to describe it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/apache-airflow-aws-saas-api-pipelines/">Apache Airflow on AWS: Building SaaS and API Pipelines That Don&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
