<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>AWS Lambda | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/aws-lambda/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/aws-lambda/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Tue, 03 Nov 2026 07:06:02 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>AWS Lambda | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/aws-lambda/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>AWS Secrets Manager vs Parameter Store: Stop Deciding on Price</title>
		<link>https://john-nessime.com/blog/devops/aws-secrets-manager-vs-parameter-store/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 03 Nov 2026 07:06:02 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Amazon ECS]]></category>
		<category><![CDATA[Amazon RDS]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[AWS Secrets Manager]]></category>
		<category><![CDATA[CloudTrail]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[HashiCorp Vault]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[Infrastructure as Code]]></category>
		<category><![CDATA[Key Rotation]]></category>
		<category><![CDATA[Least Privilege]]></category>
		<category><![CDATA[Parameter Store]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Terraform]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=442</guid>

					<description><![CDATA[<p>Most teams pick between AWS Secrets Manager and SSM Parameter Store on price, then get bitten months later by rotation semantics, a shared throughput quota, or a recovery window that blocks a redeploy. Here are the four levers that actually decide it, per value, plus a decision procedure you can run against a real inventory.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-secrets-manager-vs-parameter-store/">AWS Secrets Manager vs Parameter Store: Stop Deciding on Price</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The pull request comment was one line: why is the database password in Parameter Store and the Stripe key in Secrets Manager?</p>



<p class="wp-block-paragraph">Nobody had a good answer. Both were added in a hurry, months apart, by different people, and each choice made sense on the day it was made. That is how most AWS accounts end up with secrets in two places. Not by design, by accumulation.</p>



<p class="wp-block-paragraph">The comparison usually gets framed as a pricing question: Parameter Store&#8217;s standard tier has no storage charge, Secrets Manager bills per secret, so use Parameter Store until somebody complains. That framing is wrong, and it stays invisible for months before it costs you anything. What separates these two services is what happens when a value <em>changes</em>, not what it costs to sit there.</p>



<p class="wp-block-paragraph">This post covers the failure modes that actually bite, the four levers that decide <strong>AWS Secrets Manager vs Parameter Store</strong> for any given value, a decision procedure you can run against a real inventory, and the arguments about this that do not survive contact with a running system.</p>



<h2 class="wp-block-heading">The two failure modes that actually bite</h2>



<h3 class="wp-block-heading">Rotation that succeeds while your application quietly breaks</h3>



<p class="wp-block-paragraph">Secrets Manager&#8217;s single user rotation strategy changes the password on the existing database user. AWS documents a short window between the password changing in the database and the secret being updated, during which calls using the rotated credentials can be denied. Retry logic covers that window.</p>



<p class="wp-block-paragraph">That is the documented gap. The one that catches people is quieter.</p>



<p class="wp-block-paragraph">Your application fetched the credential once at boot, cached it in a module-level variable, and never looked again. Rotation completes. The rotation function logs success. Nothing breaks. Then a container recycles, picks up the new value and works fine, while older containers keep running on connections opened before the change. It looks healthy right up until a connection pool churns and part of your fleet starts failing authentication against a service you have not deployed in weeks.</p>



<p class="wp-block-paragraph">The alternating users strategy exists for exactly this. Instead of changing one user&#8217;s password, Secrets Manager clones the user and alternates which one it updates, so a valid credential exists on both sides of every change. The price is a second database user and a separate superuser secret with permission to do the cloning. AWS recommends single user for most cases and for interactive or one-off credentials, and alternating users where the application needs high availability.</p>



<p class="wp-block-paragraph">The rule I use: if consumers cache and you rotate on a schedule, you need alternating users or an explicit refresh path. A shorter cache TTL narrows the window, it does not close it.</p>



<h3 class="wp-block-heading">A throughput ceiling you do not own</h3>



<p class="wp-block-paragraph">Parameter Store read throughput is a quota on the account and Region, not on the parameter and not on the caller. Every cold start, every task boot, every CI job in that account draws from the same budget. The default is sized for low-volume workloads, and there is an opt-in higher throughput setting that raises the ceiling for an additional charge.</p>



<p class="wp-block-paragraph">The failure presents as an application bug. A scale-out event or a chatty pipeline fans out, <code>GetParameter</code> starts returning <code>ThrottlingException</code>, and services fail at boot for reasons unrelated to the code anybody just shipped. Nothing in the console tells you that you are near the line.</p>



<p class="wp-block-paragraph">Both services have quotas. The difference is blast radius: Secrets Manager throttling is annoying, Parameter Store throttling is a shared resource problem where one team&#8217;s new batch job degrades another team&#8217;s deploys. Alarm on throttling for both before you need to. CloudWatch does it natively, and if your dashboards live in Grafana Cloud or a self-hosted Grafana, pull the metric across so the alert lands where people actually look.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Where Parameter Store wins, and where it hurts</h2>



<p class="wp-block-paragraph">Parameter Store is not primarily a secrets store. It is a hierarchical configuration store that encrypts when you ask it to. Hold that framing and its behaviour becomes predictable.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong></p>



<ul class="wp-block-list">
<li><strong>Configuration that is not confidential.</strong> AMI IDs, feature flags, queue names, tuning values. Putting these in Secrets Manager means paying a per-secret charge for data with no confidentiality requirement.</li>



<li><strong>Hierarchies and bulk reads.</strong> Path-based naming plus <code>GetParametersByPath</code> pulls a whole subtree in one call, which is what a service wants at startup.</li>



<li><strong>No storage charge on the standard tier,</strong> so nobody is incentivized to cram unrelated values together to save money.</li>



<li><strong>Parameter policies and AWS public parameters</strong> live here, including read-only values AWS publishes for things like current AMI IDs.</li>
</ul>



<p class="wp-block-paragraph"><strong>Where it hurts:</strong></p>



<ul class="wp-block-list">
<li><strong>No rotation.</strong> You write the function, schedule it, and own verification and rollback yourself.</li>



<li><strong>CloudFormation cannot create an encrypted parameter.</strong> The <code>AWS::SSM::Parameter</code> resource accepts <code>String</code> and <code>StringList</code> only. You can reference an existing encrypted parameter through the <code>ssm-secure</code> dynamic reference, and only on the resource properties that support that pattern.</li>



<li><strong>No recovery window.</strong> Delete is delete, and parameter history does not survive deleting the parameter.</li>



<li><strong>Cross-account sharing requires the advanced tier,</strong> through Resource Access Manager.</li>
</ul>



<h2 class="wp-block-heading">Where AWS Secrets Manager wins, and where it hurts</h2>



<p class="wp-block-paragraph">Secrets Manager is a lifecycle service. Storage is the least interesting thing it does.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong></p>



<ul class="wp-block-list">
<li><strong>Managed rotation</strong> for Amazon RDS, Aurora, Redshift and DocumentDB, using AWS-published rotation functions and two documented strategies.</li>



<li><strong>Versioning with staging labels.</strong> <code>AWSCURRENT</code>, <code>AWSPENDING</code> and <code>AWSPREVIOUS</code> give rotation a defined state machine and give you a last known good version to fall back to.</li>



<li><strong>A recovery window on delete,</strong> between seven and thirty days, defaulting to thirty.</li>



<li><strong>Cross-region replication and resource policies</strong> as first-class features, so multi-Region and cross-account access are properties of the secret rather than things you build.</li>
</ul>



<p class="wp-block-paragraph"><strong>Where it hurts:</strong></p>



<ul class="wp-block-list">
<li><strong>Cost scales by multiplication.</strong> Storage bills per secret per month, prorated hourly, and every cross-region replica bills as its own secret. The multiplier is secrets times environments times Regions.</li>



<li><strong>The recovery window is a deployment trap.</strong> Destroy an environment, apply again, and the create fails because the name is still reserved.</li>



<li><strong>Managed rotation only covers databases.</strong> Anything else means writing your own function, so you carry the operational burden without the free tier.</li>



<li><strong>The pricing model pushes you toward bundling.</strong> One secret per value is the clean design and the expensive one, so teams pack fields into JSON and lose per-value IAM granularity.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Four levers that decide AWS Secrets Manager vs Parameter Store</h2>



<h3 class="wp-block-heading">1. Does the value have to change on a schedule?</h3>



<p class="wp-block-paragraph">If yes, and it is a credential for RDS, Aurora, Redshift or DocumentDB, use Secrets Manager and stop deliberating. You are not paying for storage. You are paying to avoid writing, testing and maintaining a rotation function, and to inherit a staging-label state machine that makes a failed rotation recoverable instead of an outage.</p>



<p class="wp-block-paragraph">If the value never changes on a schedule, an AMI ID, a webhook URL, a vendor key you replace manually once a year, then rotation machinery buys you nothing.</p>



<h3 class="wp-block-heading">2. How many independent gates guard the read path?</h3>



<p class="wp-block-paragraph">This is the lever people skip, and it matters more than the service choice.</p>



<p class="wp-block-paragraph">Both services encrypt through KMS. The question is whether KMS is a second, independent authorization decision or a formality. An encrypted parameter under the AWS managed key for Systems Manager is decrypted through that key on behalf of the service, so in practice the SSM permission is the gate. Encrypt under a customer managed key instead and the caller must also be allowed <code>kms:Decrypt</code> on that specific key. Now two policies in two services both have to say yes, and you can scope the key policy to a named set of roles independently of who may call Systems Manager. The first gate protects the API surface, the second protects the plaintext.</p>



<p class="wp-block-paragraph">The same split exists in Secrets Manager, between the AWS managed key and one you create.</p>



<pre class="wp-block-code"><code># Store an encrypted parameter under a customer managed key
# instead of the AWS managed default.
aws ssm put-parameter 
  --name "/prod/billing/db-password" 
  --type SecureString 
  --key-id "alias/prod-app-secrets" 
  --value "REPLACE_ME" 
  --overwrite

# Reading it now needs ssm:GetParameter AND
# kms:Decrypt on alias/prod-app-secrets.
aws ssm get-parameter 
  --name "/prod/billing/db-password" 
  --with-decryption 
  --query "Parameter.Value" 
  --output text</code></pre>



<p class="wp-block-paragraph">There is a second payoff. A customer managed key makes the decrypt call attributable in CloudTrail against a key policy you control, which turns &#8220;someone read a secret&#8221; into &#8220;this role decrypted this value with this key at this time&#8221;. That is usually what an auditor is asking for, and it is independent of which store you land on.</p>



<h3 class="wp-block-heading">3. What does the retrieval path look like under load?</h3>



<p class="wp-block-paragraph">Neither service is designed to be called on every request. Both punish you for it in different currencies: Parameter Store with a shared throughput ceiling, Secrets Manager with a per-call charge.</p>



<p class="wp-block-paragraph">The answer for both is caching, and AWS ships the mechanism rather than making you build it. The Secrets Manager Agent, now published under the name AWS Workload Credentials Provider, runs as a sidecar or local service, holds secrets in memory, and serves them to your application over a local HTTP endpoint, so the app makes one call instead of thousands and needs no AWS SDK dependency to do it. For Lambda, the AWS Parameters and Secrets Lambda Extension does the equivalent inside the execution environment and covers both services.</p>



<p class="wp-block-paragraph">Caching moves the problem rather than removing it. A cached credential is a stale credential during rotation, which loops straight back to lever one.</p>



<h3 class="wp-block-heading">4. How does the value get created and destroyed?</h3>



<p class="wp-block-paragraph">Creation first. CloudFormation cannot create an encrypted parameter, so if your pipeline is CloudFormation or CDK and the value must be encrypted from the moment it exists, Secrets Manager is the path of least resistance. The usual workarounds, a Lambda-backed custom resource or creating the parameter out of band, both move the secret outside the template that claims to own it. That is a real cost, not a detail.</p>



<p class="wp-block-paragraph">Destruction is where the surprise lives. Secrets Manager schedules deletion after a recovery window with a minimum of seven days and a default of thirty, and the name stays reserved for the duration.</p>



<pre class="wp-block-code"><code># Shorten the recovery window to the documented minimum.
aws secretsmanager delete-secret 
  --secret-id prod/billing/db 
  --recovery-window-in-days 7

# Or skip the window entirely. There is no undo after this.
aws secretsmanager delete-secret 
  --secret-id ephemeral/pr-1482/db 
  --force-delete-without-recovery</code></pre>



<p class="wp-block-paragraph">AWS notes the actual deletion runs as an asynchronous background process, so deleting and immediately recreating a secret with the same name needs backoff and retry even with the force flag. This is why per-branch environments built with Terraform or OpenTofu fail on their second run. Use the force flag deliberately, put a unique suffix in the name, or keep ephemeral values in Parameter Store where deletion is immediate.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Cost levers, without quoting numbers that will age</h2>



<p class="wp-block-paragraph">Published rates change. The billing mechanisms are stable, and the mechanisms are what let you predict the bill.</p>



<ul class="wp-block-list">
<li><strong>Parameter Store standard tier</strong> carries no storage charge. The advanced tier is charged per parameter, and the higher throughput setting is charged separately as an account and Region level switch.</li>



<li><strong>Secrets Manager</strong> charges per stored secret per month, prorated hourly, plus a charge per block of API calls. Replicas count as separate stored secrets.</li>



<li><strong>KMS</strong> charges a monthly fee per customer managed key plus a request charge on cryptographic operations. This applies to both services and it is the line most estimates forget.</li>



<li><strong>Rotation</strong> has no separate line item, but the rotation function bills as ordinary Lambda usage.</li>
</ul>



<p class="wp-block-paragraph">Two consequences. The Secrets Manager multiplier is not the number of secrets, it is secrets across environments across Regions, and replicas are the sneaky term because nobody remembers enabling them. And the advanced parameter tier is a one-way door: you can promote a standard parameter to advanced, but you cannot demote it, because reverting would truncate the value. Undoing that means delete and recreate.</p>



<p class="wp-block-paragraph">If you cannot see which term is growing, break the bill down by service and tag before you migrate anything. Cost Explorer handles the single-account case; for cross-account views without building the aggregation yourself, tools like Vantage or CloudZero will answer &#8220;is this secret count, replication, or retrieval frequency&#8221; in a few minutes. That is a better use of an afternoon than guessing.</p>



<h2 class="wp-block-heading">A decision procedure you can run today</h2>



<p class="wp-block-paragraph">Take an actual list of values and walk each one through this. It takes about a minute each.</p>



<ol class="wp-block-list">
<li><strong>Is it confidential at all?</strong> If no, Parameter Store, plain <code>String</code> type, done. Do not encrypt configuration for the aesthetics.</li>



<li><strong>Is it a database credential that should rotate on a schedule?</strong> Secrets Manager with managed rotation. Choose single user or alternating users based on whether consumers cache.</li>



<li><strong>Does it need scheduled rotation but is not a supported database?</strong> You are writing a function either way. Secrets Manager if you want versioning, staging labels and a recovery window; Parameter Store if the free tier matters more and you accept owning the lifecycle.</li>



<li><strong>Does it need to exist in more than one Region or account?</strong> Secrets Manager gives you replication and resource policies directly. Parameter Store shares across accounts through Resource Access Manager, advanced tier only.</li>



<li><strong>Does CloudFormation or CDK create it?</strong> Encrypted parameters are not creatable there, so it is Secrets Manager or an accepted out-of-band step.</li>



<li><strong>Is it read frequently?</strong> Cache it either way, then return to step two, because caching and rotation interact and that interaction is where outages come from.</li>



<li><strong>Everything else</strong> goes to Parameter Store, encrypted under a customer managed key.</li>
</ol>



<p class="wp-block-paragraph">Most accounts land with the large majority of values in Parameter Store and a short list of rotating database credentials in Secrets Manager. That split is healthy. What is not healthy is being unable to say why a particular value is where it is.</p>



<p class="wp-block-paragraph">Once you accept you will use both, the operational work is making them look like one system: one naming convention across both stores, IAM scoped to the path rather than the service, KMS keys separated by trust boundary rather than by service, and the rule written down where the next person will find it. Two lines in a repository README beats a policy document nobody opens.</p>



<h2 class="wp-block-heading">Arguments that do not survive contact</h2>



<p class="wp-block-paragraph"><strong>&#8220;Parameter Store is free, so use it for everything.&#8221;</strong> The standard tier has no storage charge, which is not the same as free. KMS requests bill, the higher throughput setting bills, advanced parameters bill, and the rotation function you write instead of using a managed one has a real cost in engineering hours and incident risk.</p>



<p class="wp-block-paragraph"><strong>&#8220;Secrets Manager is expensive.&#8221;</strong> Per secret it is small. The bill grows through multiplication, not unit price. Before migrating anything, count secrets across environments and Regions and find out how many replicas exist that nobody asked for. Deleting unused replicas is usually a bigger win than a migration project.</p>



<p class="wp-block-paragraph"><strong>&#8220;Just use Vault and skip the argument.&#8221;</strong> HashiCorp Vault is a better answer if you are multi-cloud, need dynamic short-lived credentials, or want secrets engines AWS does not offer. It also hands you a cluster to run, upgrade, seal and unseal, back up and monitor, plus a licensing decision. On a single-cloud AWS estate with a handful of rotating credentials, the AWS services are less capable and less work. Both statements are true at once, and which dominates depends on how much platform team you have.</p>



<p class="wp-block-paragraph"><strong>&#8220;It is encrypted at rest, so it is secure.&#8221;</strong> Encryption at rest protects against someone reading the storage layer, which is not the threat you have. Your threat is an over-permissive role and a compromised workload. That is an IAM policy and a KMS key policy question, and the answer is the same on either service.</p>



<p class="wp-block-paragraph"><strong>&#8220;We will add rotation later.&#8221;</strong> Rotation is not a feature flag, it is a coordination problem between the store, the target system and every consumer&#8217;s caching behaviour. Deciding that a credential will eventually rotate changes how you write the consumer today. Retrofitting it means touching every consumer anyway.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I migrate a value between the two without downtime?</h3>



<p class="wp-block-paragraph">Yes, with a dual read. Write the value to the new location, deploy consumers that read the new location and fall back to the old one, then use CloudTrail to confirm nothing still reads the old path before deleting it. Do not skip the CloudTrail step; the consumer you forgot is always a job that runs monthly. If the value passed through a local file during the move, erase that file properly rather than with <code>rm</code>, using <code>shred</code> on Linux or a secure deletion tool such as those from O&amp;O Software on Windows.</p>



<h3 class="wp-block-heading">Does Parameter Store support rotation?</h3>



<p class="wp-block-paragraph">Not as a built-in feature. You write a function and invoke it on a schedule, typically with an EventBridge rule. You also inherit everything managed rotation handles for you: verifying the new credential works before committing to it, rolling back cleanly when it does not, and giving consumers an overlap window.</p>



<h3 class="wp-block-heading">Which one should I use for third-party API keys?</h3>



<p class="wp-block-paragraph">It depends on whether you rotate them. Most vendor keys get rotated manually and rarely, in which case an encrypted parameter under a customer managed key is enough. If the provider supports programmatic key rotation and you want that automated, Secrets Manager&#8217;s version staging is what makes the swap safe.</p>



<h3 class="wp-block-heading">Can workloads outside AWS read from either service?</h3>



<p class="wp-block-paragraph">Both, over the API, with AWS credentials. That means a long-lived access key or a federated identity, and you should federate wherever the platform supports it. If part of your estate runs on VPS hosts at a provider like Contabo or InterServer, this is the boundary to think hardest about, because you have just placed an AWS credential on a machine whose network you do not control. Scope the role narrowly, apply condition keys where the network path allows, and treat that credential as the one most likely to leak.</p>



<h3 class="wp-block-heading">What happens if I delete something by accident?</h3>



<p class="wp-block-paragraph">Secrets Manager lets you restore during the recovery window, which is the single best argument for putting genuinely important credentials there. Parameter Store has no equivalent. Once the parameter is gone your options are whatever exists in state files, backups or someone&#8217;s terminal history, and none of those is a plan.</p>



<h3 class="wp-block-heading">Should ECS or EKS workloads pull secrets differently?</h3>



<p class="wp-block-paragraph">ECS can inject values from either service directly into a task&#8217;s environment, which removes SDK code from the container but reads the value at task start, so a rotation does not reach a running task. On EKS you have both the Secrets Store CSI driver with the AWS provider, which mounts values as files, and the agent sidecar pattern, which serves them over localhost and can refresh without restarting the pod. Choose based on whether your application can reload a credential mid-life. If it cannot, mounting and injection are equivalent, and rotation means a rollout.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The <strong>AWS Secrets Manager vs Parameter Store</strong> decision is a lifecycle decision wearing a pricing costume. What separates the two is not what they store, it is what they do when the value changes, when it gets deleted, when a Region falls over, and when ten thousand containers ask for it at once.</p>



<p class="wp-block-paragraph">Decide per value, not per project. Configuration and rarely-changing keys belong in Parameter Store. Database credentials that rotate on a schedule belong in Secrets Manager. Put a customer managed key in front of both so the second gate does real work, cache reads so you are not fighting a quota, and make sure whoever wrote the caching understands what happens the next time a rotation runs. If you cannot explain why a given value lives where it does, that is the value that will surprise you.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your secrets setup?</h2>



<p class="wp-block-paragraph">Most of the work here is not choosing a service, it is untangling what already exists. Things I can help with:</p>



<ul class="wp-block-list">
<li>Auditing an AWS account for where secrets actually live, which roles can decrypt them, and which values are duplicated across environments or replicated into Regions nobody uses</li>



<li>Designing a naming hierarchy and path-scoped IAM so one policy covers a service instead of one policy per value</li>



<li>Setting up managed rotation for RDS and Aurora credentials, including the alternating users pattern when the application caches and cannot be changed</li>



<li>Writing and testing custom rotation functions for non-database secrets, with the rollback path actually exercised rather than assumed</li>



<li>Moving values between the two stores with a dual-read cutover and CloudTrail verification that nothing is left reading the old path</li>



<li>Working out whether a growing Secrets Manager line item is secret count, replication or retrieval frequency, and fixing the one actually responsible</li>
</ul>



<p class="wp-block-paragraph">Send me a redacted IAM policy, a CloudTrail query result, or the output of listing your parameters by path, and I will tell you what I would change first and why.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-secrets-manager-vs-parameter-store/">AWS Secrets Manager vs Parameter Store: Stop Deciding on Price</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building an Ecommerce Recommendation Pipeline on AWS: The Parts That Bite</title>
		<link>https://john-nessime.com/blog/cloud-computing/aws-ecommerce-recommendation-pipeline/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 26 Oct 2026 06:37:42 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Machine Learning]]></category>
		<category><![CDATA[Retail Technology]]></category>
		<category><![CDATA[Amazon Personalize]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Batch Processing]]></category>
		<category><![CDATA[Clickstream Ingestion]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Ecommerce Analytics]]></category>
		<category><![CDATA[Idle Resources]]></category>
		<category><![CDATA[Item Cold Start]]></category>
		<category><![CDATA[Kinesis Data Streams]]></category>
		<category><![CDATA[Model Serving]]></category>
		<category><![CDATA[Recommendation Systems]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Training-Serving Skew]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=894</guid>

					<description><![CDATA[<p>An idle Amazon Personalize setup does not throw errors. It just bills. A practical walkthrough of the five decisions that shape an AWS ecommerce recommendation pipeline: domain recommenders versus custom recipes, bulk versus streaming ingestion, schema limits, training cadence, and real-time versus batch serving, with the failure modes that stay invisible until the invoice arrives.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/aws-ecommerce-recommendation-pipeline/">Building an Ecommerce Recommendation Pipeline on AWS: The Parts That Bite</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question usually arrives from finance rather than from engineering. Something along the lines of: why is there an Amazon Personalize charge on an account where nobody has shipped a recommendation carousel yet?</p>



<p class="wp-block-paragraph">The answer is almost always the same. Someone followed a tutorial, created a dataset group, trained a solution, deployed a campaign to see if the output looked reasonable, then got pulled onto something else. The campaign is still holding a throughput floor. The solution is still retraining on a schedule. Neither is wired to a storefront. Both are billing.</p>



<p class="wp-block-paragraph">I want that failure mode first because it is invisible. A broken AWS ecommerce recommendation pipeline throws errors and gets fixed by Friday. An idle one throws nothing. It sits there being technically correct and quietly expensive.</p>



<p class="wp-block-paragraph">This post covers the five decisions that actually shape the pipeline: which resource type to train, how events get in, what the schema can hold, how fresh the model needs to be, and how a recommendation reaches the page. I will flag the spots where the default behavior is not the behavior you want.</p>



<h2 class="wp-block-heading">The bill arrives before the carousel does</h2>



<p class="wp-block-paragraph">Two things in Personalize cost money on a clock rather than on demand, and both default toward spending.</p>



<p class="wp-block-paragraph"><strong>Solution training.</strong> New custom solutions use automatic training by default. Personalize creates a new solution version every seven days unless you say otherwise, and the cadence is configurable between one and thirty days. The first run starts within an hour of the solution going active. AWS is explicit that you incur training costs while the solution is active, and that switching automatic training off is how you stop them.</p>



<p class="wp-block-paragraph"><strong>Inference capacity.</strong> A campaign has <code>minProvisionedTPS</code>, a domain recommender has <code>minRecommendationRequestsPerSecond</code>. Both default to 1. Both set a billing floor. Personalize scales above the floor as traffic climbs but never below it, and you are billed on the higher of the floor and your actual rate. A campaign serving zero requests still bills its floor for every hour it stays active.</p>



<p class="wp-block-paragraph">Two habits fix most of this. While you are evaluating rather than serving, create solutions with automatic training off and trigger training yourself. And delete campaigns you are not calling instead of leaving them parked.</p>



<pre class="wp-block-code"><code>import boto3

personalize = boto3.client('personalize')

# Automatic training off while evaluating.
# You then call create_solution_version() yourself when you want a model.
response = personalize.create_solution(
    name='storefront-user-personalization',
    datasetGroupArn='arn:aws:personalize:REGION:ACCOUNT:dataset-group/NAME',
    recipeArn='arn:aws:personalize:::recipe/aws-user-personalization-v2',
    performAutoTraining=False
)
print(response['solutionArn'])</code></pre>



<h2 class="wp-block-heading">Decision one: domain recommender or custom solution</h2>



<p class="wp-block-paragraph">A domain dataset group created with the ECOMMERCE domain exposes pre-configured use cases. A custom dataset group makes you pick a recipe and configure the solution yourself.</p>



<p class="wp-block-paragraph">The ECOMMERCE use cases map onto the modules you were going to build anyway: Recommended for you, Frequently bought together, Customers who viewed X also viewed, Best sellers, Most viewed. Each carries an event-type requirement. The purchase-driven ones need at minimum a thousand Purchase events; the view-driven ones need at minimum a thousand View events. That is a useful early check on your event taxonomy, before you spend a training run finding out.</p>



<p class="wp-block-paragraph">Where the domain route wins: you skip the recipe argument, the defaults are sensible, and you reach a testable result faster. For a catalog under a few thousand SKUs with a normal retail event mix, this is the one I reach for first.</p>



<p class="wp-block-paragraph">Where it hurts: you are constrained to the shape of the provided use cases, you get less control over hyperparameters, and you cannot create next best action resources inside a domain dataset group at all. If you need to recommend actions rather than products, you are in custom territory.</p>



<p class="wp-block-paragraph">On the custom side the recipes worth knowing are User-Personalization-v2 for a personalized feed, Similar-Items for a related-products rail, Personalized-Ranking-v2 for reordering a category or search page, Trending-Now for what is moving, and Popularity-Count as the baseline you should measure against. The v2 recipes use a transformer architecture, handle much larger catalogs, and return item metadata in the inference response, saving a catalog lookup on the render path. They also bill on a different model from the older recipes, so do not carry a cost estimate across.</p>



<p class="wp-block-paragraph">One thing to watch while reading around: the legacy HRNN recipes are no longer available. Their documentation stays online for reference and plenty of third-party tutorials still use them. If a guide points a solution at <code>aws-hrnn</code>, its advice about cold items and training cadence is stale too.</p>



<h2 class="wp-block-heading">Decision two: how interaction data gets in</h2>



<h3 class="wp-block-heading">Bulk imports replace unless you tell them not to</h3>



<p class="wp-block-paragraph">This is the one that costs people a month of confused model quality.</p>



<p class="wp-block-paragraph">A dataset import job defaults to <code>FULL</code>, which overwrites all existing bulk data in the dataset. It leaves records you sent individually through the event tracker alone, but everything previously loaded from S3 is replaced by whatever is in this job. The alternative is <code>INCREMENTAL</code>, which appends and only replaces records sharing an ID.</p>



<p class="wp-block-paragraph">Picture a nightly export producing a rolling thirty-day extract, fed into an import job left on the default. Every night the dataset is truncated to thirty days. Nothing errors. The job reports success. Model quality just degrades as the training window shrinks, and it is not obvious why, because the pipeline is green.</p>



<pre class="wp-block-code"><code>import boto3

personalize = boto3.client('personalize')

# INCREMENTAL appends to what is already there.
# FULL, the default, replaces all previously bulk-imported records.
personalize.create_dataset_import_job(
    jobName='interactions-nightly',
    datasetArn='arn:aws:personalize:REGION:ACCOUNT:dataset/NAME/INTERACTIONS',
    dataSource={'dataLocation': 's3://your-bucket/interactions/'},
    roleArn='arn:aws:iam::ACCOUNT:role/PersonalizeS3Role',
    importMode='INCREMENTAL'
)</code></pre>



<p class="wp-block-paragraph">Two limits shape the schedule. An incremental interactions import is capped well below a full one, so treat it as a delta mechanism and not a backfill. And an interactions import job needs a minimum number of new records to be accepted, so a low-traffic store cannot run one every fifteen minutes and expect success. Batch the deltas into a cadence that clears the floor.</p>



<h3 class="wp-block-heading">Streaming events through an event tracker</h3>



<p class="wp-block-paragraph">The live path is a separate API. Create an event tracker, take the tracking ID it returns, and call PutEvents against the <code>personalize-events</code> client.</p>



<pre class="wp-block-code"><code>import boto3
import time

personalize_events = boto3.client('personalize-events')

personalize_events.put_events(
    trackingId='YOUR_TRACKING_ID',
    userId='user-8842',
    sessionId='sess-19f3ac',
    eventList=[{
        'eventId': 'evt-1',
        'eventType': 'View',
        'itemId': 'SKU-4471',
        'sentAt': int(time.time())
    }]
)</code></pre>



<p class="wp-block-paragraph">Do not put that call inline in the request path of a product page. Recommendation telemetry is not worth a network round trip on your time to first byte, and it is certainly not worth a page error when the API has a bad minute. Push events onto Kinesis Data Streams or Amazon Data Firehose, drain with Lambda, call PutEvents from there.</p>



<p class="wp-block-paragraph">That indirection buys a second thing you want anyway: Firehose can land a durable copy of the same events in S3, which becomes the source for bulk imports and for analysis outside Personalize. Events that only ever existed inside a managed service are events you cannot re-examine when the model does something odd.</p>



<p class="wp-block-paragraph">The <code>sessionId</code> field matters more than it looks. It lets you record anonymous browsing before login and have Personalize associate that session with the user afterward. Skip it and you train on your smallest, least representative slice of traffic.</p>



<h2 class="wp-block-heading">Decision three: schema and event types, decided once</h2>



<p class="wp-block-paragraph">Personalize schemas are Avro. An interactions schema needs a user ID, an item ID and a timestamp. An event type field is optional, but the domain use cases key off it, so you want it.</p>



<pre class="wp-block-code"><code>{
  "type": "record",
  "name": "Interactions",
  "namespace": "com.amazonaws.personalize.schema",
  "fields": [
    { "name": "USER_ID",    "type": "string" },
    { "name": "ITEM_ID",    "type": "string" },
    { "name": "EVENT_TYPE", "type": "string" },
    { "name": "TIMESTAMP",  "type": "long" }
  ],
  "version": "1.0"
}</code></pre>



<p class="wp-block-paragraph">The budget for extra fields is tighter than people expect. An interactions dataset allows only a handful of metadata columns beyond the reserved fields, and the count of distinct event types plus those metadata columns is itself capped. That is a hard design constraint. You cannot ship twenty-eight granular event types and also carry rich per-event context.</p>



<p class="wp-block-paragraph">Spend it deliberately. View, AddToCart and Purchase carry most retail pipelines. Wishlist adds earn a slot if customers use them. Hover, scroll depth and image zoom do not, because they inflate volume without changing what the model learns about intent. Once you have several event types you can weight them differently at training time, which beats the alternative people reach for first, which is dropping views entirely.</p>



<p class="wp-block-paragraph">Before any of that, settle the item ID question. Variants and parent products are different objects in Shopify and WooCommerce alike, and a model trained on variant IDs will happily suggest the blue medium of a shirt the customer already owns in blue medium. Pick a level, write it down, use the same one in your bulk extract and your live payload. Getting those out of step is the most common reason a technically healthy pipeline produces recommendations that feel broken.</p>



<p class="wp-block-paragraph">On volume: the documented floor is a thousand item interactions from at least twenty-five unique users with two or more interactions each, plus a minimum share of your user base clearing that two-interaction bar. AWS recommends considerably more for quality. Near the minimum, the honest read is that you are not building a recommendation pipeline yet. You are building the event capture that lets you build one later, and a popularity-ranked rail will serve customers better meanwhile.</p>



<p class="wp-block-paragraph">Timestamps are Unix time. Mixed timezones, or milliseconds where seconds are expected, will not throw. They will scramble the ordering the model learns from, which is the entire signal.</p>



<h2 class="wp-block-heading">Decision four: freshness without a retraining bill</h2>



<p class="wp-block-paragraph">Two mechanisms get conflated constantly, and the difference sets your monthly cost.</p>



<p class="wp-block-paragraph"><strong>Automatic training</strong> creates a new solution version. The model learns from your latest data. It costs a training run each time.</p>



<p class="wp-block-paragraph"><strong>Automatic updates</strong> are something else entirely. For User-Personalization-v2, User-Personalization and Next-Best-Action, and for the Top picks for you and Recommended for you domain use cases, Personalize refreshes the deployed model every two hours so new items can surface through exploration. That is not retraining. The model does not learn from your recent interactions. It just stops new products from being structurally invisible.</p>



<p class="wp-block-paragraph">So automatic updates solve the cold item problem and do nothing about model drift. AWS&#8217;s own guidance is that even with them, training cadence should stay at least weekly.</p>



<p class="wp-block-paragraph">Trending-Now is the interesting exception. It evaluates interactions over a configurable interval and identifies what is climbing, without waiting for a training run. For a store with promotional spikes or seasonal churn, a Trending-Now rail beside a personalized rail covers a lot of ground for very little retraining.</p>



<p class="wp-block-paragraph">The trap is Similar-Items, which has no automatic updates. Add products daily, run a related-products rail on Similar-Items, and new products only appear after the next training. Same-day visibility means daily training. That is a real budget line, and it is the point where catalog velocity stops being a merchandising detail and becomes an infrastructure decision.</p>



<h2 class="wp-block-heading">Decision five: how the recommendation reaches the page</h2>



<p class="wp-block-paragraph">Real-time means a campaign or a recommender answering GetRecommendations. Batch means an inference job reading input from S3 and writing recommendations back to S3, with no campaign involved and therefore no throughput floor.</p>



<p class="wp-block-paragraph">Most small and mid-sized stores reach for real-time by reflex and do not need it. Ask what the module actually is. A homepage rail refreshed daily, an email send, a merchandising export: those are batch problems. Run the batch overnight, load results into DynamoDB or a Redis instance, serve the carousel from a key lookup. Single-digit millisecond reads, no per-request inference charge, and nothing sitting idle at three in the morning.</p>



<p class="wp-block-paragraph">Real-time earns its cost where the recommendation has to react inside the session, which in practice means product detail and cart pages.</p>



<p class="wp-block-paragraph">Two things belong in the serving layer from day one either way. Filters, so you can exclude out-of-stock items and things the customer already bought; note that Personalize considers a bounded number of interactions per user per event type when filtering, and a user with no imported interactions for several months stops counting toward filtering at all. And a timeout with a fallback, because a carousel must never be able to delay a product page. Fall back to a cached bestseller list and log the fallback rate as a metric, so a rise shows up in CloudWatch or Grafana rather than in a conversion report a fortnight later.</p>



<h2 class="wp-block-heading">Troubleshooting an AWS ecommerce recommendation pipeline</h2>



<ul class="wp-block-list">
<li><strong>Everyone gets the same recommendations.</strong> Usually too few interactions per user, or user IDs that are not stable across sessions. Check that anonymous activity is recorded with a session ID and reconciled at login.</li>



<li><strong>New products never show up.</strong> Check that your recipe supports automatic updates. If it does not, this is a training cadence problem, not a data problem.</li>



<li><strong>Training fails on data volume.</strong> Compare against the documented minimums. The share of users with two or more interactions catches people out: plenty of users is not the same as plenty of returning users.</li>



<li><strong>The dataset shrank and nothing errored.</strong> A bulk import running in FULL mode against a rolling extract. Check the import mode on every scheduled job.</li>



<li><strong>Recommendations ignore what the user just clicked.</strong> Confirm events reach the event tracker, and that the user ID sent to PutEvents matches the one sent to GetRecommendations exactly.</li>



<li><strong>The bill rose during a quiet week.</strong> Automatic training runs on a calendar, not on traffic. Check cadence, and check for campaigns nobody is calling.</li>



<li><strong>Filters are not excluding purchased items.</strong> Filtering looks at a bounded slice of each user&#8217;s history per event type. Heavy users push older purchases outside it.</li>
</ul>



<p class="wp-block-paragraph">Auditing an account you did not build starts with listing what exists, not with reading the application code.</p>



<pre class="wp-block-code"><code>aws personalize list-dataset-groups
aws personalize list-solutions --dataset-group-arn ARN
aws personalize list-campaigns --solution-arn ARN
aws personalize list-recommenders --dataset-group-arn ARN

# Then see what each campaign is holding open
aws personalize describe-campaign --campaign-arn ARN</code></pre>



<p class="wp-block-paragraph">Orphaned campaigns from an abandoned proof of concept show up in that output in about ninety seconds.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Leaving automatic training on for a solution that exists only to satisfy curiosity.</li>



<li>Raising the provisioned throughput floor for a traffic spike, then never lowering it afterward.</li>



<li>Modelling on variant IDs in the bulk extract and product IDs in the live events, or the reverse.</li>



<li>Recording events only for authenticated users, starving the model of most of your traffic.</li>



<li>Designing a rich event taxonomy that collides with the cap on event types plus metadata columns.</li>



<li>Calling PutEvents synchronously on the render path.</li>



<li>Following an implementation guide that still references the retired HRNN recipes.</li>



<li>Shipping with no fallback, so the first slow inference call becomes a slow product page.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Keep a raw copy of every interaction event in S3, independent of Personalize. Managed services are easier to reason about when you can replay what you sent them.</li>



<li>Start with a domain dataset group. Move to custom recipes when you can name the specific thing the domain route will not do.</li>



<li>Measure against a popularity baseline before celebrating anything. If a personalized rail cannot beat bestsellers, the pipeline is not earning its cost yet.</li>



<li>Set training cadence from catalog velocity, not from habit. A stable catalog of a few hundred products does not need what a weekly fashion drop needs.</li>



<li>Prefer batch inference plus a cache for anything that does not have to react within the session.</li>



<li>Alarm on idle inference resources. Utilization near zero for days is a cost bug, and it is the only bug here that never pages anyone.</li>



<li>Keep the staging storefront on cheap infrastructure. The application side of this runs fine on a modest VPS from a provider like Contabo or InterServer while you are still shaping the event schema, and it keeps experiments off the production account.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How much data do I need before Personalize is worth it?</h3>



<p class="wp-block-paragraph">The documented training floor is a thousand item interactions from at least twenty-five unique users with two or more interactions each, and AWS recommends substantially more for quality recommendations. Below that, popularity ranking and hand-curated rails serve customers better and cost nothing to run.</p>



<h3 class="wp-block-heading">Can I build this without Amazon Personalize?</h3>



<p class="wp-block-paragraph">Yes, and for some shops you should. Item-to-item collaborative filtering over purchase history is a modest amount of code and runs fine as a scheduled job writing to a cache. You give up cold-start handling, exploration and session-level personalization. You gain a near-zero cost and full visibility into why an item was recommended.</p>



<h3 class="wp-block-heading">What does an idle Personalize setup cost?</h3>



<p class="wp-block-paragraph">More than zero, which is the surprising part. An active campaign bills its provisioned throughput floor whether or not anyone calls it, and a solution with automatic training on bills a training run on its cadence regardless of traffic. Rates change, so apply current pricing yourself, but the shape is a floor plus a recurring training charge.</p>



<h3 class="wp-block-heading">Real-time or batch recommendations?</h3>



<p class="wp-block-paragraph">Batch unless the module has to respond to the current session. Homepage rails, email and merchandising exports are batch problems. Product detail and cart pages are where real-time inference genuinely changes what the customer sees.</p>



<h3 class="wp-block-heading">Do new products appear automatically?</h3>



<p class="wp-block-paragraph">Only with recipes that support automatic updates, or with Trending-Now. Automatic updates refresh the deployed model every couple of hours so new items can be explored, but they do not retrain it. With a recipe like Similar-Items, a new product waits for the next training run.</p>



<h3 class="wp-block-heading">How do I connect a Shopify or WooCommerce store?</h3>



<p class="wp-block-paragraph">There is no first-party connector. Emit events from the storefront to your own endpoint, push them onto a stream, and call PutEvents from a consumer. For history, export orders and product views to S3 and load them with a dataset import job. Ordinary plumbing; the hard part is settling the item ID question first.</p>



<h3 class="wp-block-heading">How often should I retrain?</h3>



<p class="wp-block-paragraph">At least weekly is the documented baseline, even for recipes with automatic updates. Go faster only when your catalog turns over faster, and price the extra training runs before committing to a daily cadence.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">An AWS ecommerce recommendation pipeline runs two clocks, and they are easy to mistake for one. The freshness clock decides whether a customer can see the product you added this morning. The billing clock runs on training cadence and provisioned throughput, and it keeps running when nobody is looking at the site at all.</p>



<p class="wp-block-paragraph">Get the event schema right, keep a raw copy of everything in S3, stay on batch until the session genuinely demands otherwise, and set training cadence from how fast your catalog actually moves. Do that and the rest is straightforward. Skip it and you end up with something that works, costs more than the revenue it lifts, and nobody can say why.</p>



<h2 class="wp-block-heading">Need a second pair of eyes on your recommendation pipeline?</h2>



<p class="wp-block-paragraph">This is the kind of work I do for ecommerce teams building on AWS:</p>



<ul class="wp-block-list">
<li>Auditing an existing Personalize setup for orphaned campaigns, over-provisioned throughput floors and training cadences that outrun the catalog.</li>



<li>Designing the event schema and item ID strategy before you commit, including the variant-versus-product decision for Shopify and WooCommerce catalogs.</li>



<li>Building the ingestion path: storefront events through Kinesis or Firehose into Lambda, with a durable S3 copy for bulk imports and replay.</li>



<li>Choosing between domain recommenders and custom recipes for your catalog size and event mix, and modelling what each costs to run.</li>



<li>Moving eligible modules from real-time campaigns to batch inference plus a cache layer, then measuring what that takes off the bill.</li>



<li>Setting up monitoring so idle inference capacity and rising fallback rates surface as alerts rather than as a surprise invoice.</li>
</ul>



<p class="wp-block-paragraph">If something is already running, send me your dataset import job configuration and a list of your active campaigns, and I will tell you what I would change first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/aws-ecommerce-recommendation-pipeline/">Building an Ecommerce Recommendation Pipeline on AWS: The Parts That Bite</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>RDS vs DynamoDB: The Benchmark Isn&#8217;t the Decision</title>
		<link>https://john-nessime.com/blog/technical-guides/rds-vs-dynamodb-access-patterns/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 29 Sep 2026 06:05:28 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Databases]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Aurora]]></category>
		<category><![CDATA[Amazon RDS]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Capacity Planning]]></category>
		<category><![CDATA[Connection Pooling]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Modeling]]></category>
		<category><![CDATA[DynamoDB]]></category>
		<category><![CDATA[Hot Partition]]></category>
		<category><![CDATA[Over-Engineering]]></category>
		<category><![CDATA[PostgreSQL]]></category>
		<category><![CDATA[Provisioned Throughput]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Throttling]]></category>
		<category><![CDATA[Workload Shape]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=738</guid>

					<description><![CDATA[<p>Latency charts don't decide this. The query nobody wrote down at the start does. How to build an access pattern inventory, where DynamoDB and RDS each genuinely win and hurt, a six-question decision procedure, and the arguments on both sides that fall apart under a real workload.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/rds-vs-dynamodb-access-patterns/">RDS vs DynamoDB: The Benchmark Isn&#8217;t the Decision</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone drops a benchmark link into the design review. Flat latency line for DynamoDB, a rising one for Postgres, and the data store is chosen before anybody has described what the application does.</p>



<p class="wp-block-paragraph">That argument settles architecture decisions all the time, and it&#8217;s the wrong argument. Both services return a primary-key lookup faster than your network hop and your ORM overhead. What hurts six months later is the query nobody wrote down at the start, and no benchmark will warn you about it.</p>



<p class="wp-block-paragraph">So this post skips the latency charts. It covers the access pattern inventory you write before picking anything, where each service genuinely wins and genuinely hurts, a decision procedure you can run in an afternoon, and the arguments on both sides that fall apart on contact with a real workload.</p>



<h2 class="wp-block-heading">Why the benchmark answers a question you don&#8217;t have</h2>



<p class="wp-block-paragraph">A benchmark measures a fixed operation under fixed conditions. Your application is not a fixed operation. It&#8217;s a set of reads and writes that grows every time somebody asks a new question about the data.</p>



<p class="wp-block-paragraph">The failure mode that decides RDS vs DynamoDB is almost never throughput. It&#8217;s a product manager asking, eight months in, for every account that hasn&#8217;t logged in this quarter, broken down by plan tier. On Postgres that&#8217;s an awkward query and maybe an index. On DynamoDB, if you didn&#8217;t model for it, it&#8217;s a full table scan, a new global secondary index, a backfill, and a standing write-capacity tax to keep it in sync.</p>



<p class="wp-block-paragraph">That asymmetry is the whole decision. A relational database defers the modeling cost and charges a little on every query for the flexibility. DynamoDB front-loads it: design keys around known patterns, get cheap predictable reads for exactly those, and pay dearly for anything else.</p>



<h2 class="wp-block-heading">Write the access pattern inventory first</h2>



<p class="wp-block-paragraph">Before comparing anything, write the list. Every read and write the application performs, in plain language, with the shape of the operation and how fast it needs to be. This much is enough:</p>



<pre class="wp-block-code"><code>PATTERN                                SHAPE           FREQ    LATENCY
get order by id                        point read      high    p99 &lt; 50ms
list a customer's orders, newest first range read      high    p99 &lt; 100ms
orders by status, across customers     cross-entity    low     seconds fine
revenue by product category, monthly   aggregate       low     minutes fine
find orders by partial address         text match      medium  p99 &lt; 300ms
mark order shipped + write audit row   multi-item write high   p99 &lt; 100ms</code></pre>



<p class="wp-block-paragraph">Now read the SHAPE column. If nearly every row is a point or range read, and the exceptions tolerate latency, DynamoDB is a live option. If the column is full of cross-entity reads, aggregates and text matching, you&#8217;re describing a relational workload and can stop shopping.</p>



<h3 class="wp-block-heading">The rows people leave out</h3>



<p class="wp-block-paragraph">The inventory is only useful if it&#8217;s honest, and the omissions are predictable:</p>



<ul class="wp-block-list">
<li>The admin back office. Support staff search by email, phone, partial name, anything. Ad hoc queries, and they never appear in the first inventory.</li>



<li>Reporting. Someone will want a monthly rollup. Decide now whether it lives in the operational store or streams out to a warehouse.</li>



<li>Deletion. Regulatory deletes cascade across entities. In a normalized schema that&#8217;s a transaction. In a denormalized single table it&#8217;s a fan-out you have to write and test yourself.</li>



<li>Backfills. Every schema change becomes a migration job when no schema is enforced at the storage layer.</li>
</ul>



<p class="wp-block-paragraph">Write them down even if the answer is &#8220;we&#8217;ll handle it elsewhere.&#8221; A pattern you consciously exile to a warehouse is fine. One you forgot is a rewrite.</p>



<h2 class="wp-block-heading">Where DynamoDB wins, and where it doesn&#8217;t</h2>



<p class="wp-block-paragraph">Give it its genuine case first, because it&#8217;s a strong one.</p>



<h3 class="wp-block-heading">Where it wins</h3>



<ul class="wp-block-list">
<li><strong>Connection model.</strong> It&#8217;s an HTTPS API with SigV4 auth, so there&#8217;s no pool to exhaust. That&#8217;s why it pairs cleanly with Lambda. A relational database behind high Lambda concurrency needs RDS Proxy or a pooler, which is another component you operate.</li>



<li><strong>Predictable reads at any size.</strong> A query against a well-chosen partition key costs the same at ten thousand items or ten billion. Nothing in Postgres gives you that for free.</li>



<li><strong>Idle cost.</strong> On-demand mode bills per request unit consumed, so a table with no traffic costs storage and nothing else. Useful for tenant-per-table designs and low-traffic side services.</li>



<li><strong>Operational surface.</strong> No minor version upgrades, no vacuum, no failover window, no parameter groups to argue about.</li>
</ul>



<p class="wp-block-paragraph">The query API is narrow on purpose. Give it an exact partition key plus an optional sort key condition, and it returns an ordered slice of an item collection:</p>



<pre class="wp-block-code"><code>aws dynamodb query 
  --table-name AppData 
  --key-condition-expression "PK = :pk AND begins_with(SK, :sk)" 
  --expression-attribute-values '{":pk":{"S":"CUSTOMER#42"},":sk":{"S":"ORDER#"}}'</code></pre>



<p class="wp-block-paragraph">One customer&#8217;s orders, in sort order, in one request, no join. If that call shape covers most of your inventory, you&#8217;re looking at the right service.</p>



<h3 class="wp-block-heading">Where it doesn&#8217;t</h3>



<ul class="wp-block-list">
<li><strong>Anything you didn&#8217;t model.</strong> New patterns mean new keys or a new index, and a new global secondary index means a backfill plus write capacity on every future write.</li>



<li><strong>Filtering is not querying.</strong> A <code>FilterExpression</code> runs after the read, so you pay capacity for every item examined and only save bandwidth. A filter that discards most of what it reads is a design smell.</li>



<li><strong>Aggregates.</strong> There is no <code>GROUP BY</code> and no <code>COUNT(*)</code> over a table. You maintain aggregates yourself, usually with DynamoDB Streams feeding a Lambda that updates a counter item, which means you own the correctness problem when that consumer fails.</li>



<li><strong>Hard physical limits.</strong> An item caps at 400 KB. A Query or Scan page returns at most 1 MB before you paginate. A single partition sustains roughly 3,000 read and 1,000 write units per second, so a hot key throttles while the table sits well under its quota.</li>



<li><strong>Index budget.</strong> Global secondary indexes default to twenty per table; local secondary indexes cap at five and can only be defined at table creation. You can&#8217;t index your way out of a bad key design forever.</li>
</ul>



<h2 class="wp-block-heading">Where RDS wins, and where it doesn&#8217;t</h2>



<h3 class="wp-block-heading">Where it wins</h3>



<ul class="wp-block-list">
<li><strong>Unknown future queries.</strong> The entire point. Joins, aggregates, window functions, ad hoc analysis, and a new index in one statement when something turns out slow.</li>



<li><strong>Constraints the database enforces.</strong> Foreign keys, uniqueness, check constraints, transactions across arbitrary tables. In DynamoDB each of those is application code you write and maintain.</li>



<li><strong>Extensions.</strong> PostGIS for geospatial, pgvector for embeddings, full-text search built in, all against the same rows in the same transaction.</li>



<li><strong>Portability.</strong> Postgres and MySQL run on RDS, in a container, or on a plain VPS from a provider like Contabo or InterServer, so migrating off is an export and an import. DynamoDB&#8217;s data model doesn&#8217;t port without rewriting your data access layer.</li>



<li><strong>Everyone already knows SQL.</strong> Underrated on a small team. Teaching four developers single-table design costs real review time.</li>
</ul>



<h3 class="wp-block-heading">Where it doesn&#8217;t</h3>



<ul class="wp-block-list">
<li><strong>Connections under serverless concurrency.</strong> Each Lambda execution environment opens its own connection, and the database refuses new ones once <code>max_connections</code> is reached. RDS Proxy multiplexes to fix that, but some operations pin a client to a backend connection and pinned connections aren&#8217;t reused. Watch <code>DatabaseConnectionsCurrentlySessionPinned</code> before trusting the pool.</li>



<li><strong>Instances are always on.</strong> A provisioned instance bills whether or not anyone queries it. Aurora Serverless v2 changes that with a minimum of zero capacity units on a supported engine version, but resuming from paused is slow enough that the first user after an idle period feels it. AWS documents resume as taking roughly fifteen seconds.</li>



<li><strong>Vertical scaling has a ceiling.</strong> Writes go to one primary and read replicas help reads only. Sharding is a project, not a setting.</li>



<li><strong>Maintenance is yours.</strong> Version upgrades, failover testing, long transactions blocking vacuum, connection storms on restart. Managed does not mean absent.</li>
</ul>



<p class="wp-block-paragraph">If you want a database that hibernates properly, it lives in the scaling configuration and the idle timeout is yours to set:</p>



<pre class="wp-block-code"><code>aws rds create-db-cluster 
  --db-cluster-identifier app-cluster 
  --engine aurora-postgresql 
  --serverless-v2-scaling-configuration 
      MinCapacity=0,MaxCapacity=4,SecondsUntilAutoPause=600 
  --manage-master-user-password</code></pre>



<p class="wp-block-paragraph">A minimum of zero only works on engine versions supporting automatic pause and resume, so check yours before planning a budget around it. If the cluster refuses the setting, that&#8217;s usually why.</p>



<h2 class="wp-block-heading">A decision procedure for RDS vs DynamoDB</h2>



<p class="wp-block-paragraph">Run these in order against the inventory you wrote. The first clear answer wins.</p>



<ol class="wp-block-list">
<li><strong>Can you enumerate the access patterns, and are they stable?</strong> If the product is still moving and nobody can say what the admin tools will need, pick the store that tolerates being wrong. That&#8217;s RDS.</li>



<li><strong>Does any pattern need a cross-entity aggregate on the hot path?</strong> A user-facing screen needing a count or sum across entities in under a second, with no way to precompute it, means RDS.</li>



<li><strong>Is there a natural partition key with even distribution?</strong> Tenant ID, user ID, device ID. If traffic concentrates on one key, the per-partition ceiling finds you regardless of capacity mode. No good key means no DynamoDB.</li>



<li><strong>What&#8217;s the concurrency shape of the compute tier?</strong> Thousands of short-lived Lambda invocations argue for DynamoDB. A few long-running containers with a warm pool remove the strongest argument against RDS.</li>



<li><strong>Does the workload go quiet?</strong> Bursty and mostly idle favors on-demand DynamoDB or a scale-to-zero Aurora cluster. Steady traffic makes provisioned capacity or a right-sized instance cheaper.</li>



<li><strong>Who maintains this in a year?</strong> If it&#8217;s one part-time developer, weight familiarity heavily. A design nobody on the team can debug is a liability whatever the benchmark said.</li>
</ol>



<h2 class="wp-block-heading">The cost model nobody actually models</h2>



<p class="wp-block-paragraph">The two bill on axes that don&#8217;t compare. RDS charges instance hours, storage, and depending on engine and storage type, I/O, busy or not. DynamoDB charges request units and storage, so an idle table is nearly free and a chatty one isn&#8217;t.</p>



<p class="wp-block-paragraph">The number that matters is request units per user action, and almost nobody calculates it. Reads bill in 4 KB increments and writes in 1 KB increments, strongly consistent reads cost twice what eventually consistent ones do, and transactional operations double again. Every index you project into consumes write units on every write that touches it. A screen firing six queries plus an audit write can cost many times the naive estimate.</p>



<p class="wp-block-paragraph">Work it out per action, multiply by real traffic, and sanity check the instance side with my <a href="https://john-nessime.com/tools/aws-cost-estimator/">AWS cost estimator</a> before the decision hardens. AWS recommends on-demand as the default capacity mode for most workloads, but steady predictable traffic is still cheaper provisioned, and you can flip between them:</p>



<pre class="wp-block-code"><code># which mode is this table on?
aws dynamodb describe-table --table-name AppData 
  --query "Table.BillingModeSummary.BillingMode"

# what is it actually consuming?
aws cloudwatch get-metric-statistics 
  --namespace AWS/DynamoDB 
  --metric-name ConsumedWriteCapacityUnits 
  --dimensions Name=TableName,Value=AppData 
  --statistics Sum --period 3600 
  --start-time $(date -u -d '1 day ago' +%Y-%m-%dT%H:%M:%SZ) 
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ)</code></pre>



<p class="wp-block-paragraph">Switching billing mode is rate limited, so treat it as a planned change rather than an incident-time knob.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<ul class="wp-block-list">
<li><strong>&#8220;DynamoDB scales infinitely.&#8221;</strong> The service does. Your partition key might not. Uneven access concentrates load on one partition, and adaptive capacity only helps within limits.</li>



<li><strong>&#8220;Postgres can&#8217;t handle our scale.&#8221;</strong> Usually said before anyone has run <code>EXPLAIN (ANALYZE, BUFFERS)</code> on the slow query. A missing index and an N+1 in the ORM explain most claims of outgrowing relational.</li>



<li><strong>&#8220;We&#8217;ll add the indexes later.&#8221;</strong> True in Postgres, expensive in DynamoDB. A new global secondary index backfills across the whole table and then taxes every subsequent write.</li>



<li><strong>&#8220;Serverless database means no operations.&#8221;</strong> You trade instance operations for capacity and key-design operations. Throttling, hot partitions and stream consumer lag all need dashboards, exactly like a relational instance does.</li>



<li><strong>&#8220;We can just use both.&#8221;</strong> Sometimes right, often a way to avoid deciding. Two stores means two consistency models, two restore procedures, and a sync path that becomes the least reliable component you own.</li>
</ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<p class="wp-block-paragraph">My default for a new application with an unfinished product definition is managed Postgres. Not because it&#8217;s better, but because being wrong about a relational schema costs an afternoon and being wrong about a key design costs a migration.</p>



<p class="wp-block-paragraph">I&#8217;d reach for DynamoDB first when the workload is a small closed set of patterns keyed off one identifier, with high write volume, tolerance for eventual consistency, and concurrency that would flatten a connection pool. Session stores, telemetry ingestion, per-user feeds, idempotency keys, feature flags. It&#8217;s excellent at all of those.</p>



<p class="wp-block-paragraph">The hybrid worth doing is narrow: relational as the system of record, DynamoDB for one high-volume pattern that joins nothing. The hybrid to avoid splits related entities across both stores and reconciles them in application code.</p>



<p class="wp-block-paragraph">Whichever you land on, instrument it on day one. Throttled requests and consumed capacity on the DynamoDB side, connection counts and slow queries on the relational side, on a CloudWatch or Grafana Cloud dashboard someone actually opens. If you&#8217;re restricting item-level access with IAM condition keys such as <code>dynamodb:LeadingKeys</code>, run the policy through my <a href="https://john-nessime.com/tools/aws-iam-policy-checker/">IAM policy checker</a> before it ships.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is DynamoDB always faster than RDS?</h3>



<p class="wp-block-paragraph">For a primary-key lookup at large scale its latency is more consistent, because that operation&#8217;s cost doesn&#8217;t grow with table size. For a query it wasn&#8217;t modeled for it can be far slower, because the fallback is a scan. Speed depends on the match between query and key design, not on the service.</p>



<h3 class="wp-block-heading">Can I run analytics on DynamoDB?</h3>



<p class="wp-block-paragraph">Not directly, and you shouldn&#8217;t try. Export the table to S3 or stream changes out, then query with Athena or load a warehouse. Treat operational and analytical stores as separate systems.</p>



<h3 class="wp-block-heading">What about Aurora Serverless as the middle ground?</h3>



<p class="wp-block-paragraph">It solves idle cost, not connections or write scaling. If you&#8217;re considering DynamoDB because of Lambda concurrency or single-primary write limits, Aurora Serverless doesn&#8217;t address that. If it&#8217;s because you&#8217;re paying for an idle dev database, it does.</p>



<h3 class="wp-block-heading">How do I migrate from DynamoDB to RDS if I get it wrong?</h3>



<p class="wp-block-paragraph">Export to S3, reshape denormalized items into normalized tables, load, and dual-write during cutover. The data movement is routine. The hard part is that your application code assumed a key-value access shape everywhere, so the rewrite lands in the service layer, not the pipeline.</p>



<h3 class="wp-block-heading">Is DynamoDB cheaper than RDS for a small application?</h3>



<p class="wp-block-paragraph">Usually yes when traffic is low and bursty, because on-demand charges per request against an idle table while an instance bills continuously. The advantage narrows as traffic steadies and reverses once throughput justifies provisioned capacity or a reserved instance.</p>



<h3 class="wp-block-heading">Do I need RDS Proxy if I use Lambda with Postgres?</h3>



<p class="wp-block-paragraph">At meaningful concurrency, yes. Below that, a direct connection reused across invocations is fine, and the proxy adds cost plus a pinning failure mode. Add it when connection counts approach your limit, not preemptively in development.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">The one thing worth remembering about RDS vs DynamoDB: you&#8217;re not choosing a performance profile, you&#8217;re choosing when to pay the modeling cost. DynamoDB takes it up front and pays you back in predictability. A relational database lets you defer it and charges interest later in query complexity.</p>



<p class="wp-block-paragraph">Write the access pattern inventory. Be honest about the admin screens and the reporting nobody has asked for yet. If the list is short, stable and keyed on one identifier, DynamoDB will serve you for years. If it isn&#8217;t, take the relational database and spend the saved effort on indexes and connection handling.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second opinion on the data store decision?</h2>



<p class="wp-block-paragraph">Cheap to review now, expensive to revisit later. Things I help with:</p>



<ul class="wp-block-list">
<li>Building the access pattern inventory with your team and pressure-testing it against the queries your roadmap implies</li>



<li>Reviewing a DynamoDB key and index design before it ships, including partition key distribution and index write amplification</li>



<li>Diagnosing throttling and hot partitions on an existing table, and whether the fix is a key change, a capacity change or a cache</li>



<li>Sizing RDS or Aurora for a serverless compute tier, including connection handling, RDS Proxy configuration and pinning</li>



<li>Modeling the cost of both options against your real request volume instead of a rule of thumb</li>



<li>Planning a migration in either direction, with a dual-write cutover and a rollback that works</li>
</ul>



<p class="wp-block-paragraph">Send me your access pattern list, a slow query plan, or your throttling metrics and I&#8217;ll tell you what I&#8217;d do with it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/rds-vs-dynamodb-access-patterns/">RDS vs DynamoDB: The Benchmark Isn&#8217;t the Decision</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Turning External Documents Into Structured Data With Amazon Textract</title>
		<link>https://john-nessime.com/blog/technical-guides/amazon-textract-structured-data/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 22 Sep 2026 06:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon SNS]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Bedrock Data Automation]]></category>
		<category><![CDATA[Boto3]]></category>
		<category><![CDATA[Confidence Scoring]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Human In The Loop]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[OCR]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[Serverless]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=390</guid>

					<description><![CDATA[<p>Amazon Textract almost never fails loudly. A working guide to turning external PDFs and scans into structured data: picking the right operation per document family, parsing the block graph, routing on per-field confidence, and catching the limits that silently truncate your records.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/amazon-textract-structured-data/">Turning External Documents Into Structured Data With Amazon Textract</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A vendor sends over a year of delivery notes. It arrives as one PDF, several hundred pages, scanned on a machine that was clearly overdue for service. Someone asks how long it will take to get the reference numbers into the warehouse table, and the honest answer is that the OCR call is the easy part.</p>



<p class="wp-block-paragraph">Document pipelines are not hard because optical character recognition is inaccurate. Amazon Textract reads text well. They are hard because everything after the API call belongs to you, and because Textract almost never fails loudly. You get a 200 response, a list of blocks, a confidence score, and a record with an empty field where the purchase order number should have been. Nothing alerts. The row lands in the warehouse with a null, and someone in finance notices three weeks later while reconciling.</p>



<p class="wp-block-paragraph">That silent gap is the whole problem. Getting reliable Amazon Textract structured data out of files you did not design has less to do with OCR accuracy than with everything around it: which operation to call for which document family, why the response is a graph rather than a record, how to route on confidence instead of trusting it, and the limits that quietly truncate your data. All of it assumes documents you cannot ask anyone to change.</p>



<h2 class="wp-block-heading">Why external documents break pipelines that passed testing</h2>



<p class="wp-block-paragraph">Internal documents are generated by a template you control. External documents vary along three axes at once, and each one has its own failure mode:</p>



<ul class="wp-block-list">
<li><strong>Container.</strong> One logical document per file, or forty stapled into one PDF, or a single page split across three JPEGs.</li>

<li><strong>Text origin.</strong> Born-digital PDFs with a real text layer, flatbed scans, and photographs taken at an angle in bad light. Same file extension, wildly different accuracy.</li>

<li><strong>Layout stability.</strong> The vendor redesigns their invoice template and moves the PO number from the header to a footer block. Nobody tells you.</li>
</ul>



<p class="wp-block-paragraph">You cannot fix any of that upstream. You can only detect it at ingest and route accordingly. The sections below are organized by the failure family each one produces.</p>



<h2 class="wp-block-heading">Failure family one: the file is out of spec before Textract sees it</h2>



<p class="wp-block-paragraph">Textract accepts JPEG, PNG, PDF, and TIFF. It does not support XFA-based PDFs, and it will not open a password-protected PDF. Images are accepted up to 10,000 pixels on any side. For PDFs, the maximum page dimensions are 40 inches and 9,000 points.</p>



<p class="wp-block-paragraph">Two more constraints catch people out. Text detection covers English, French, German, Italian, Portuguese, and Spanish, and several capabilities are narrower still: handwriting, invoices and receipts, identity documents, and Queries are English only. Vertical text, common in Japanese and Chinese documents, is not supported at all. Rotation is fine, including heavy in-plane rotation. Vertical writing direction is not.</p>



<p class="wp-block-paragraph">Build an ingest gate that checks encryption, page count, byte size, and dimensions before it spends an API call, and give each rejection a reason code a human can act on. Then catch exceptions properly instead of wrapping the call in a bare retry. <code>BadDocumentException</code> means Textract could not read the file, and retrying fails identically. <code>DocumentTooLargeException</code> means route to the asynchronous path or split the file. <code>InvalidS3ObjectException</code> is usually IAM or a bucket-region mismatch, not a document problem. Only <code>InternalServerErrorException</code> deserves a retry with backoff.</p>



<h2 class="wp-block-heading">Failure family two: synchronous versus asynchronous is a correctness gate</h2>



<p class="wp-block-paragraph">This is the one that silently eats data, so it goes early.</p>



<p class="wp-block-paragraph">The synchronous operations, <code>DetectDocumentText</code> and <code>AnalyzeDocument</code>, cap at 10 MB in memory. For PDF and TIFF input they also cap at <strong>one page</strong>. Not one page per call with automatic paging. One page, full stop. Hand a synchronous call a 40-page PDF and you get page one, and a pipeline that reports success.</p>



<p class="wp-block-paragraph">The asynchronous operations are a different shape. PDF and TIFF go up to 500 MB and 3,000 pages, while JPEG and PNG stay at 10 MB. You start a job, Textract works in the background, and you collect results by job ID.</p>



<p class="wp-block-paragraph">A synchronous call looks like this. The document lives in S3, and you name the features you want:</p>



<pre class="wp-block-code"><code>aws textract analyze-document 
  --document '{"S3Object":{"Bucket":"my-doc-bucket","Name":"inbound/note-0042.png"}}' 
  --feature-types '["FORMS","TABLES"]' 
  --region us-east-1</code></pre>



<p class="wp-block-paragraph">The asynchronous version returns a job identifier immediately and publishes completion to an SNS topic. That is what you want in production, because polling a job inside a Lambda function means paying for a function that sleeps:</p>



<pre class="wp-block-code"><code>import boto3

textract = boto3.client("textract")

start = textract.start_document_analysis(
    DocumentLocation={
        "S3Object": {"Bucket": "my-doc-bucket", "Name": "inbound/batch-0042.pdf"}
    },
    FeatureTypes=["FORMS", "TABLES"],
    NotificationChannel={
        "SNSTopicArn": "arn:aws:sns:us-east-1:111122223333:textract-jobs",
        "RoleArn": "arn:aws:iam::111122223333:role/TextractSnsPublishRole",
    },
)

job_id = start["JobId"]</code></pre>



<p class="wp-block-paragraph">Results come back paginated, and forgetting the continuation token is a second, quieter way to lose most of a file:</p>



<pre class="wp-block-code"><code>blocks = []
next_token = None

while True:
    kwargs = {"JobId": job_id}
    if next_token:
        kwargs["NextToken"] = next_token

    page = textract.get_document_analysis(**kwargs)
    blocks.extend(page["Blocks"])

    next_token = page.get("NextToken")
    if not next_token:
        break</code></pre>



<p class="wp-block-paragraph">Check <code>JobStatus</code> before you trust <code>Blocks</code>. A job can come back partially successful, which means some pages processed and some did not. If you treat that as success you will publish a document that is missing its middle.</p>



<h2 class="wp-block-heading">Failure family three: the response is a graph, not a record</h2>



<p class="wp-block-paragraph">Textract does not return fields. It returns a flat list of <code>Block</code> objects, each with an ID and relationships pointing at other IDs. Reassembling those into a row is your code, and it is where most of the real work lives.</p>



<p class="wp-block-paragraph">For form data, the shape is specific. A key-value pair is two blocks of type <code>KEY_VALUE_SET</code>, distinguished by their <code>EntityTypes</code> array containing either <code>KEY</code> or <code>VALUE</code>. The key block carries a relationship of type <code>VALUE</code> holding the ID of its value block, and a relationship of type <code>CHILD</code> holding the IDs of the <code>WORD</code> blocks that spell out the label. The value block carries only <code>CHILD</code> relationships pointing at its own words. So reading one field means three hops.</p>



<pre class="wp-block-code"><code>blocks_by_id = {b["Id"]: b for b in blocks}

def child_text(block):
    parts = []
    for rel in block.get("Relationships", []):
        if rel["Type"] != "CHILD":
            continue
        for cid in rel["Ids"]:
            child = blocks_by_id[cid]
            if child["BlockType"] == "WORD":
                parts.append(child["Text"])
            elif child["BlockType"] == "SELECTION_ELEMENT":
                parts.append(child["SelectionStatus"])
    return " ".join(parts)

fields = {}
for b in blocks:
    if b["BlockType"] != "KEY_VALUE_SET":
        continue
    if "KEY" not in b.get("EntityTypes", []):
        continue

    value_text = ""
    for rel in b.get("Relationships", []):
        if rel["Type"] == "VALUE":
            for vid in rel["Ids"]:
                value_text = child_text(blocks_by_id[vid])

    fields[child_text(b)] = (value_text, b["Confidence"])</code></pre>



<p class="wp-block-paragraph">Two things about that snippet. The <code>SELECTION_ELEMENT</code> branch matters because checkboxes and radio buttons are values too, and their content is a selection status rather than text. And the confidence on the key block is not a separate key confidence: Textract returns the same value for both halves of a pair. If you planned to compare label confidence against value confidence to catch mismatches, that signal does not exist.</p>



<p class="wp-block-paragraph">Tables follow the same pattern with different block types, and cells carry row and column indices plus span values for merged cells. Layout adds another set, including title, header, footer, section header, page number, list, and figure blocks, and it sequences text into human reading order. That last property is why Layout is worth requesting when the output is headed for a language model rather than a database: chunking on reading order beats chunking on raw line order.</p>



<p class="wp-block-paragraph">Before hand-rolling any of this, look at the AWS-maintained parsers. The <code>amazon-textract-response-parser</code> package and the higher-level <code>amazon-textract-textractor</code> library both turn the block graph into objects with sensible properties. Writing your own traversal is a good way to understand the model and a poor way to spend two sprints.</p>



<h2 class="wp-block-heading">Getting Amazon Textract structured data out of the right API</h2>



<p class="wp-block-paragraph">Textract is a family of operations, not one endpoint, and the biggest single accuracy win available is usually picking a narrower one. Features are billed per page and they stack, so requesting <code>TABLES</code> on a corpus with no tables costs money and returns noise.</p>



<ul class="wp-block-list">
<li><strong>DetectDocumentText.</strong> Raw text and geometry. Use it when you only need a search index or a text layer.</li>

<li><strong>AnalyzeDocument with FORMS.</strong> Key-value pairs discovered from labels on the page. Works when the label is physically present and stable.</li>

<li><strong>AnalyzeDocument with TABLES.</strong> Grid reconstruction with row and column indices. The right call for line-item grids and statements.</li>

<li><strong>AnalyzeDocument with QUERIES.</strong> You ask natural language questions and get answers back. This is the fix for layout drift.</li>

<li><strong>AnalyzeDocument with LAYOUT and SIGNATURES.</strong> Reading order and structural elements; signature locations for compliance checks.</li>

<li><strong>AnalyzeExpense.</strong> Purpose-built for invoices and receipts, with normalized field types.</li>

<li><strong>AnalyzeID.</strong> Purpose-built for identity documents.</li>
</ul>



<h3 class="wp-block-heading">Why Queries beats FORMS on documents you don&#8217;t control</h3>



<p class="wp-block-paragraph">FORMS gives you whatever labels the document happens to contain. That means your downstream schema is hostage to a vendor&#8217;s wording. When they change &#8220;P.O. No.&#8221; to &#8220;Order Reference&#8221;, your parser looks for a key that no longer exists and writes a null.</p>



<p class="wp-block-paragraph">Queries inverts this. You supply the question and an alias, and the alias comes back attached to the answer, so your output key is fixed by you rather than by the document:</p>



<pre class="wp-block-code"><code>aws textract analyze-document 
  --document '{"S3Object":{"Bucket":"my-doc-bucket","Name":"inbound/note-0042.png"}}' 
  --feature-types '["QUERIES"]' 
  --queries-config '{"Queries":[
      {"Text":"What is the purchase order number?","Alias":"PO_NUMBER"},
      {"Text":"What is the delivery date?","Alias":"DELIVERY_DATE"}
  ]}' 
  --region us-east-1</code></pre>



<p class="wp-block-paragraph">The alias is the entire point. It is a contract between your schema and the extraction layer that survives a vendor redesign. There are ceilings: 15 queries per page synchronously, 30 per page asynchronously, English only. Phrase questions using words that actually appear on the page, and add positional hints when you know the layout, because the model uses them. If Queries underperforms on your corpus, adapters let you tune the feature against your own examples and pass an adapter ID and version at call time.</p>



<h3 class="wp-block-heading">Invoices are their own problem</h3>



<p class="wp-block-paragraph">For invoices and receipts, reach for AnalyzeExpense first. Its response is shaped around the domain rather than around the page: each <code>ExpenseDocument</code> holds <code>SummaryFields</code> for header-level data and <code>LineItemGroups</code> containing the line items. Fields carry a normalized <code>Type</code> such as <code>VENDOR_NAME</code>, plus a <code>ValueDetection</code> and an optional <code>LabelDetection</code>.</p>



<p class="wp-block-paragraph">That optional label is the thing to notice. AnalyzeExpense can identify a vendor name printed only inside a logo, and pull line items out of a grid with no column headers. FORMS structurally cannot do either, because FORMS needs a key to exist. On receipts that is not a small accuracy delta, it is the difference between a usable field and an empty one.</p>



<h2 class="wp-block-heading">Failure family four: confidence is per field and it is not your risk model</h2>



<p class="wp-block-paragraph">Every extracted element carries a confidence score. Two mistakes follow from that.</p>



<p class="wp-block-paragraph">The first is averaging into a document-level score. A document where every field is strong except the invoice total is not a mostly-good document. It is a document with one field that must not be trusted, surrounded by noise. Score fields, not documents.</p>



<p class="wp-block-paragraph">The second is treating one threshold as universal. A misread address on a delivery note is an annoyance. A misread bank account number is an incident. Set thresholds per field from the cost of being wrong, then route into three lanes:</p>



<ol class="wp-block-list">
<li><strong>Auto-accept.</strong> Above threshold and passing a format check. Straight into the target table.</li>

<li><strong>Review.</strong> Below threshold, or above it but failing a validation rule. A human sees the value and the cropped region it came from.</li>

<li><strong>Reject.</strong> Field absent entirely, or the document failed the ingest gate. Back to the sender with a reason.</li>
</ol>



<p class="wp-block-paragraph">Validation rules do more work than thresholds here. A total that does not equal the sum of its line items is wrong at any confidence. Check-digit rules on account and tax identifiers catch character-level errors the model was perfectly confident about. Confidence tells you how sure the model is. Validation tells you whether the answer is even possible.</p>



<p class="wp-block-paragraph">Textract can hand low-confidence results to Amazon Augmented AI through a human loop configuration on the API call, which saves building a review queue. Whether that beats a small internal tool depends on how much domain context your reviewers need on screen. Either way, capture the corrections. Reviewer edits are the only labeled data you get for free, and they are how you calibrate thresholds against your own corpus instead of guessing.</p>



<h2 class="wp-block-heading">Failure family five: reprocessing you didn&#8217;t budget for</h2>



<p class="wp-block-paragraph">Textract bills per page, per feature. Your parser, meanwhile, will change repeatedly in the first months of a pipeline&#8217;s life, because that is what parsers do.</p>



<p class="wp-block-paragraph">So separate extraction from parsing, permanently. Write the raw response JSON to S3 keyed by a hash of the document bytes plus the operation and feature set requested, then parse from that stored artifact. Reprocessing becomes free, deterministic, and diffable: you can replay a parser change across the whole back catalogue and compare outputs before promoting it. Without that cache, every parser bug fix is a bill.</p>



<pre class="wp-block-code"><code>import hashlib

digest = hashlib.sha256(doc_bytes).hexdigest()
features = "-".join(sorted(["FORMS", "TABLES"]))
key = f"textract-raw/{digest}/analyze-document/{features}.json"</code></pre>



<p class="wp-block-paragraph">Hashing content rather than filename also deduplicates the same scan arriving twice under different names, which happens constantly with email-driven intake. For the asynchronous operations, a client request token gives idempotency at the job level; reusing a token with different parameters raises a mismatch error rather than quietly starting a second job.</p>



<p class="wp-block-paragraph">One caveat: source documents often contain personal data, so a cache is a retention decision. Set S3 lifecycle rules on both the originals and the raw JSON, and if working copies land on a workstation or a build box, delete them properly. Tools like O&amp;O SafeErase exist for exactly that gap between &#8220;moved to trash&#8221; and &#8220;actually gone&#8221;.</p>



<h2 class="wp-block-heading">Where Textract stops being the right tool</h2>



<p class="wp-block-paragraph">Being fair about this matters, because AWS now offers overlapping paths to the same outcome. Amazon Bedrock Data Automation is a managed document processing service that handles classification and extraction behind one API, priced per document rather than per page and per feature. AWS positions it as the default starting point for new builds, and for mixed document types with no dominant format that recommendation is sound. You write far less orchestration.</p>



<p class="wp-block-paragraph">Textract keeps the advantage where the corpus is high volume and standardized, where geometry and per-field confidence are needed for auditability, where regional availability constrains you, or where you want extraction to be deterministic and separately testable. Bounding boxes and confidence scores are not a nice-to-have in a regulated workflow, they are the evidence trail.</p>



<p class="wp-block-paragraph">The hybrid is common and sensible: Textract for extraction, a Bedrock model for reasoning over the extracted text. Just do not send a multimodal model a full-page image because OCR felt inconvenient. It costs considerably more per page and gives you no geometry.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Only the first page came through.</strong> A synchronous call on a multi-page PDF or TIFF. Move to the asynchronous operations, and assert the page count Textract reports against the count you expected.</li>

<li><strong>Output truncates partway through a long document.</strong> Missing continuation token handling on the get operation. Loop until the token is absent.</li>

<li><strong>Fields visible on the page never appear in output.</strong> The label is implied rather than printed, or sits inside a logo. Switch that field to Queries, or to AnalyzeExpense if it is an invoice.</li>

<li><strong>Quality collapsed for one sender only.</strong> Template change. Diff the current layout against a stored sample, which is the argument for keeping raw responses.</li>

<li><strong>Throttling during a backfill.</strong> Transactions per second and concurrent job counts are per-account, per-region quotas. Queue submissions rather than fanning out, and request an increase before the migration.</li>

<li><strong>Table cells land in the wrong columns.</strong> Usually a skewed scan. Deskew and raise contrast before submitting, because image preparation is cheaper than post-processing repairs.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Testing on three clean sample documents supplied by the vendor rather than a random sample of real intake.</li>

<li>Requesting every feature type on every page because it is simpler than routing, then being surprised by the bill.</li>

<li>Using FORMS for invoices when AnalyzeExpense exists and understands unlabeled fields.</li>

<li>Treating a confidence score as an accuracy guarantee instead of pairing it with validation rules.</li>

<li>Discarding the raw JSON after parsing, which turns every parser fix into a re-extraction charge.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Classify before you extract. Route each document to the narrowest API that covers it.</li>

<li>Use Queries with explicit aliases for fields your schema depends on, so vendor wording changes cannot break your column names.</li>

<li>Store raw responses immutably, keyed by content hash, and parse from storage.</li>

<li>Set per-field confidence thresholds from the cost of being wrong, and pair every threshold with a format or arithmetic validation.</li>

<li>Instrument it like any other pipeline: extraction latency, straight-through rate, review queue depth, per-field null rate. A slow rise in nulls on one field is the earliest signal of a template change. Prometheus with Grafana, or Grafana Cloud if you would rather not run it yourself, covers this well.</li>

<li>Keep documents and pipeline in the same region, use VPC endpoints for sensitive data, and apply lifecycle rules to originals and extracted output alike.</li>

<li>Keep a golden set of real documents with known correct answers, and run it on every parser change.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can Amazon Textract process multi-page PDFs?</h3>



<p class="wp-block-paragraph">Yes, but only through the asynchronous operations. Synchronous calls accept a single page for PDF and TIFF input. Asynchronous processing handles PDF and TIFF up to 500 MB and 3,000 pages.</p>



<h3 class="wp-block-heading">What is the difference between FORMS and Queries?</h3>



<p class="wp-block-paragraph">FORMS returns whatever key-value pairs it finds from labels printed on the page, so your output keys follow the document. Queries lets you ask for values by natural language question and attach your own alias to each answer, so output keys stay fixed when the layout changes. For a schema you must keep stable, Queries is the safer contract.</p>



<h3 class="wp-block-heading">How much does Amazon Textract cost?</h3>



<p class="wp-block-paragraph">Billing is per page and varies by operation and feature, and requesting several features on one page adds their costs together. Check the current rate card for your region and multiply by pages, not documents. The practical lever is routing: send each document to the narrowest operation that answers your question, and cache raw responses so parser iteration is free.</p>



<h3 class="wp-block-heading">Does Textract handle handwriting?</h3>



<p class="wp-block-paragraph">It detects handwritten text, in English only. Treat handwritten fields as a separate confidence tier with lower auto-accept thresholds, because the variance across writers is much wider than for printed text.</p>



<h3 class="wp-block-heading">Which languages does Amazon Textract support?</h3>



<p class="wp-block-paragraph">Text detection covers English, French, German, Italian, Portuguese, and Spanish. Handwriting, invoices and receipts, identity documents, and Queries are English only. Vertically written text is not supported.</p>



<h3 class="wp-block-heading">Should I use Textract or Bedrock Data Automation?</h3>



<p class="wp-block-paragraph">Bedrock Data Automation is the lower-effort default for mixed document types, with classification and extraction behind a single managed API. Textract fits better at high volume on standardized documents, where per-field confidence and geometry matter for audit, or where extraction needs to be deterministic and independently testable. Combining them is normal.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">If one thing survives this post, make it this: Amazon Textract structured data extraction fails quietly, not loudly. The API returns success while your record is missing its most important field, because a synchronous call read one page of forty, or because a vendor renamed a label, or because a value came back at low confidence and nothing was watching.</p>



<p class="wp-block-paragraph">So build for the silence. Gate at ingest, use asynchronous operations by default, pick the narrowest API per document family, pin your schema with query aliases, store the raw response, and route on per-field confidence backed by validation rules that know what a possible answer looks like. Do that and the OCR really does become the easy part.</p>



<h2 class="wp-block-heading">Need help turning documents into a pipeline you can trust?</h2>



<p class="wp-block-paragraph">Most of the difficulty in document extraction shows up after the proof of concept, once real intake replaces the sample files. That is the part I work on:</p>



<ul class="wp-block-list">
<li>Designing the ingest gate and classification step so each document family reaches the right Textract operation instead of one call doing everything.</li>

<li>Building the asynchronous path properly: S3 events, SNS completion, paginated retrieval, dead-letter handling, idempotent reprocessing.</li>

<li>Replacing brittle FORMS parsing with query aliases or AnalyzeExpense, and validating the change against a golden set of your own documents.</li>

<li>Setting per-field confidence thresholds and validation rules, then wiring a review queue whose corrections feed back into tuning.</li>

<li>Adding observability that catches template drift early: per-field null rates, straight-through rate, and review queue depth on a dashboard.</li>

<li>Getting the cost curve under control through routing, raw-response caching, and lifecycle rules on stored documents and JSON.</li>
</ul>



<p class="wp-block-paragraph">If you have a pipeline misbehaving, send me a sample document with the fields you need and the raw Textract JSON it produced. That is usually enough to see where it is going wrong.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/amazon-textract-structured-data/">Turning External Documents Into Structured Data With Amazon Textract</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Amazon Textract Data Extraction: What Breaks on Real Contracts and Reports</title>
		<link>https://john-nessime.com/blog/devops/amazon-textract-data-extraction/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 20 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon SNS]]></category>
		<category><![CDATA[Amazon SQS]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Batch Processing]]></category>
		<category><![CDATA[Boto3]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Legal Tech]]></category>
		<category><![CDATA[OCR]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Serverless]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=270</guid>

					<description><![CDATA[<p>Textract rarely fails loudly. It returns a plausible result that is quietly incomplete: a truncated result set, a tick box read as an empty string, a clause split across a page break. A practitioner's guide to the failure modes that actually bite when you point Amazon Textract at contracts, technical reports and correspondence, plus how to choose between sync and async, which feature types are worth paying for, and where Textract stops being the right tool.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/amazon-textract-data-extraction/">Amazon Textract Data Extraction: What Breaks on Real Contracts and Reports</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The pipeline passed every test. Three sample agreements, all the right fields, clean JSON out the other side. Then it ran against the real archive and somebody in legal noticed that every contract longer than about forty pages was missing its termination clause. No errors. No failed jobs. The dashboard was green the whole time.</p>



<p class="wp-block-paragraph">The bug was a missing loop. The job had finished, the results were there, and the code had read the first chunk of them and stopped.</p>



<p class="wp-block-paragraph">That is the shape of most problems with Amazon Textract data extraction on real documents. The service rarely fails loudly. It returns a plausible-looking result that is quietly incomplete, and you find out weeks later when somebody asks a question the data cannot answer. This post covers the failures that actually bite on contracts, technical reports and correspondence: sync versus async, reading the block graph without losing fields, which feature types are worth paying for, and where Textract stops being the right tool.</p>



<h2 class="wp-block-heading">Why Amazon Textract data extraction is not just OCR</h2>



<p class="wp-block-paragraph">Plain OCR gives you words and their positions. Textract gives you a graph. Every response is a flat array of <code>Block</code> objects, and each block carries an ID plus relationships to other block IDs. A page relates to its lines, a line to its words, a key to its value, a table to its cells.</p>



<p class="wp-block-paragraph">That structure is the whole point, and it is where the pain lives. Nothing is nested for you. To read one form field you find the <code>KEY_VALUE_SET</code> block with entity type <code>KEY</code>, follow its <code>VALUE</code> relationship to another block, then follow that block&#8217;s <code>CHILD</code> relationships to the words. Three hops, and getting a hop wrong returns an empty string rather than an exception.</p>



<p class="wp-block-paragraph">Contracts and reports make this harder than invoices do. An invoice has a total. A master services agreement has a liability cap buried in a numbered sub-clause that spans a page break.</p>



<h2 class="wp-block-heading">Synchronous or asynchronous: the choice is made for you</h2>



<p class="wp-block-paragraph">People reach for <code>AnalyzeDocument</code> first because it returns results in the same call and is easy to test in a notebook. Then they hit the quotas, which AWS documents as hard limits you cannot raise. Synchronous operations cap JPEG, PNG, PDF and TIFF at 10 MB in memory, and cap PDF and TIFF at <strong>one page</strong>. Asynchronous operations keep JPEG and PNG at 10 MB but take PDF and TIFF up to 500 MB and 3,000 pages.</p>



<p class="wp-block-paragraph">One page. That single constraint decides your architecture. Any real contract goes through <code>StartDocumentAnalysis</code> and <code>GetDocumentAnalysis</code>, which means it goes through S3, which means you need somewhere to stage documents and a way to learn when the job is done.</p>



<p class="wp-block-paragraph">Three other constraints from the same quota page are worth knowing before you promise anything to a client:</p>



<ul class="wp-block-list">
<li>Text detection covers English, French, German, Italian, Portuguese and Spanish. Query detection is English only.</li>

<li>Vertical text is not supported. Rotation is fine, including odd in-plane angles, but vertically written scripts are not.</li>

<li>Password-protected PDFs are rejected and XFA-based PDFs are unsupported. Both turn up in legal archives more often than you would expect.</li>
</ul>



<p class="wp-block-paragraph">Pass a <code>ClientRequestToken</code> on submission. It is an idempotency token: the same token returns the same <code>JobId</code> instead of starting a duplicate job. If your submitter is a Lambda function behind an S3 event, and S3 events can be delivered more than once, that is the difference between paying once and paying twice.</p>



<pre class="wp-block-code"><code>import boto3

textract = boto3.client("textract")

job = textract.start_document_analysis(
    DocumentLocation={
        "S3Object": {"Bucket": "contracts-intake", "Name": "msa/acme-2.pdf"}
    },
    FeatureTypes=["FORMS", "TABLES"],
    ClientRequestToken="msa-acme-2-v1",
    NotificationChannel={
        "SNSTopicArn": "arn:aws:sns:eu-west-1:111122223333:textract-done",
        "RoleArn": "arn:aws:iam::111122223333:role/TextractSnsPublish",
    },
)</code></pre>



<p class="wp-block-paragraph">The notification channel is optional but you want it. Polling in a loop inside a Lambda function burns billed duration doing nothing, and Lambda&#8217;s execution ceiling will cut you off on long documents anyway. Publish to SNS, fan out to SQS, let a second function do the reading, and you get a natural place to hang a dead letter queue.</p>



<h2 class="wp-block-heading">The result set you never finished reading</h2>



<p class="wp-block-paragraph">This is the one that cost the termination clauses. <code>GetDocumentAnalysis</code> returns results in pages. When there are more blocks than fit in one response, the response carries a <code>NextToken</code> and you have to call again with it. If you do not, you get the beginning of the document and nothing tells you so. <code>JobStatus</code> still reports success, because the job did succeed. Your code just stopped reading.</p>



<p class="wp-block-paragraph">Short test documents fit in a single response. That is exactly why this survives testing and dies in production.</p>



<pre class="wp-block-code"><code>def fetch_all_blocks(textract, job_id):
    blocks, next_token = [], None

    while True:
        kwargs = {"JobId": job_id}
        if next_token:
            kwargs["NextToken"] = next_token

        response = textract.get_document_analysis(**kwargs)

        if response["JobStatus"] == "FAILED":
            raise RuntimeError(response.get("StatusMessage", "job failed"))

        blocks.extend(response["Blocks"])
        next_token = response.get("NextToken")

        if not next_token:
            return blocks</code></pre>



<p class="wp-block-paragraph">Then make the failure detectable. Treat the highest page number present in the blocks as a checksum against the page count of the source file. Submit a 60-page PDF, get blocks that stop at page 12, and something is wrong no matter what the job status says.</p>



<h2 class="wp-block-heading">Choosing feature types without paying for all of them</h2>



<p class="wp-block-paragraph"><code>AnalyzeDocument</code> and <code>StartDocumentAnalysis</code> take a <code>FeatureTypes</code> list: <code>FORMS</code>, <code>TABLES</code>, <code>QUERIES</code>, <code>SIGNATURES</code> and <code>LAYOUT</code>. This is not cosmetic. Textract bills per page and the per-page rate depends on which features you enabled, so turning all five on because you might need them later multiplies your bill across the whole archive. Rates change and vary by region, so check the current pricing page rather than trusting a number in a blog post. The mechanism does not change: each feature adds to the per-page cost, and you pay it on every page you submit, including blank separator sheets.</p>



<ul class="wp-block-list">
<li><strong>FORMS</strong> finds key-value pairs where the document has an explicit label. Good for cover sheets, signature blocks, report headers. Useless for prose clauses with no label.</li>

<li><strong>TABLES</strong> reconstructs rows, columns and cells, and also identifies table titles, footers and merged cells. This is what you want for rate cards and appendix tables.</li>

<li><strong>QUERIES</strong> lets you ask natural-language questions and get answers back under an alias you chose. This is the feature that makes contracts tractable.</li>

<li><strong>SIGNATURES</strong> returns the location and confidence of handwritten signatures, electronic signatures and initials. It tells you something is signed at a coordinate, not whose signature it is.</li>

<li><strong>LAYOUT</strong> groups text into titles, headers, footers, section headers, paragraphs and lists in reading order. Skippable on single-column documents. On two-column reports it is the difference between coherent text and interleaved nonsense.</li>
</ul>



<p class="wp-block-paragraph">A practical pattern: run a cheap text-detection pass to classify each document, then apply the expensive feature set only to the pages that matter. A 200-page appendix of scanned site photographs does not need FORMS and TABLES.</p>



<h2 class="wp-block-heading">Reading the block graph without losing fields</h2>



<p class="wp-block-paragraph">Here is the failure that produces silent empty strings. A key-value pair&#8217;s value is not always text. If the value is a tick box, the value block&#8217;s children include a <code>SELECTION_ELEMENT</code> block, and that block has no text at all. It has a <code>SelectionStatus</code> of <code>SELECTED</code> or <code>NOT_SELECTED</code>. Concatenate only <code>WORD</code> children and a ticked box comes back as an empty string, which looks exactly like a field nobody filled in.</p>



<pre class="wp-block-code"><code>def block_text(block, block_map):
    parts = []
    for rel in block.get("Relationships", []):
        if rel["Type"] != "CHILD":
            continue
        for child_id in rel["Ids"]:
            child = block_map[child_id]
            if child["BlockType"] == "WORD":
                parts.append(child["Text"])
            elif child["BlockType"] == "SELECTION_ELEMENT":
                parts.append(child["SelectionStatus"])
    return " ".join(parts)</code></pre>



<p class="wp-block-paragraph">Build <code>block_map</code> once as a dictionary keyed on block ID before you traverse anything. Scanning the block list linearly for each lookup turns a three-hop traversal into an accidental quadratic, which you will notice the first time a 3,000-page bundle goes through.</p>



<h2 class="wp-block-heading">Queries: asking for clauses that have no label</h2>



<p class="wp-block-paragraph">FORMS works when the document says &#8220;Effective Date:&#8221; next to the date. Contracts frequently do not. The governing law sits in a paragraph of prose halfway down page nine. Queries handles that. You pass questions in plain English, each with an <code>Alias</code> you choose, and the response pairs <code>QUERY</code> blocks with <code>QUERY_RESULT</code> blocks through an <code>ANSWER</code> relationship. The alias is what makes this usable downstream: you match on <code>governing_law</code>, not on the exact wording of the question, so you can reword questions without breaking your schema.</p>



<pre class="wp-block-code"><code>QueriesConfig = {
    "Queries": [
        {"Text": "What is the governing law?",
         "Alias": "governing_law", "Pages": ["*"]},
        {"Text": "What is the termination notice period?",
         "Alias": "termination_notice", "Pages": ["9-*"]},
    ]
}</code></pre>



<p class="wp-block-paragraph">The limits matter. AWS documents a maximum of 15 queries per page for synchronous operations and 30 per page for asynchronous ones. A fifty-field schema cannot be asked in one pass. You either split it across multiple calls, which multiplies per-page spend, or you narrow <code>Pages</code> so each query runs only where the answer plausibly lives. Signature blocks are at the end, definitions near the front. Running every query against every page is the most common way people accidentally triple their bill.</p>



<p class="wp-block-paragraph">Queries are pre-trained on a spread of business documents including paystubs, bank statements, loan applications and mortgage notes. Your niche contract template was not in that set. Expect good results on common concepts and mediocre ones on house-specific terminology.</p>



<h2 class="wp-block-heading">Documents that fight back</h2>



<h3 class="wp-block-heading">Clauses that span a page break</h3>



<p class="wp-block-paragraph">Textract analyses pages. A clause starting at the bottom of page 11 and finishing at the top of page 12 is two disconnected fragments as far as the block graph is concerned. Queries will often return the fragment on one page and miss the qualifier on the other, which is the worst outcome available because the answer looks complete. The fix is not clever: reassemble full text in reading order using LAYOUT, then run clause-level logic over that rather than over per-page answers. Use Queries to <em>locate</em> a clause and the reassembled text to <em>read</em> it.</p>



<h3 class="wp-block-heading">Two-column reports and email threads</h3>



<p class="wp-block-paragraph">Without LAYOUT, a two-column technical report reads as alternating lines from both columns. Everything downstream inherits that corruption and it passes a spot check, because individual lines look fine. LAYOUT sequences elements in reading order and returns block types for titles, headers, footers and section headers, which is also what you need to chunk a report by section.</p>



<p class="wp-block-paragraph">Printed email chains are quoted text inside quoted text, newest message at the top, the same signature block repeated five times. Textract extracts all of it faithfully, which is the problem. Splitting a thread into individual messages is a text-processing job you do afterwards, not something a feature type solves.</p>



<h3 class="wp-block-heading">PDFs that were never scanned</h3>



<p class="wp-block-paragraph">This is the cheapest win available and almost everybody misses it. A large share of contract archives are born-digital PDFs exported from a word processor, and they already contain a text layer. Running OCR over them pays a service to guess at text you could have read directly. Put a triage step in front of the pipeline; <code>pdftotext</code> from poppler-utils pulls the embedded layer if there is one:</p>



<pre class="wp-block-code"><code>pdftotext -layout contract.pdf - | head -c 2000</code></pre>



<p class="wp-block-paragraph">Substantial text back means born-digital: route it down a cheaper path and reserve Textract for genuinely scanned material. Triage across a large archive is an embarrassingly parallel batch job that sits better on a plain VPS from somewhere like Contabo or InterServer than on per-invocation serverless billing, since you are CPU-bound for minutes at a time rather than reacting to events. One caveat: born-digital does not mean clean. Some exporters produce a text layer with broken word spacing or mangled ligatures, so sample before you trust it.</p>



<h2 class="wp-block-heading">Confidence scores you can actually act on</h2>



<p class="wp-block-paragraph">Every block carries a confidence score and the instinct is to threshold on it. That is usually wrong. Confidence tells you how sure the model is about <em>which characters are on the page</em>, not whether the field is semantically right. Textract can read a date with total certainty and hand you the wrong one, because it picked up the printing date in the footer instead of the effective date in the recitals. High confidence, completely wrong answer.</p>



<p class="wp-block-paragraph">It is also per block. A key-value pair has a score on the key, another on the value and separate ones on each word, so there is no single number to gate on. Validate the extracted value instead. Does the date parse and fall in a plausible range? Does the total equal the sum of the line items? Does the party name appear in the signature block as well as the preamble? Route to human review on failed validation and treat low confidence as one input among several.</p>



<h2 class="wp-block-heading">When Custom Queries adapters are worth the effort</h2>



<p class="wp-block-paragraph">If pre-trained Queries keeps getting your house document type wrong, train an adapter. Create it with <code>CreateAdapter</code>, annotate sample documents, train a version with <code>CreateAdapterVersion</code>, then reference it at inference time.</p>



<pre class="wp-block-code"><code>AdaptersConfig = {
    "Adapters": [
        {"AdapterId": ADAPTER_ID, "Version": "1", "Pages": ["1-5"]},
        {"AdapterId": ADAPTER_ID, "Version": "1", "Pages": ["6-*"]},
    ]
}</code></pre>



<p class="wp-block-paragraph">That <code>Pages</code> string takes digits, hyphens and an asterisk with no blank spaces, a page can only have one adapter applied to it, and an asterisk meaning all pages must be the only element in the list.</p>



<p class="wp-block-paragraph">Two things to weigh. AWS sets a minimum of five samples per query level for training or testing and says plainly that more is better, so treat five as a demo floor rather than a number that survives layout variation. And successful trainings per month are capped per account, so the tight annotate-train-evaluate loop you want is not available. Adapters suit one high-volume, house-specific document type with a stable schema. They are a poor fit for a long tail of one-off templates, which is what most legal archives actually are.</p>



<h2 class="wp-block-heading">Where Textract stops and a language model starts</h2>



<p class="wp-block-paragraph">This is a real decision now, so both sides deserve a hearing. Textract&#8217;s case is strong on standardised, high-volume documents. It is deterministic in a way generative models are not, it returns bounding-box geometry so you can point at exactly where a value came from, it gives per-element confidence, and it bills predictably per page. When an auditor asks why a field has the value it does, geometry beats a model&#8217;s reasoning.</p>



<p class="wp-block-paragraph">The generative side, whether that is Amazon Bedrock Data Automation as a managed document processing service or a model called directly through Amazon Bedrock, wins where the task needs reasoning rather than pattern matching. &#8220;Does this agreement contain an assignment restriction, and if so summarise it&#8221; is a comprehension question, and Textract has no answer for it. Generative approaches also cope better with layouts that were in nobody&#8217;s training set.</p>



<p class="wp-block-paragraph">Most production pipelines use both, which is what AWS&#8217;s own reference architectures push: Textract for faithful extraction with geometry and confidence, then a model over the extracted text for classification and comprehension. Choosing from scratch, ask whether your documents are standardised and high-volume. If yes, Textract plus a thin post-processing layer is cheaper and more auditable. If they are varied and volume is modest, a managed generative service gets you working faster. Google Document AI and Azure AI Document Intelligence cover similar ground if you are not committed to AWS.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Extraction stops partway through long documents.</strong> You are not following <code>NextToken</code>. Compare the highest page number in your blocks against the source page count.</li>

<li><strong>Fields come back empty on forms you know were filled in.</strong> The value is a <code>SELECTION_ELEMENT</code> and you are only reading <code>WORD</code> children.</li>

<li><strong>The document is rejected as too large.</strong> You are on the synchronous API. Multi-page PDFs must go through <code>StartDocumentAnalysis</code>, and the 10 MB cap is on in-memory size, not file size on disk.</li>

<li><strong>Textract cannot read the document at all.</strong> Check for password protection and XFA-based forms. Both are documented as unsupported.</li>

<li><strong>The service cannot access the S3 object.</strong> Usually the execution role missing <code>s3:GetObject</code>, or a bucket in a different region from the Textract endpoint you called.</li>

<li><strong>Throttling under batch load.</strong> Transactions-per-second quotas are per account per region. Smooth spiky traffic through a queue and retry with exponential backoff and jitter rather than raising concurrency.</li>

<li><strong>Garbled text on reports that look fine to you.</strong> Multi-column layout without LAYOUT enabled, or a source scan below roughly 150 DPI. Re-scanning beats any amount of post-processing.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Testing only on short documents, which hides the pagination bug entirely.</li>

<li>Enabling every feature type on every page because it is easier than deciding.</li>

<li>Running OCR over born-digital PDFs that already carry a text layer.</li>

<li>Treating a confidence score as a correctness score.</li>

<li>Scanning the block array linearly instead of building an ID map.</li>

<li>Polling for job completion inside a Lambda function instead of using SNS.</li>

<li>Storing only extracted values and discarding the raw JSON, so reprocessing means paying again.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Persist the full Textract JSON to S3 keyed on a hash of the source document. Reprocessing then costs storage, not extraction.</li>

<li>Classify first, extract second. A cheap text-detection pass tells you which expensive feature set each document needs.</li>

<li>Validate on the value, not the score. Parse dates, check ranges, cross-reference names against other parts of the document.</li>

<li>Use aliases on every query and treat them as your schema contract.</li>

<li>Set a per-job page-count assertion and alert on mismatches. That is your canary for silent truncation.</li>

<li>Keep source documents in an encrypted bucket with a lifecycle policy. Contracts and correspondence should not accumulate indefinitely by accident.</li>

<li>Track cost per document rather than per API call. Feature stacking makes the per-call figure meaningless.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can Amazon Textract handle multi-page PDF contracts?</h3>



<p class="wp-block-paragraph">Yes, through the asynchronous API only. Synchronous operations cap PDF and TIFF at one page. Asynchronous operations accept PDF and TIFF up to 500 MB and 3,000 pages, staged through S3.</p>



<h3 class="wp-block-heading">Why is my Textract output missing the end of the document?</h3>



<p class="wp-block-paragraph">Almost certainly because you read the first response from <code>GetDocumentAnalysis</code> and stopped. Results are paginated with a <code>NextToken</code> and the job status still reports success, so nothing signals the truncation. Loop until <code>NextToken</code> is absent.</p>



<h3 class="wp-block-heading">How many Textract Queries can I ask per page?</h3>



<p class="wp-block-paragraph">AWS documents 15 per page for synchronous operations and 30 per page for asynchronous ones. Larger schemas need multiple passes, which costs more, so narrowing each query&#8217;s page range is worth the effort.</p>



<h3 class="wp-block-heading">Does Textract work on handwriting and signatures?</h3>



<p class="wp-block-paragraph">It detects handwriting, and the SIGNATURES feature returns the location and confidence of handwritten signatures, electronic signatures and initials. It does not verify identity: it tells you something was signed and where, not by whom.</p>



<h3 class="wp-block-heading">What languages does Amazon Textract support?</h3>



<p class="wp-block-paragraph">Text detection covers English, French, German, Italian, Portuguese and Spanish. Queries detection is English only, vertically written text is unsupported, and Textract does not return the detected language in its output.</p>



<h3 class="wp-block-heading">Should I use Textract or a generative model?</h3>



<p class="wp-block-paragraph">Textract when documents are standardised, volume is high, and you need auditable geometry and per-field confidence. A generative service when layouts vary widely, volume is modest, or the task needs comprehension rather than extraction. Most production pipelines use both.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Amazon Textract data extraction fails quietly far more often than it fails loudly. A truncated result set, a tick box read as an empty string, a clause split across a page break, a confidently extracted wrong date: none of these throw an exception and none show up on a dashboard.</p>



<p class="wp-block-paragraph">So build the assertions that make silence detectable. Compare expected page counts against extracted ones, validate values rather than trusting scores, and keep the raw JSON so you can prove what the service actually returned. Textract is genuinely good at reading documents. Your job is noticing when it did not read all of them.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a document extraction pipeline that does not lose fields?</h2>



<p class="wp-block-paragraph">I build and fix document processing pipelines on AWS. Typical work looks like this:</p>



<ul class="wp-block-list">
<li>Auditing an existing Textract pipeline for silent truncation, dropped selection elements and mis-scoped queries</li>

<li>Designing the async architecture end to end: S3 intake, SNS and SQS fan-out, Lambda or container workers, dead letter handling and retries</li>

<li>Cutting per-page cost by classifying first, triaging born-digital PDFs out of the OCR path and narrowing feature types per page range</li>

<li>Turning block-graph JSON into a clean schema in CSV, a relational database or a data lake, with validation rules and a review queue for exceptions</li>

<li>Building the hybrid path where Textract handles extraction and a model on Amazon Bedrock handles classification and comprehension</li>

<li>Adding the observability that makes quiet failures loud: page-count assertions, per-document cost tracking, confidence distribution monitoring</li>
</ul>



<p class="wp-block-paragraph">If you have a redacted sample document, a chunk of Textract JSON or an extraction coming back half empty, send it over and I will tell you what is going wrong with it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/amazon-textract-data-extraction/">Amazon Textract Data Extraction: What Breaks on Real Contracts and Reports</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Salesforce AWS Integration Architecture: The Four Paths and What Breaks Silently</title>
		<link>https://john-nessime.com/blog/devops/salesforce-aws-integration-architecture/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 19 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AppFlow]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Change Data Capture]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Event Relay]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[Named Credentials]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Platform Events]]></category>
		<category><![CDATA[PrivateLink]]></category>
		<category><![CDATA[Pub/Sub API]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[Serverless]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=209</guid>

					<description><![CDATA[<p>Most Salesforce AWS integrations don't fail loudly, they drift. This guide covers the four integration paths between Salesforce and AWS, when to use each, the silent failure mode hiding in every one of them, and the reconciliation job you have to build yourself.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-aws-integration-architecture/">Salesforce AWS Integration Architecture: The Four Paths and What Breaks Silently</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone in the ops channel asks why the opportunity count on the QuickSight dashboard doesn&#8217;t match the count in the Salesforce report. It&#8217;s off by a few hundred. Nobody can answer that, and worse, nobody can say when the two numbers stopped agreeing. Every pipeline is green. The scheduled flow reports successful runs. The EventBridge rule shows invocations. Everything is healthy except the data.</p>



<p class="wp-block-paragraph">That&#8217;s the shape of most Salesforce AWS integration problems. They&#8217;re rarely outages. They&#8217;re divergence, accumulating quietly while every status page stays green, and you find out about it when someone in finance runs a number past a customer.</p>



<p class="wp-block-paragraph">This guide covers the four integration paths between Salesforce and AWS, which one to reach for and when, the specific failure each one hides, and the reconciliation layer you&#8217;ll have to build yourself because neither vendor ships it.</p>



<h2 class="wp-block-heading">Why a Salesforce AWS integration fails silently</h2>



<p class="wp-block-paragraph">Start here, because it changes how you design everything downstream. There are three mechanisms that lose data without raising an error anywhere you&#8217;re looking.</p>



<p class="wp-block-paragraph"><strong>Events sent to an unassociated partner event bus are dropped on the floor.</strong> When you create an Event Relay, Salesforce creates a partner event source in your AWS account. Until you associate that source with an event bus, EventBridge discards anything the partner publishes to it. AWS is explicit that those events are not persisted at rest. There&#8217;s no queue holding them for you, no backlog to drain once you finish the setup. If the relay is running and the bus isn&#8217;t associated, you are streaming into a void.</p>



<p class="wp-block-paragraph"><strong>The Salesforce event bus retains high-volume events for 72 hours.</strong> Platform events and change data capture events sit on the event bus for three days, and each one carries a replay ID so a durable subscriber can resume from where it left off. Past that window, they&#8217;re purged. Salesforce doesn&#8217;t guarantee availability beyond it. So a relay that goes into an error state on a Friday afternoon is a data loss event by Monday morning, and the only symptom during those three days is an absence of messages, which looks identical to a quiet weekend.</p>



<p class="wp-block-paragraph"><strong>Deletes don&#8217;t show up in query-based syncs.</strong> A scheduled pull selects records that exist. Records that stopped existing between runs are simply not in the result set, so your S3 prefix or Redshift table keeps a row that Salesforce no longer has. Nothing errors. The row count on the AWS side just creeps upward relative to the CRM, forever, until someone notices.</p>



<p class="wp-block-paragraph">Every architecture decision below is really a decision about which of these three you&#8217;re accepting and how you&#8217;re going to detect it.</p>



<h2 class="wp-block-heading">A Salesforce AWS integration starts with a direction, not a service</h2>



<p class="wp-block-paragraph">The most common mistake I see is picking a service first, usually whichever one appeared in a conference demo, and then bending the requirement to fit it. Two questions decide the path, and they take about a minute to answer.</p>



<ol class="wp-block-list">
<li><strong>Which way does the data move?</strong> Salesforce to AWS, or AWS to Salesforce. If the answer is &#8220;both&#8221;, you have two integrations, not one, and they will have different failure modes and different owners.</li>

<li><strong>What shape is the workload?</strong> Batch, where you want a consistent set of records on a schedule, or event-driven, where you want a reaction within seconds of a change.</li>
</ol>



<p class="wp-block-paragraph">That gives you four cells, and each cell has a default answer.</p>



<h3 class="wp-block-heading">Salesforce to AWS, batch: Amazon AppFlow</h3>



<p class="wp-block-paragraph">Amazon AppFlow is the managed answer for pulling Salesforce objects into S3, Redshift, or the Glue Data Catalog on a schedule or on demand. It handles the OAuth connection, field mapping, and incremental pulls without you writing or hosting anything. For analytics loads it&#8217;s usually the right default, and it&#8217;s the path of least resistance if the destination is a data lake you&#8217;re already cataloguing with Glue and querying with Athena.</p>



<p class="wp-block-paragraph">The trade-off is that it&#8217;s a polling client, and it spends your Salesforce API allocation to do it. AWS documents this plainly with an example: an hourly flow pulling five pages of data makes 120 API calls a day, and all of them count against the org&#8217;s 24-hour API request limit. Multiply that by every object you&#8217;re syncing and every environment you&#8217;re syncing from, and AppFlow can become the largest single consumer of an allocation that Apex callouts, integration users, and third-party tools are also drawing on.</p>



<p class="wp-block-paragraph">The failure mode that catches people: if a field referenced in the flow&#8217;s mapping is deleted in Salesforce, the flow run fails. Someone cleans up an unused custom field on a Tuesday and your nightly load breaks on Wednesday, in a service that nobody on the Salesforce side knows exists. AppFlow will automatically pick up newly created Salesforce fields when the destination is S3, but the reverse case, a removed field, needs a human to edit the mapping. Put the flow names somewhere the Salesforce admins will see them before they run a field cleanup.</p>



<h3 class="wp-block-heading">Salesforce to AWS, event-driven: Event Relay and Amazon EventBridge</h3>



<p class="wp-block-paragraph">Event Relay subscribes to a channel on the Salesforce event bus and streams platform events and change data capture events into Amazon EventBridge as a partner event source. No middleware, no subscriber you have to keep alive, no polling and therefore no API allocation burn on the read path. From EventBridge you fan out to Lambda, Step Functions, SQS, Firehose, whatever the downstream is.</p>



<p class="wp-block-paragraph">Setup runs across both consoles, and the order matters. On the Salesforce side you create a named credential that points at your AWS account and region. It uses no authentication protocol, because trust is established by the partner event source relationship rather than by a credential in the request. The region goes in uppercase, which is the single most common reason a first attempt doesn&#8217;t work.</p>



<pre class="wp-block-code"><code>POST /services/data/vXX.X/tooling/sobjects/NamedCredential/

{
  "FullName": "AwsRelayCredential",
  "Metadata": {
    "label": "AwsRelayCredential",
    "endpoint": "arn:aws:US-EAST-1:111122223333",
    "principalType": "NamedUser",
    "protocol": "NoAuthentication",
    "generateAuthorizationHeader": true
  }
}</code></pre>



<p class="wp-block-paragraph">The endpoint is not a URL. It&#8217;s an ARN-shaped string carrying the AWS region and the twelve-digit account ID, and that&#8217;s the whole addressing scheme. Get the account wrong and the partner event source appears in a stranger&#8217;s console, not yours.</p>



<p class="wp-block-paragraph">You then create an event channel, add the platform events or change events you want as channel members, and create the relay configuration against that channel. A relay is created in a stopped state deliberately, so you get a chance to verify the AWS account and region before anything flows. That pause is the useful part of the design, not an inconvenience to click through.</p>



<p class="wp-block-paragraph">Now switch to AWS and associate the source. This is the step that, skipped, produces the silent drop described earlier.</p>



<pre class="wp-block-code"><code># Find the partner event source Salesforce created in your account
aws events list-event-sources --region us-east-1

# Create the matching bus. Both values must be the source name, character for character.
aws events create-event-bus 
  --name "aws.partner/salesforce.com/&lt;org-and-channel-suffix&gt;" 
  --event-source-name "aws.partner/salesforce.com/&lt;org-and-channel-suffix&gt;" 
  --region us-east-1

# Confirm the source moved from PENDING to ACTIVE
aws events describe-event-source 
  --name "aws.partner/salesforce.com/&lt;org-and-channel-suffix&gt;" 
  --region us-east-1</code></pre>



<p class="wp-block-paragraph"><code>describe-event-source</code> returning <code>ACTIVE</code> is your proof that a matching bus exists and is enabled. <code>PENDING</code> means either you haven&#8217;t created the bus or it&#8217;s deactivated, and every event published in that state is gone. This is the single check worth wiring into a synthetic monitor, because it&#8217;s cheap to run and it catches the most expensive mistake.</p>



<p class="wp-block-paragraph">Only once the source is active do you set the relay state to run. From the Salesforce side, the feedback object is where relay errors surface rather than in any log you&#8217;d normally tail:</p>



<pre class="wp-block-code"><code>SELECT Id, RemoteResource, Status, ErrorMessage, ErrorTime, ErrorIdentifier
FROM EventRelayFeedback
WHERE EventRelayConfigId = '7k2XXXXXXXXXXXXXXX'</code></pre>



<p class="wp-block-paragraph">If you&#8217;d rather not use Event Relay at all, the Pub/Sub API is the direct route: a gRPC interface to the same event bus, delivering Avro-encoded payloads, which you subscribe to from your own consumer. That&#8217;s more code and a process to keep running, on ECS or a VPS from a provider like InterServer or Hetzner if you&#8217;re keeping it outside AWS, but you get explicit control over replay ID checkpointing. Teams already running Kafka often go this way and land the stream through a managed connector from Confluent instead. Event Relay is the lower-operational-cost option; the Pub/Sub API is the one to reach for when you need the checkpoint under your own control.</p>



<h3 class="wp-block-heading">AWS to Salesforce, event-driven: EventBridge API Destinations</h3>



<p class="wp-block-paragraph">Going the other way, an API destination lets EventBridge call the Salesforce REST API directly, with no Lambda in the middle. You create a connection holding OAuth client credentials from a Salesforce connected app, point the destination at your org&#8217;s My Domain endpoint, and route matched events to it. Publishing back into an inbound platform event is the common pattern, so the write lands on the Salesforce event bus and Apex or Flow picks it up from there.</p>



<p class="wp-block-paragraph">Two constraints shape the design, and both are documented:</p>



<ul class="wp-block-list">
<li><strong>Five second client execution timeout.</strong> If Salesforce takes longer than that to respond, EventBridge times out the request and treats it as a retryable failure. Anything you make synchronous behind that endpoint, a trigger doing a callout, a validation chain, a busy org at peak, will eat into it.</li>

<li><strong>The default retry policy is generous, which is the problem.</strong> For event buses, EventBridge will retry a failed delivery for up to 24 hours and up to 185 times by default. If the endpoint is genuinely broken, that&#8217;s a day of retries against your org, consuming API calls, before the event is discarded with nowhere to go.</li>
</ul>



<p class="wp-block-paragraph">Tighten the policy and always attach a dead letter queue. Failed events land somewhere you can inspect and replay rather than evaporating at the end of the retry window:</p>



<pre class="wp-block-code"><code>{
  "Id": "salesforce-writeback",
  "Arn": "arn:aws:events:us-east-1:111122223333:api-destination/sf-platform-event/&lt;id&gt;",
  "RoleArn": "arn:aws:iam::111122223333:role/EventBridgeApiDestinationRole",
  "RetryPolicy": {
    "MaximumRetryAttempts": 20,
    "MaximumEventAgeInSeconds": 3600
  },
  "DeadLetterConfig": {
    "Arn": "arn:aws:sqs:us-east-1:111122223333:salesforce-writeback-dlq"
  }
}</code></pre>



<p class="wp-block-paragraph">Set the invocation rate limit on the destination to something your org can absorb. It exists specifically so you can align delivery with a third-party API&#8217;s limits, and Salesforce is very much a third-party API with limits.</p>



<p class="wp-block-paragraph">One thing worth knowing before you design around it: API destinations don&#8217;t support mutual TLS. If your security review requires mTLS on outbound calls, you&#8217;re putting a Lambda or a proxy in the path regardless.</p>



<h3 class="wp-block-heading">AWS to Salesforce, batch: Lambda and the Bulk API</h3>



<p class="wp-block-paragraph">For loading volume back into the org, nothing managed does this well and you write it yourself. A Lambda or a container job authenticates with the JWT bearer flow against a connected app, submits a Bulk API job, and polls for results. The REST API is fine for small writes; past a few thousand records the per-call overhead and the API allocation make the Bulk API the only sensible choice.</p>



<p class="wp-block-paragraph">This is the cell where you own everything: retries, idempotency, partial-failure handling, and backoff when the org throttles you. Use external ID fields and upsert semantics rather than insert, so a retried batch converges instead of duplicating. Store the job ID before you submit, not after, so a Lambda timeout mid-flight leaves you something to reconcile against. And keep the Salesforce credential in Secrets Manager with rotation wired up, because a hardcoded connected app secret in an environment variable is the finding that shows up in every audit.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The network layer: public internet, PrivateLink, or Private Connect</h2>



<p class="wp-block-paragraph">All four paths above work over the public internet with TLS, and for most workloads that&#8217;s genuinely fine. The question is whether your compliance posture or your risk appetite says otherwise.</p>



<p class="wp-block-paragraph">Salesforce Private Connect is the managed private path. Salesforce operates a transit VPC in each supported AWS region, and you create a PrivateLink connection between that transit VPC and yours. Traffic between the two never traverses the public internet. The terminology is worth getting right because it trips people up in design reviews:</p>



<ul class="wp-block-list">
<li><strong>Inbound</strong> means traffic flowing <em>into</em> Salesforce. AWS to Salesforce, or on-premises to Salesforce.</li>

<li><strong>Outbound</strong> means traffic flowing <em>out of</em> Salesforce. Salesforce calling a service in your VPC.</li>
</ul>



<p class="wp-block-paragraph">The direction is named from Salesforce&#8217;s point of view, not yours, and getting it backwards means provisioning the wrong connection.</p>



<p class="wp-block-paragraph">AppFlow can ride on this too. Choosing the PrivateLink option when you create the Salesforce connection has AppFlow route through Private Connect and manage the endpoint lifecycle for you, which is a meaningful simplification over building it by hand.</p>



<p class="wp-block-paragraph">Where I&#8217;d push back: Private Connect is a licensed add-on, and each VPC needs its own connection, so a multi-account AWS estate with separate VPCs per environment multiplies the cost. If the driver is a checkbox on a questionnaire rather than a real threat model, TLS with strict IP allowlisting on the Salesforce side and a well-scoped connected app gets you most of the way for nothing. If the driver is a regulator who wants to see that customer data never touches a public network path, Private Connect is the clean answer and the audit conversation ends quickly. Decide which one you&#8217;re in before you start pricing it.</p>



<h2 class="wp-block-heading">The reconciliation job nobody builds</h2>



<p class="wp-block-paragraph">This is the part that separates an integration that survives from one that quietly rots, and it&#8217;s about thirty lines of code plus a schedule.</p>



<p class="wp-block-paragraph">Run a count comparison on a window that both sides can express. Nightly is plenty:</p>



<pre class="wp-block-code"><code>-- Salesforce side, via the REST query endpoint
SELECT COUNT(Id) FROM Opportunity
WHERE LastModifiedDate = LAST_N_DAYS:1

-- AWS side, same window, in Athena
SELECT COUNT(DISTINCT id) FROM crm.opportunity
WHERE last_modified_date &gt;= date_add('day', -1, current_date)</code></pre>



<p class="wp-block-paragraph">If the two numbers differ by more than a small tolerance for clock skew at the window edges, alert. That single check catches a stopped relay, a failed flow, a mapping that silently dropped a field, and a partner event source sitting in <code>PENDING</code>. It catches all of them without you having to instrument each path separately, which is exactly why it&#8217;s worth building first.</p>



<p class="wp-block-paragraph">For the delete problem, Salesforce exposes a dedicated endpoint. Ask it for records removed in a window and apply those deletions on the AWS side:</p>



<pre class="wp-block-code"><code>GET /services/data/vXX.X/sobjects/Opportunity/deleted/
    ?start=&lt;ISO-8601 timestamp, URL encoded&gt;
    &amp;end=&lt;ISO-8601 timestamp, URL encoded&gt;</code></pre>



<p class="wp-block-paragraph">The window you can ask about is bounded by how long deleted records stay recoverable, so run it more often than that bound, not less. Change data capture delivers delete events too, which is the better answer when it&#8217;s available, but the endpoint is what you use to backfill after a gap, and after a 72-hour gap it&#8217;s the <em>only</em> thing that will tell you what you missed.</p>



<p class="wp-block-paragraph">Put both checks on one dashboard alongside your AppFlow run status and EventBridge failed-invocation metrics. CloudWatch will do it, and if you&#8217;re already running Grafana Cloud or Datadog for the rest of the estate, pull the Salesforce-side numbers in as a custom metric so one panel answers &#8220;are the two systems agreeing right now&#8221;.</p>



<h2 class="wp-block-heading">Troubleshooting a Salesforce AWS integration</h2>



<ul class="wp-block-list">
<li><strong>Relay is running, nothing arrives in EventBridge.</strong> Run <code>describe-event-source</code>. If the state is <code>PENDING</code>, the bus was never created or has been deactivated, and everything published so far is gone. Create the bus, then treat the gap as a backfill job.</li>

<li><strong>Relay won&#8217;t start, or flips to an error state.</strong> Query <code>EventRelayFeedback</code> for the config. The usual causes are a wrong AWS account ID in the named credential, a region in lowercase, or the channel having no members.</li>

<li><strong>Events arrive but no rule matches.</strong> Check the rule is attached to the partner event bus, not the default bus. This one costs people an afternoon. Match on the source prefix and the detail type, and test with an event pattern before wiring a real target.</li>

<li><strong>An AppFlow flow that ran for months suddenly fails.</strong> Look for a field in the mapping that no longer exists in the source object. Also check for a bad gateway error, which points at a connection configured with the wrong instance URL and is fixed by deleting the connection and recreating it rather than editing it.</li>

<li><strong>API destination returns 401 intermittently.</strong> The connection refreshes its OAuth token when it sees a 401 or 407. Persistent 401s usually mean the connected app has no usable scopes for the client credentials flow, or the run-as user was changed. Check the connected app&#8217;s scopes before you touch anything in AWS.</li>

<li><strong>Everything times out at almost exactly five seconds.</strong> That&#8217;s the API destination timeout, not Salesforce being slow. Move the work behind an inbound platform event so the HTTP call returns immediately and the processing happens asynchronously in the org.</li>

<li><strong>Unexplained API limit exhaustion.</strong> Sum the scheduled AppFlow runs against page counts before blaming a rogue integration user. A frequent flow across several objects adds up faster than most people expect.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Starting the relay before associating the partner event bus, and assuming the events queued somewhere.</li>

<li>Treating the 72-hour retention as a comfortable buffer. It&#8217;s three days including a weekend, and an unmonitored relay burns through it without a single alert.</li>

<li>Using change data capture as a replacement for an initial load. It streams changes from the moment you enable it; it doesn&#8217;t backfill history.</li>

<li>Building the whole integration bidirectionally through one mechanism, then discovering you&#8217;ve created an echo loop where a write from AWS fires a change event that triggers another write.</li>

<li>Sizing the AppFlow schedule for freshness without checking what it does to the org&#8217;s API allocation.</li>

<li>Leaving the default 24-hour EventBridge retry policy on an API destination with no dead letter queue.</li>

<li>Provisioning Private Connect for every VPC because it&#8217;s the secure-sounding option, without modelling the per-connection cost across environments.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Build the reconciliation check before you build the second integration path. It&#8217;s the only thing that tells you the whole system is working.</li>

<li>Monitor the partner event source state, not just rule invocations. Invocations go to zero on a quiet day and on a broken day, and the state field distinguishes them.</li>

<li>Give every integration its own connected app and its own integration user. Shared credentials make it impossible to attribute API consumption or revoke one path without breaking three.</li>

<li>Make writes idempotent with external IDs and upsert. Every path here retries, and some retry aggressively.</li>

<li>Version your event payloads. Platform events are immutable once published, and adding a required field later means a new event definition, not an edit.</li>

<li>Keep the AWS-side resource names discoverable from the Salesforce side. Put the flow name or event bus name in the named credential label so a Salesforce admin can see what&#8217;s attached before they change something.</li>

<li>Define the whole thing in Terraform or CloudFormation on the AWS side and in metadata on the Salesforce side, so a sandbox refresh doesn&#8217;t leave you clicking through two consoles from memory.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need MuleSoft to integrate Salesforce with AWS?</h3>



<p class="wp-block-paragraph">No. For the four patterns above, the native services cover it without an integration platform. MuleSoft, Boomi, and similar tools earn their place when you&#8217;re orchestrating across many systems, need transformation logic that lives outside both platforms, or have a governance requirement for a single integration layer. For a two-system integration, they&#8217;re usually more machinery than the problem needs.</p>



<h3 class="wp-block-heading">Does Event Relay use my Salesforce API allocation?</h3>



<p class="wp-block-paragraph">Event delivery counts against your platform event delivery allocation, which is a separate budget from the 24-hour API request limit that REST calls and AppFlow polling draw on. That separation is one of the strongest arguments for the event-driven path when API consumption is already tight in the org.</p>



<h3 class="wp-block-heading">Can I replay Salesforce events that AWS missed?</h3>



<p class="wp-block-paragraph">Within the 72-hour retention window, yes, using replay IDs from a durable subscriber. Beyond it, no. The events are purged and there&#8217;s no recovery path through the event bus. Backfilling from that point means querying the objects directly and using the deleted-records endpoint to catch removals, which is exactly why the reconciliation job matters.</p>



<h3 class="wp-block-heading">Should I use AppFlow or write my own Lambda extractor?</h3>



<p class="wp-block-paragraph">Start with AppFlow. It removes the auth handling, the pagination, and the scheduling, and for standard object pulls into S3 or Redshift it&#8217;s hard to justify writing that yourself. Move to a custom extractor when you need transformation logic AppFlow can&#8217;t express, when you&#8217;re hitting connector limitations on a specific object, or when you want tighter control over API consumption than a fixed schedule allows.</p>



<h3 class="wp-block-heading">How do I send data from AWS back into Salesforce without a Lambda?</h3>



<p class="wp-block-paragraph">EventBridge API destinations call the Salesforce REST API directly using OAuth client credentials from a connected app. Point them at an inbound platform event endpoint so the write returns fast and processing happens asynchronously in the org. You still need a Lambda if you require mTLS or payload transformation beyond what the input transformer handles.</p>



<h3 class="wp-block-heading">Is Private Connect required for a secure Salesforce AWS integration?</h3>



<p class="wp-block-paragraph">Not required. Everything runs over TLS by default. Private Connect removes the public internet from the path entirely, which matters when a regulator or a security review asks you to demonstrate it. It&#8217;s a licensed add-on with a per-connection cost, so model it against your actual VPC count before committing.</p>



<h3 class="wp-block-heading">Why does my EventBridge rule never fire even though events are arriving?</h3>



<p class="wp-block-paragraph">Almost always because the rule is on the default event bus rather than the partner event bus. Partner events land on their own bus and rules are bus-scoped, so a correct pattern on the wrong bus matches nothing and reports nothing.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A Salesforce AWS integration doesn&#8217;t usually break loudly. It drifts. A relay stops, a mapping goes stale, a delete never propagates, and the two systems keep answering questions confidently with different numbers. The services themselves are well built and the setup is genuinely not hard once you know the region goes in uppercase and the bus has to be associated first.</p>



<p class="wp-block-paragraph">So pick the path by direction and shape, wire the failure modes you&#8217;ve accepted into alerts, and build the nightly count comparison before you build anything clever. That one job is worth more than any amount of pipeline polish, because it&#8217;s the only thing in the architecture that can tell you the whole thing is still true.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your Salesforce and AWS setup?</h2>



<p class="wp-block-paragraph">I work with teams on the infrastructure side of CRM integrations, usually where the data is disagreeing and nobody can prove why. Things I can help with:</p>



<ul class="wp-block-list">
<li>Reviewing an existing Salesforce to AWS pipeline and finding where records are being lost between the two.</li>

<li>Standing up Event Relay into EventBridge end to end, including the partner bus association and the monitoring that proves it&#8217;s live.</li>

<li>Designing the AppFlow schedule and object set so it doesn&#8217;t eat the org&#8217;s API allocation.</li>

<li>Building the reconciliation and delete-propagation jobs, with alerting in CloudWatch, Grafana, or whatever you already run.</li>

<li>Writing the write-back path properly: Bulk API jobs, idempotent upserts, dead letter queues, and credential rotation through Secrets Manager.</li>

<li>Working out whether Private Connect is worth the licence in your specific environment, or whether TLS and tighter scoping gets you there.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;ve got a flow that&#8217;s failing, a relay stuck in an error state, or two record counts that won&#8217;t agree, send me the config or the error and I&#8217;ll tell you what I&#8217;d check first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-aws-integration-architecture/">Salesforce AWS Integration Architecture: The Four Paths and What Breaks Silently</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Extracting Clauses, Parties, Dates and Obligations with Amazon Bedrock</title>
		<link>https://john-nessime.com/blog/technical-guides/extract-contract-data-amazon-bedrock/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 18 Aug 2026 11:17:11 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Bedrock Data Automation]]></category>
		<category><![CDATA[Confidence Scoring]]></category>
		<category><![CDATA[Contract Analysis]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Human In The Loop]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Legal Tech]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Step Functions]]></category>
		<category><![CDATA[Structured Outputs]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=392</guid>

					<description><![CDATA[<p>Valid JSON is not correct data. A practical guide to extracting clauses, parties, dates and obligations from contracts with Amazon Bedrock: the two build paths, schema design, the three failure families that produce confident wrong answers, and the validation layer that catches them.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/extract-contract-data-amazon-bedrock/">Extracting Clauses, Parties, Dates and Obligations with Amazon Bedrock</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The extraction job finishes clean. Every contract in the batch produces valid JSON, every field is populated, the schema validator passes, and the results load into the obligation tracker without a single error. Three weeks later someone in legal asks why the tracker says the renewal notice window is 30 days when the contract says 90. That is the moment you learn the real problem with contract data extraction: the failure mode is not malformed output. It is well-formed output that is wrong.</p>



<p class="wp-block-paragraph">This post is a practical guide to how I&#8217;d extract contract data with Amazon Bedrock: the clauses, parties, dates, and obligations that feed contract lifecycle tools, obligation trackers, and compliance dashboards. It covers the two build paths Bedrock gives you, how to design the extraction schema, the three failure families that produce confident wrong answers, and the validation layer that catches them before they reach a system someone trusts.</p>



<h2 class="wp-block-heading">Two ways to extract contract data with Amazon Bedrock</h2>



<p class="wp-block-paragraph">Bedrock offers two genuinely different routes to the same destination, and picking the wrong one costs you either flexibility or months of pipeline plumbing you didn&#8217;t need to build.</p>



<h3 class="wp-block-heading">Path one: call a model directly</h3>



<p class="wp-block-paragraph">You send the contract to a foundation model, typically Anthropic Claude, through the Converse or InvokeModel API, with a prompt and a schema describing the fields you want back. Converse accepts PDF documents as a content block, so contract bytes go straight from S3 without a separate OCR step. One detail that catches people: full visual understanding of a PDF through Converse with Claude requires citations to be enabled. Without them the API falls back to plain text extraction, and anything living in a table or a scanned signature page silently disappears from the model&#8217;s view.</p>



<p class="wp-block-paragraph">The piece that changed this path from fragile to dependable is structured outputs. Instead of asking the model nicely for JSON and writing defensive parsing code for the day it prepends an apology, you attach a JSON Schema to the request and Bedrock enforces it during token generation through constrained decoding. The response cannot fail to parse. One mechanic worth knowing: the first request with a new schema triggers a grammar compilation step, and compiled grammars are cached for a limited time, so schema churn has a latency cost.</p>



<h3 class="wp-block-heading">Path two: Bedrock Data Automation</h3>



<p class="wp-block-paragraph">Bedrock Data Automation (BDA) is the managed intelligent document processing service. You define a blueprint, which is a list of fields with types and natural language instructions for how to find and normalize each one, attach it to a project, and invoke the async processing API against files in S3. BDA returns two things a raw model call does not give you out of the box: a confidence score per field and visual grounding, meaning bounding boxes pointing back to where on the page each value came from. A project can hold multiple document blueprints, so a mixed intake of NDAs, master service agreements, and amendments routes each document to the right extraction logic automatically.</p>



<p class="wp-block-paragraph">My honest read on the choice: BDA wins when you want the boring parts managed and you need per-field confidence and grounding for a human review loop, which for contracts you almost always do. The direct model path wins when you need reasoning the blueprint format can&#8217;t express, like interpreting a clause against the definitions section. Plenty of real pipelines use both: BDA for the structured pass, a direct Claude call for the clauses that need interpretation. Amazon Textract still earns its place for high-volume standardized forms, but contracts are exactly the variable, long-tail documents layout-based extraction was never great at.</p>



<h2 class="wp-block-heading">Design the schema before you touch the API</h2>



<p class="wp-block-paragraph">The schema is not a formality. With structured outputs, the field names, descriptions, and ordering you write become context the model generates against, so the schema is effectively part of the prompt. A vague schema produces structurally valid garbage: JSON that parses perfectly and means nothing.</p>



<p class="wp-block-paragraph">Three design rules that pay for themselves. First, one concept per field: &#8220;renewal terms&#8221; as a single string is unusable downstream; split it into whether renewal exists, whether it is automatic, the duration, and the notice period to prevent it. Second, write field descriptions like instructions to a junior reviewer, including what the field is not. Third, make every field nullable and tell the model to return null when the contract is silent. A model forced to fill a required field will fill it with something.</p>



<p class="wp-block-paragraph">Here is the shape I&#8217;d start with for a general commercial agreement, trimmed to the core:</p>



<pre class="wp-block-code"><code>{
  "type": "object",
  "properties": {
    "parties": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "legal_name": { "type": ["string", "null"],
            "description": "Full registered entity name from the preamble or signature block, not the defined shorthand" },
          "defined_term": { "type": ["string", "null"],
            "description": "The shorthand assigned in the preamble, e.g. Supplier, Client" },
          "role": { "type": ["string", "null"] }
        }
      }
    },
    "effective_date": { "type": ["string", "null"],
      "description": "Date the agreement takes effect. Null if only a signature date exists." },
    "governing_law": { "type": ["string", "null"] },
    "obligations": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "obligor": { "type": ["string", "null"] },
          "description": { "type": ["string", "null"] },
          "deadline_text": { "type": ["string", "null"],
            "description": "The deadline exactly as written, e.g. within 30 days of the Effective Date" },
          "source_clause": { "type": ["string", "null"],
            "description": "Section number where this obligation appears" }
        }
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Notice <code>deadline_text</code> keeps the deadline as written instead of asking the model to compute a calendar date. That is deliberate, and the reason is the second failure family below.</p>



<h2 class="wp-block-heading">Failure family one: parties that aren&#8217;t who they say they are</h2>



<p class="wp-block-paragraph">Contracts name their parties once, in the preamble, then never again. From that point on everything refers to &#8220;the Supplier&#8221; or &#8220;the Receiving Party.&#8221; A naive extraction returns the defined term as the party name, which is useless for matching against a vendor master or a CRM. Worse, the signature block sometimes names a different legal entity than the preamble, usually an affiliate signing on behalf of a subsidiary, and the model will happily pick whichever it saw last.</p>



<p class="wp-block-paragraph">The fix is structural, not clever prompting. Extract the legal name and the defined term as separate fields, as in the schema above, with instructions to take legal names from the preamble and cross-check the signature block. Then validate outside the model: if the two entities differ, flag the document for review rather than picking one. That mismatch is occasionally a genuine drafting error in the contract itself, which makes it exactly the kind of thing you want surfaced, not smoothed over.</p>



<h2 class="wp-block-heading">Failure family two: dates that mean different things</h2>



<p class="wp-block-paragraph">A contract carries several dates that look interchangeable and are not: execution, effective, commencement, expiry, and the deadlines derived from notice periods. The extraction failure that costs money is almost never a misread date string. It is a correctly read date assigned to the wrong concept, or a relative deadline flattened into a wrong absolute one.</p>



<p class="wp-block-paragraph">Relative dates are the invisible one. &#8220;Either party may terminate by written notice no later than ninety days before the end of the then-current term&#8221; contains no date at all. If your schema demands a date type, the model computes one, and its arithmetic rests on assumptions about the term start that may be wrong, especially when an amendment has reset the term. That computed date looks exactly as trustworthy in your tracker as a real one. This is why I extract deadline language verbatim and compute calendar dates in code, where the arithmetic is testable and the inputs are auditable.</p>



<p class="wp-block-paragraph">Cheap validation catches most of the rest. The effective date must not fall after the expiry. A notice period must be a plausible number of days. If the agreement auto-renews, a renewal duration should exist. Fields that fail these checks get flagged. None of this needs a model; it is twenty lines of code standing between extraction and the systems people trust.</p>



<h2 class="wp-block-heading">Failure family three: obligations, cross-references, and the amendment problem</h2>



<p class="wp-block-paragraph">Obligations are the hardest target because they are relational. &#8220;The Supplier shall deliver the reports described in Schedule 3 within the period set out in Section 4.2, except as provided in Section 9&#8221; is one obligation spread across three locations. Chunk the contract carelessly and the model sees the reference with no Section 4.2 in sight. It will either drop the deadline or invent a plausible one, and the second outcome is far worse because nothing downstream looks broken.</p>



<p class="wp-block-paragraph">Contracts resist chunking more than almost any other document type, because defined terms and cross-references make every section depend on distant ones. The practical answers, in order of preference:</p>



<ol class="wp-block-list">
<li>Don&#8217;t chunk. Modern model context windows fit most commercial agreements whole, and whole-document extraction is the single biggest accuracy win available. Reach for chunking only when a document genuinely doesn&#8217;t fit.</li>



<li>If you must split, prepend the definitions section and the table of contents to every chunk, and split on section boundaries rather than fixed token counts so no clause straddles a cut.</li>



<li>Capture the source clause reference for every obligation, as in the schema above, so a reviewer can jump straight to the language instead of re-reading the contract.</li>
</ol>



<p class="wp-block-paragraph">Then there are amendments. An amendment is a short document that quietly rewrites the base agreement, and if you extract each file independently your dataset will assert the original terms with full confidence long after they stopped being true. Treat the contract family, not the file, as the unit of extraction: process the base agreement, process each amendment, and resolve them in order so later documents override earlier ones. BDA&#8217;s multi-blueprint routing helps here, since amendments have a recognizably different shape and can carry their own blueprint focused on what changed.</p>



<h2 class="wp-block-heading">The validation layer that makes it trustworthy</h2>



<p class="wp-block-paragraph">Schema compliance tells you the output is well-formed. It says nothing about whether it is true. The trust comes from three checks layered after extraction.</p>



<ul class="wp-block-list">
<li><strong>Cross-field consistency.</strong> Dates in order, notice periods plausible, every obligation carrying an obligor that matches an extracted party. Pure code, no model.</li>



<li><strong>Grounding.</strong> Every extracted value should point back to a location in the document. BDA gives you this natively through bounding boxes; on the direct model path, requiring a source clause reference per field is the lightweight version. A value that cannot be traced to the page is a value you cannot audit.</li>



<li><strong>Confidence-based routing.</strong> Use BDA&#8217;s per-field confidence scores to route low-confidence extractions to human review and let high-confidence ones flow through. The goal is not eliminating review; it is spending reviewer minutes only where the model is unsure. Party names and governing law typically extract very reliably, while obligations from unusual templates are where automation honestly ends and review begins.</li>
</ul>



<p class="wp-block-paragraph">Operationally this looks like S3 event notifications triggering a Step Functions workflow: extract, validate, then branch to auto-accept or a review queue. Instrument it from day one. Track the auto-accept rate, per-field confidence distributions, and validation failure counts; a slow drift in confidence is usually the first sign a new contract template has entered the intake. I ship these to Grafana Cloud with the rest of the pipeline metrics, and since direct model calls bill by tokens while BDA bills per unit of content processed, a cost dashboard in something like Vantage or CloudZero shows quickly whether whole-document extraction is worth its token count for your mix. It usually is.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<p class="wp-block-paragraph"><strong>Fields come back null for content that is clearly in the document.</strong> On the Converse path with PDFs, check whether you are getting text-only extraction; content in tables, stamps, or scanned pages needs the visual analysis mode. For scans, verify image quality before blaming the model.</p>



<p class="wp-block-paragraph"><strong>The same contract extracts differently on reruns.</strong> Set temperature to zero for extraction work, then look for instructions the model can resolve more than one way. Ambiguity in a schema description is the usual culprit.</p>



<p class="wp-block-paragraph"><strong>First requests with a new schema are slow.</strong> That is grammar compilation for structured outputs. Stabilize your schemas and warm them before batch runs rather than generating schemas dynamically per request.</p>



<p class="wp-block-paragraph"><strong>BDA accuracy is mediocre on your templates.</strong> Blueprint instructions are tunable, and BDA can refine them automatically from a small set of example documents with known correct values. Feed it your hardest real examples, not your cleanest ones.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating schema-valid JSON as verified data. Structure and truth are separate properties, and the second one needs its own checks.</li>



<li>Extracting the defined term as the party name and matching it against vendor records.</li>



<li>Letting the model compute calendar dates from relative deadline language instead of extracting the language and computing in code.</li>



<li>Chunking by token count through a document whose clauses reference each other across fifty pages.</li>



<li>Processing amendments as standalone documents and silently keeping stale base-agreement terms.</li>



<li>Making every schema field required, which converts &#8220;the contract is silent&#8221; into a fabricated answer.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Extract whole documents whenever the context window allows. Every chunking strategy is a workaround, not a feature.</li>



<li>Keep a small labeled evaluation set of your own contracts and score every prompt or blueprint change against it before shipping.</li>



<li>Require a source reference for every extracted value, whether that is a BDA bounding box or a clause number from the model.</li>



<li>Route on confidence and validation results, and measure your auto-accept rate as a first-class metric.</li>



<li>Version your schemas and store the schema version alongside every extraction, so you can tell which records predate a fix.</li>



<li>Keep humans in the loop for high-stakes fields. The realistic goal is automating the bulk and focusing legal review, not replacing it.</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Should I use Bedrock Data Automation or call Claude directly for contract extraction?</h3>



<p class="wp-block-paragraph">BDA if you want managed parsing, per-field confidence scores, and visual grounding for a review workflow. Direct model calls if you need interpretive reasoning over clauses or conversational access to the document. Many production pipelines combine both.</p>



<h3 class="wp-block-heading">Do I still need Amazon Textract?</h3>



<p class="wp-block-paragraph">Usually not as the primary extractor for contracts, since layout-based extraction struggles with variable legal drafting. It remains a good fit for high-volume standardized forms, and some pipelines route documents between Textract and generative extraction by type.</p>



<h3 class="wp-block-heading">How accurate is contract data extraction in practice?</h3>



<p class="wp-block-paragraph">It varies sharply by field type. Parties, governing law, and explicit dates extract reliably. Obligations, renewal mechanics, and anything expressed through cross-references are meaningfully harder, which is why confidence routing and human review exist. Distrust any vendor quoting one accuracy number for &#8220;contracts.&#8221;</p>



<h3 class="wp-block-heading">How do I extract obligations specifically, not just metadata?</h3>



<p class="wp-block-paragraph">Model each obligation as its own object with obligor, description, deadline language, and source clause. Extract against the whole document so cross-references resolve, and validate that every obligor matches an extracted party.</p>



<h3 class="wp-block-heading">Can Bedrock handle scanned contracts?</h3>



<p class="wp-block-paragraph">Yes. BDA processes scanned documents as part of its parsing stage, and Claude&#8217;s visual document understanding reads scanned pages when invoked with visual analysis. Scan quality still matters; poor scans degrade every approach.</p>



<h3 class="wp-block-heading">Is it safe to put contracts through Bedrock?</h3>



<p class="wp-block-paragraph">Bedrock runs within your AWS account boundary, and AWS states that customer content is not used to train the underlying models. For sensitive agreements the usual controls apply: VPC endpoints for private connectivity, KMS encryption on the buckets, and IAM scoped to the pipeline roles. Confirm the specifics against your own compliance requirements.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">If you remember one thing, make it this: when you extract contract data with Amazon Bedrock, valid output is the starting line, not the finish. Structured outputs and BDA blueprints have genuinely solved the formatting problem, which means the remaining risk is concentrated in meaning: the wrong entity, the wrong date concept, the obligation whose deadline lives two sections away. Design the schema around those failures, validate outside the model, ground every value to the page, and route on confidence. That is the difference between a demo and a pipeline a legal team will actually trust.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help building a contract extraction pipeline on AWS?</h2>



<p class="wp-block-paragraph">I design and build document intelligence pipelines on AWS as a freelance DevOps engineer. If you&#8217;re extracting data from contracts or other legal documents, here&#8217;s what I can help with:</p>



<ul class="wp-block-list">
<li>Designing extraction schemas and BDA blueprints for your actual contract types, including amendments and multi-document families</li>



<li>Building the full serverless pipeline: S3 intake, Step Functions orchestration, extraction, validation, and review routing</li>



<li>Adding the validation and grounding layer that catches confident wrong answers before they reach your CLM or tracker</li>



<li>Setting up evaluation sets and accuracy measurement so prompt and blueprint changes are tested, not guessed</li>



<li>Locking down the security side: VPC endpoints, KMS encryption, and IAM scoping for sensitive documents</li>



<li>Cost tuning across model choice, whole-document versus chunked extraction, and BDA versus direct model calls</li>
</ul>



<p class="wp-block-paragraph">Send me a sample contract (redacted is fine), your target field list, or the output of a pipeline that isn&#8217;t behaving, and I&#8217;ll tell you what I&#8217;d do with it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/extract-contract-data-amazon-bedrock/">Extracting Clauses, Parties, Dates and Obligations with Amazon Bedrock</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</title>
		<link>https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 17 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Cursor Pagination]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[GraphQL]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Orchestration]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[Step Functions]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=215</guid>

					<description><![CDATA[<p>A GraphQL source can hand you a 200 OK, a populated data block, and a quietly broken column in the same response. Here is how to build a GraphQL data ingestion pipeline on AWS that catches partial errors, respects cost-based rate limits, resumes cleanly from a cursor, and notices when the schema moves under you.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/">Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The row counts looked fine. That was the problem.</p>



<p class="wp-block-paragraph">A nightly sync from a partner&#8217;s GraphQL API had run for weeks without a single failed execution. Green across the board in the console. Then someone asked why one column had been empty since the previous Tuesday. Not sparse. Empty. Every row, every night, for six nights.</p>



<p class="wp-block-paragraph">The API had started returning a field-level error on that one field. Status code 200. A populated <code>data</code> block. An <code>errors</code> array sitting right next to it. The fetcher parsed <code>data</code>, wrote the records to S3, exited zero, and the orchestrator logged a success. Nothing in the stack was wrong, exactly. Everything in the stack was checking the wrong thing.</p>



<p class="wp-block-paragraph">This post covers how to build a GraphQL data ingestion pipeline on AWS that fails honestly. It is organised by failure family rather than by service, because the AWS wiring is the easy part. The hard part is that GraphQL breaks four assumptions that REST ingestion quietly relies on, and every one of them fails silently by default.</p>



<h2 class="wp-block-heading">Why GraphQL ingestion breaks the habits REST taught you</h2>



<p class="wp-block-paragraph">Most ETL code carries three inherited reflexes from years of REST work: the status code tells you whether it worked, the response shape is the server&#8217;s business, and rate limits are counted in requests. GraphQL invalidates all three.</p>



<ul class="wp-block-list">
<li><strong>The status code is not the contract.</strong> The GraphQL over HTTP specification says a server should return 200 even when field errors are raised during execution, because a partial response is still a successful execution.</li>

<li><strong>You own the response shape.</strong> The selection set is written by you, which means schema drift shows up as a query that stops being valid, not as a payload that changes underneath you.</li>

<li><strong>Cost is query-shaped, not endpoint-shaped.</strong> One endpoint can cost almost nothing or blow your entire budget depending on how deeply you nested it.</li>
</ul>



<p class="wp-block-paragraph">Each of those becomes a distinct failure family. Let&#8217;s take them in the order they bite.</p>



<h2 class="wp-block-heading">Failure one: the 200 OK that quietly ate a column</h2>



<p class="wp-block-paragraph">This is the one that costs the most and shows up the latest. GraphQL splits errors into two categories, and they behave completely differently on the wire.</p>



<ul class="wp-block-list">
<li><strong>Request errors</strong> happen before execution starts: syntax errors, validation failures, bad variable coercion. No <code>data</code> key is returned at all. Nothing executed.</li>

<li><strong>Field errors</strong> happen during execution: a resolver threw, a downstream service timed out, a permission check failed. Execution continues, the failed field goes null, and you get <code>data</code> and <code>errors</code> together.</li>
</ul>



<p class="wp-block-paragraph">Field errors are the dangerous ones for ingestion, because they look exactly like success to anything that only inspects the transport layer. Worse, GraphQL&#8217;s null propagation rules mean a single failing non-null field bubbles up to the nearest nullable ancestor, so one broken resolver can hollow out an entire nested object while the surrounding rows land normally.</p>



<p class="wp-block-paragraph">The fix is to make the response body the contract. Classify before you commit anything:</p>



<pre class="wp-block-code"><code>import json, urllib.request

class RequestError(Exception): pass
class FieldError(Exception): pass

def graphql(endpoint, token, query, variables):
    body = json.dumps({"query": query, "variables": variables}).encode()
    req = urllib.request.Request(
        endpoint,
        data=body,
        headers={
            "Content-Type": "application/json",
            "Accept": "application/graphql-response+json, application/json",
            "Authorization": f"Bearer {token}",
        },
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        payload = json.loads(resp.read())

    # The status code told us nothing. The body is the contract.
    if "data" not in payload:
        # Nothing executed. Query or auth problem. Do not retry blindly.
        raise RequestError(payload.get("errors"))

    if payload.get("errors"):
        # Partial response. data is populated but incomplete.
        raise FieldError(payload["errors"])

    return payload["data"]</code></pre>



<p class="wp-block-paragraph">Two things worth explaining. The <code>Accept</code> header asks for <code>application/graphql-response+json</code>, the media type the GraphQL over HTTP spec recommends, which lets a compliant server return a real 4xx for pre-execution failures instead of burying them in a 200. Servers that don&#8217;t support it fall back to <code>application/json</code>, so including both is free.</p>



<p class="wp-block-paragraph">The second is the split between the two exception types. They need different handling. A request error is a bug in your query or a credential problem, and retrying it just burns budget on the same failure. A field error is transient often enough to be worth retrying, but never worth committing.</p>



<h3 class="wp-block-heading">Deciding what &#8220;partial&#8221; means for your data</h3>



<p class="wp-block-paragraph">Failing the whole page on any error is the safe default and the one I reach for first. But it&#8217;s not always right. If you are pulling a hundred fields and one optional enrichment field is flaky, refusing every page forever is its own outage.</p>



<p class="wp-block-paragraph">The workable middle ground is to classify by path. Every error object carries a <code>path</code> array pointing at the exact field that failed. Maintain a small allowlist of paths that are permitted to be null, fail the page on anything outside it, and emit the error count as a CloudWatch metric so a rising trend is visible before someone in finance notices.</p>



<h2 class="wp-block-heading">Failure two: rate limits that never send a 429</h2>



<p class="wp-block-paragraph">Your retry logic almost certainly triggers on 429 and 5xx. Several major GraphQL APIs will throttle you without ever sending one.</p>



<p class="wp-block-paragraph">GitHub&#8217;s GraphQL API is explicit about this: exceeding the primary rate limit returns a 200 response with an error message in the body and <code>x-ratelimit-remaining</code> set to zero. Your Step Functions retrier sees a success. Your Lambda sees no data. The run completes, having ingested nothing, and reports green.</p>



<p class="wp-block-paragraph">The other half of the problem is that request counting is the wrong unit. Shopify&#8217;s GraphQL Admin API and several others price queries by <em>calculated cost</em>, using a leaky bucket. The server statically analyses your query before executing it, charges the bucket the worst-case <em>requested</em> cost, executes, then refunds the difference between requested and <em>actual</em> cost. That has a consequence people miss: a query asking for the maximum connection size gets charged as if every page were full, even when the last page returns three records.</p>



<p class="wp-block-paragraph">The good news is that these APIs tell you the state of the bucket on every response, under the <code>extensions</code> key:</p>



<pre class="wp-block-code"><code>{
  "extensions": {
    "cost": {
      "requestedQueryCost": 72,
      "actualQueryCost": 38,
      "throttleStatus": {
        "maximumAvailable": 1000.0,
        "currentlyAvailable": 962.0,
        "restoreRate": 100.0
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Read those numbers, never hardcode them. Bucket size and restore rate vary by plan tier, and a query that works fine against a large merchant&#8217;s store will fail instantly against a small one. The pattern that holds up is a pre-flight check: before dispatching, compare your estimated cost against <code>currentlyAvailable</code>, and if you&#8217;re short, sleep for the shortfall divided by <code>restoreRate</code>.</p>



<p class="wp-block-paragraph">On APIs that expose their own budget as a queryable field, ask for it in the same round trip rather than spending a second call to find out:</p>



<pre class="wp-block-code"><code>query BudgetCheck {
  rateLimit {
    limit
    cost
    remaining
    resetAt
  }
}</code></pre>



<p class="wp-block-paragraph">Two more things about concurrency. First, if you fan out across Lambda invocations, each worker&#8217;s local view of the bucket is wrong, because the bucket is shared. Multiple workers independently deciding they have headroom is the classic way to get throttled while every individual worker believes it&#8217;s behaving. Either centralise the token accounting or cap concurrency low enough that the arithmetic can&#8217;t go wrong.</p>



<p class="wp-block-paragraph">Second, if the source offers a bulk or async export operation, use it for backfills. Paginating fifty thousand records through the interactive endpoint burns the same budget your incremental syncs need, and it competes with itself.</p>



<h2 class="wp-block-heading">Failure three: cursors that don&#8217;t survive a restart</h2>



<p class="wp-block-paragraph">Most production GraphQL schemas follow the Relay cursor connections convention: <code>first</code> and <code>after</code> arguments, a <code>pageInfo</code> object with <code>hasNextPage</code> and <code>endCursor</code>, and edges carrying opaque per-item cursors.</p>



<pre class="wp-block-code"><code>query Orders($cursor: String) {
  orders(first: 100, after: $cursor) {
    pageInfo {
      hasNextPage
      endCursor
    }
    edges {
      cursor
      node {
        id
        updatedAt
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">The filter argument for incremental sync is not standardised. Some schemas take a query-string filter, some take a typed timestamp argument, some expose nothing at all. Run introspection against the schema and find out rather than guessing, because a wrong argument name is a request error and your run will fail before it starts.</p>



<p class="wp-block-paragraph">The failure here is subtle. Cursors are opaque and their validity is entirely up to the server. Some are stable positional pointers. Some encode a sort key and a snapshot identifier and expire. If a page-47 fetch fails and you retry an hour later with the stored cursor, you may get an error, or you may get results from a shifted window with a hole in the middle. The second outcome is worse, because it looks like it worked.</p>



<p class="wp-block-paragraph">Three habits make restarts safe:</p>



<ol class="wp-block-list">
<li><strong>Checkpoint after the write, not after the fetch.</strong> A cursor recorded before the page lands in S3 is a promise you can&#8217;t keep.</li>

<li><strong>Store a high watermark alongside the cursor.</strong> The cursor is your fast path. The watermark, taken from the newest record you actually committed, is your fallback when the cursor is rejected.</li>

<li><strong>Overlap the watermark deliberately.</strong> Restart a few minutes before the last committed timestamp and deduplicate on the primary key downstream. Re-reading a handful of rows is cheap. Missing rows that arrived during a clock skew is not.</li>
</ol>



<p class="wp-block-paragraph">A DynamoDB item per source stream is enough state for all of this:</p>



<pre class="wp-block-code"><code>{
  "source": "orders",
  "run_id": "01hq7k...",
  "cursor": "eyJsYXN0X2lkIjoxMjM0fQ==",
  "high_watermark": "&lt;updatedAt of newest committed record&gt;",
  "pages_committed": 47,
  "status": "IN_PROGRESS"
}</code></pre>



<p class="wp-block-paragraph">Overlap only works if downstream deduplication is real. Write raw pages with an ingest-time partition and a run identifier in the object key, then let the curated layer resolve duplicates by primary key and latest <code>updatedAt</code>. If you write straight into a merged table with no dedupe step, overlapping restarts will double-count and you&#8217;ll have traded silent loss for silent inflation.</p>



<h2 class="wp-block-heading">Failure four: the schema moved and nobody told you</h2>



<p class="wp-block-paragraph">GraphQL&#8217;s deprecation story is genuinely better than REST&#8217;s. Fields carry a <code>@deprecated</code> directive with a reason string, and introspection exposes it. The problem is that nothing forces you to look.</p>



<p class="wp-block-paragraph">The failure mode is a field that gets deprecated, stops being populated, then eventually gets removed. Stage one is invisible: your query still validates, the field returns null, and your warehouse fills with nulls. Stage two is loud but late: the query becomes invalid and the whole ingest breaks at 3am.</p>



<p class="wp-block-paragraph">Put an introspection diff in CI. Fetch the schema on a schedule, store it as an artifact, and diff against the last known good version. Fail the build on removals or type changes to fields your queries actually select, and warn on new deprecations. Tools in the Apollo and Hasura ecosystems ship schema-diff tooling for this, and rolling your own against the introspection result is not much work either. Either way, the point is that the check runs on a schedule rather than when the pager goes off.</p>



<p class="wp-block-paragraph">The complementary habit is to keep queries narrow. Every field you select is a field that can break you, a field that costs budget, and a field that widens your Glue table. Selecting a whole object because it was convenient during development is a liability you carry indefinitely.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Wiring the GraphQL data ingestion pipeline on AWS</h2>



<p class="wp-block-paragraph">With the failure families understood, the AWS assembly is mostly a set of trade-offs about where the fetch loop lives.</p>



<h3 class="wp-block-heading">Where the fetch loop runs</h3>



<p class="wp-block-paragraph">Step Functions can call an HTTPS endpoint directly with an HTTP Task, no Lambda involved. It uses an EventBridge connection to hold the credentials, which means your token lives in a Secrets Manager secret managed by EventBridge rather than in an environment variable:</p>



<pre class="wp-block-code"><code>"FetchPage": {
  "Type": "Task",
  "Resource": "arn:aws:states:::http:invoke",
  "Parameters": {
    "ApiEndpoint": "https://api.example.com/graphql",
    "Method": "POST",
    "InvocationConfig": {
      "ConnectionArn": "arn:aws:events:region:account-id:connection/name/id"
    },
    "Headers": { "Content-Type": "application/json" },
    "RequestBody": {
      "query.$": "$.query",
      "variables.$": "$.variables"
    }
  },
  "Retry": [{
    "ErrorEquals": [
      "States.Http.StatusCode.429",
      "States.Http.StatusCode.502",
      "States.Http.StatusCode.503",
      "States.Http.StatusCode.504"
    ],
    "IntervalSeconds": 1,
    "BackoffRate": 2,
    "MaxAttempts": 3,
    "JitterStrategy": "FULL"
  }],
  "Next": "InspectBody"
}</code></pre>



<p class="wp-block-paragraph">The execution role needs four permissions: <code>states:InvokeHTTPEndpoint</code>, <code>events:RetrieveConnectionCredentials</code>, and both <code>secretsmanager:GetSecretValue</code> and <code>secretsmanager:DescribeSecret</code> scoped to the EventBridge-managed secret.</p>



<p class="wp-block-paragraph">It&#8217;s a clean pattern, and I like it for small, high-value syncs. Two constraints decide whether it fits. HTTP Task requests time out after 60 seconds, which a deeply nested GraphQL query can exceed without being pathological. And Step Functions reads the response within its standard state payload ceiling of roughly 256 KB, so a page of a hundred rich records will not fit. Also note the retrier above: it fires on HTTP status codes, so it will not catch a throttle that arrives as a 200 with an error in the body. That check has to happen in the state after it.</p>



<p class="wp-block-paragraph">The alternative is a Lambda driver that paginates internally and writes pages straight to S3, returning only the cursor and a count to the state machine. This dodges both limits, gives you a real HTTP client with real backoff, and keeps large payloads out of the workflow&#8217;s state. It costs you per-page retry granularity, which you buy back with the checkpoint record.</p>



<p class="wp-block-paragraph">Distributed Map is worth mentioning for the fan-out case, such as one child execution per tenant or per shard. It gives you high concurrency and a per-item execution history, which is genuinely useful for debugging. Be aware of two things before committing: express child workflows cap at five minutes rather than Lambda&#8217;s fifteen, and practitioners have published benchmarks showing throughput well below what the headline concurrency numbers suggest for small, fast items. For a lot of ingestion work, SQS plus a DynamoDB job table remains the simpler answer.</p>



<p class="wp-block-paragraph">One case genuinely falls outside serverless: sources that require a long-lived session, a persistent websocket subscription, or a synchronisation window longer than fifteen minutes with no resumable cursor. A small always-on worker on a modest VPS from a provider like InterServer or Hetzner, feeding the same S3 landing zone, is less clever and more honest than contorting Lambda around the constraint.</p>



<h3 class="wp-block-heading">The landing zone</h3>



<p class="wp-block-paragraph">Land raw responses first, unmodified, including the <code>errors</code> and <code>extensions</code> keys. Storage is the cheapest thing in this architecture and the raw layer is what lets you reprocess after you discover a parsing bug six weeks later.</p>



<ul class="wp-block-list">
<li><strong>Partition by ingest time, not event time.</strong> Event-time partitioning requires reading the record before you know where to put it, which means you can&#8217;t write until you&#8217;ve parsed. Ingest-time keys let the raw write be dumb and fast.</li>

<li><strong>Put the run identifier in the key.</strong> When you need to unwind one bad run, being able to delete or ignore by prefix beats writing a surgical delete.</li>

<li><strong>Define Glue tables explicitly rather than crawling.</strong> You wrote the selection set, so you already know the schema. A crawler inferring types from JSON will surprise you the first time an optional field is absent from an entire page.</li>

<li><strong>Convert to columnar in the curated layer.</strong> Athena over raw JSON works and is fine for debugging. Parquet is what you want for anything anyone queries repeatedly.</li>
</ul>



<p class="wp-block-paragraph">If your source pushes rather than being polled, such as webhook deliveries carrying GraphQL payloads, Amazon Data Firehose (renamed from Kinesis Data Firehose, with APIs and IAM policies unchanged) will buffer, convert to Parquet and partition on the way to S3 without you running a consumer. For scheduled pull-based syncs it adds a hop you don&#8217;t need.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Runs succeed but a column is all null.</strong> Field errors. Pull a raw response from the landing zone and look at the <code>errors</code> array. The <code>path</code> tells you exactly which field.</li>

<li><strong>Runs succeed but ingest zero rows.</strong> Likely a throttle returned as a 200. Log the response body and the rate-limit headers on empty results, always.</li>

<li><strong>Intermittent failures partway through pagination.</strong> Cursor expiry. Fall back to the watermark and check whether the source documents a cursor lifetime.</li>

<li><strong>Query fails immediately with no <code>data</code> key.</strong> Request error. Validate against current introspection; something in the schema moved.</li>

<li><strong>Throttled despite staying under the documented limit.</strong> Either concurrent workers are sharing a bucket they each think they own, or you&#8217;re being charged requested cost on near-empty pages.</li>

<li><strong>HTTP Task fails on large pages.</strong> The response exceeds the state payload ceiling. Reduce page size or move the fetch into Lambda.</li>

<li><strong>Row counts drift upward over time.</strong> Overlap without deduplication. Check that the curated layer actually resolves on the primary key.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating a 200 as success and never reading the body.</li>

<li>Retrying request errors. They are deterministic; retrying just spends budget to fail identically.</li>

<li>Hardcoding bucket sizes and restore rates that differ per plan tier and change without notice.</li>

<li>Checkpointing the cursor before the data is durable.</li>

<li>Requesting the maximum page size on every call while being charged the worst-case cost for it.</li>

<li>Selecting entire objects because it was easier during development.</li>

<li>Alerting only on execution failure, so a job that succeeds while ingesting nothing never pages anyone.</li>

<li>Overlapping restart windows with no downstream deduplication.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Classify every response into clean, partial, or request-failed before anything downstream sees it.</li>

<li>Persist raw responses with <code>errors</code> and <code>extensions</code> intact, then parse from the raw layer.</li>

<li>Alarm on row-count deltas and partial-error rate, not just on execution status.</li>

<li>Read the throttle state the API hands you and back off before you hit the wall, not after.</li>

<li>Keep both a cursor and a watermark, and know which one is authoritative on restart.</li>

<li>Diff the schema on a schedule so deprecations reach you before they reach your data.</li>

<li>Hold credentials in Secrets Manager or an EventBridge connection, never in a Lambda environment variable.</li>

<li>Trace the pipeline end to end. CloudWatch covers the basics; Grafana or Datadog earn their keep once you&#8217;re correlating child executions across a fan-out.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can Step Functions call a GraphQL API without a Lambda function?</h3>



<p class="wp-block-paragraph">Yes. An HTTP Task using the <code>arn:aws:states:::http:invoke</code> resource posts your query directly, with an EventBridge connection handling authentication. It works well for small pages. The 60-second request timeout and the state payload size ceiling are what push most real ingestion work back into Lambda.</p>



<h3 class="wp-block-heading">How do I detect GraphQL errors when the status code is always 200?</h3>



<p class="wp-block-paragraph">Inspect the body. If there&#8217;s no <code>data</code> key, nothing executed and you have a request error. If <code>data</code> is present alongside a non-empty <code>errors</code> array, you have a partial response and the data is incomplete. Only a response with <code>data</code> and no errors is unambiguously clean.</p>



<h3 class="wp-block-heading">Does AWS AppSync fit into this?</h3>



<p class="wp-block-paragraph">Only if you&#8217;re on the serving side. AppSync is a managed GraphQL API for exposing your own data, so it&#8217;s what you&#8217;d reach for if you wanted clients to push events in via mutations. For pulling data out of somebody else&#8217;s GraphQL API, AppSync isn&#8217;t part of the picture.</p>



<h3 class="wp-block-heading">Should I store raw GraphQL responses or parsed records in S3?</h3>



<p class="wp-block-paragraph">Both, in separate layers. Raw responses are your audit trail and your reprocessing path, and they&#8217;re the only way to answer &#8220;was this field null in the source, or did my parser drop it?&#8221; weeks after the fact. Parsed and columnar output goes in a curated prefix that analysts query.</p>



<h3 class="wp-block-heading">How do I run an incremental sync from a GraphQL API?</h3>



<p class="wp-block-paragraph">Find the schema&#8217;s filter argument for a modification timestamp using introspection, pass your stored watermark into it, and page forward with cursors. Overlap the window slightly on each run and deduplicate on the primary key downstream. If the schema exposes no such filter, you&#8217;re limited to full pulls or webhook-driven change capture.</p>



<h3 class="wp-block-heading">Why am I throttled when I&#8217;m making very few requests?</h3>



<p class="wp-block-paragraph">Cost-based APIs charge for query complexity, not call count. A single deeply nested query pulling a connection inside a connection can consume a large fraction of the bucket by itself. Read the cost figures the API returns and flatten the query if the requested cost is consistently much higher than the actual cost.</p>



<h3 class="wp-block-heading">Is GraphQL a worse ingestion source than REST?</h3>



<p class="wp-block-paragraph">Different, not worse. You control the payload, which cuts transfer and lets you avoid the N+1 fan-out REST forces on nested data. Deprecation metadata is machine-readable, which REST rarely offers. The cost is that error signalling and rate limiting both moved into the body, so naive clients fail silently instead of loudly.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A GraphQL data ingestion pipeline on AWS fails differently from a REST one, and the difference is almost entirely about where the truth lives. In REST, the transport tells you what happened. In GraphQL, the transport tells you the request arrived, and the body tells you what happened. Every failure family in this post traces back to that: partial errors hidden behind a 200, throttles hidden behind a 200, cursors that fail by returning the wrong rows rather than an error.</p>



<p class="wp-block-paragraph">So build the pipeline so that a green run means something. Parse before you commit, alarm on row deltas rather than exit codes, and keep the raw responses so you can prove what the source actually said. Everything else is ordinary AWS plumbing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with a GraphQL ingestion pipeline?</h2>



<p class="wp-block-paragraph">This is the kind of work I take on regularly. Specifically:</p>



<ul class="wp-block-list">
<li>Auditing an existing GraphQL sync for silent partial-error loss and telling you which columns are already affected</li>

<li>Building the fetch, checkpoint and land loop in Step Functions and Lambda with cursors that survive restarts</li>

<li>Fitting a cost-aware throttle to whatever budget model your source actually uses, including multi-worker coordination</li>

<li>Designing the raw and curated S3 layout, Glue table definitions and Athena partitioning so queries stay cheap</li>

<li>Adding introspection diffing to CI so schema deprecations surface in a pull request, not in a dashboard</li>

<li>Wiring CloudWatch metrics and alarms on row-count deltas and partial-error rates, so a green run is actually green</li>
</ul>



<p class="wp-block-paragraph">Send me a run log, a state machine definition, or a raw response with an <code>errors</code> array in it, and I&#8217;ll tell you what it&#8217;s costing you.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/">Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Streaming Shopify Events into AWS Without Losing Orders</title>
		<link>https://john-nessime.com/blog/devops/streaming-shopify-events-into-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 16 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon SQS]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Dead Letter Queue]]></category>
		<category><![CDATA[DynamoDB]]></category>
		<category><![CDATA[Ecommerce Analytics]]></category>
		<category><![CDATA[Event-Driven Architecture]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Partner Event Source]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Reliability Engineering]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[Terraform]]></category>
		<category><![CDATA[Webhooks]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=221</guid>

					<description><![CDATA[<p>Wiring Shopify webhooks into Amazon EventBridge takes an afternoon. Keeping every order is the hard part. A walk through the five failure families that actually bite when streaming Shopify events into AWS: the partner source that silently drops everything, duplicate and out-of-order deliveries, rule patterns that match nothing, targets that fail without a dead-letter queue, and the 64 KB metering rule that quietly inflates the bill.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/streaming-shopify-events-into-aws/">Streaming Shopify Events into AWS Without Losing Orders</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The partner event source in the EventBridge console said <code>Pending</code>. It had said <code>Pending</code> for six days.</p>



<p class="wp-block-paragraph">Nobody noticed, because nothing errored. No 5xx in a log. No failed delivery in Shopify&#8217;s dashboard. No alarm. Shopify had been publishing order events the entire time, and AWS had been throwing every single one of them on the floor.</p>



<p class="wp-block-paragraph">That behaviour is documented, in one short note in the AWS docs: events published to a partner event source that has not been associated with an event bus are dropped immediately and are not persisted at rest. There is no retry for that. There is no buffer. The events are gone, and the only way to get the data back is to go ask the Shopify Admin API for it after the fact.</p>



<p class="wp-block-paragraph">That is the shape of most of the pain in this integration. Streaming Shopify events into AWS is easy to stand up and easy to get quietly wrong, and every one of the quiet failures looks identical from the outside: everything is green, and some of your data isn&#8217;t there.</p>



<p class="wp-block-paragraph">This post walks the five failure families that actually cost you records, plus the reconciliation layer that most teams only build after the first incident. It assumes you can read a rule pattern and an IAM policy. It does not assume you have shipped this before.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">What the pipe actually looks like</h2>



<p class="wp-block-paragraph">Four moving parts, and only two of them live in your account.</p>



<ol class="wp-block-list"><li>A Shopify app holds the webhook subscriptions. Each subscription has a topic and a delivery method. For this path the delivery method is EventBridge and the address is an ARN, not a URL.</li><li>Shopify creates a <strong>partner event source</strong> inside your AWS account, in the region you nominated.</li><li>You associate that source with a <strong>partner event bus</strong>. This is the step everyone forgets.</li><li>Rules on that bus match events and push them at targets: Lambda, SQS, Step Functions, Firehose, whatever fits.</li></ol>



<p class="wp-block-paragraph">The ARN trips people up more than anything else in the setup. Shopify wants the <em>event source</em> ARN, not the event bus ARN. They look similar and only one of them works:</p>



<pre class="wp-block-code"><code># Correct - the event source ARN. Note the empty account field.
arn:aws:events:eu-west-1::event-source/aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;

# Wrong - this is the bus, and Shopify will reject it
arn:aws:events:eu-west-1:123456789012:event-bus/aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;</code></pre>



<p class="wp-block-paragraph">Associating the source is a single call, and both the name and the source name are the same string:</p>



<pre class="wp-block-code"><code># Create the partner event bus that accepts the source
aws events create-event-bus 
  --name "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;" 
  --event-source-name "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;" 
  --region eu-west-1

# Confirm it flipped from PENDING to ACTIVE
aws events describe-event-source 
  --name "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;" 
  --region eu-west-1</code></pre>



<p class="wp-block-paragraph">The same architecture applies if you are not on Shopify. BigCommerce and commercetools both publish to EventBridge as partner sources, and the failure families below are identical because they come from EventBridge&#8217;s semantics, not the store&#8217;s.</p>



<h2 class="wp-block-heading">Failure family one: events that never existed</h2>



<p class="wp-block-paragraph">This is the one from the opening, and it is the most expensive because it is completely silent on both sides.</p>



<p class="wp-block-paragraph">Shopify considers the delivery successful. It handed the event to the partner source, which is its contract. AWS considers nothing to have happened, because an unassociated source has no bus to write to, and EventBridge does not persist events at rest before a bus exists. Your CloudWatch metrics show nothing, because metrics are emitted per bus and per rule, and you have neither.</p>



<p class="wp-block-paragraph">The same class of hole opens up in two other ways:</p>



<ul class="wp-block-list"><li><strong>Region mismatch.</strong> The source is created in the region you gave Shopify. Your bus, your rules, your targets and your dead-letter queues all have to be in that region. A rule in the right account but the wrong region matches nothing, forever, without complaint.</li><li><strong>Environment drift.</strong> A staging store pointed at a production source, or a source created against an account ID that belonged to an old sandbox. Nothing errors. Events just land somewhere you are not looking.</li></ul>



<p class="wp-block-paragraph">The fix is boring and it works: treat the source state as a monitored asset. A scheduled job that calls <code>describe-event-source</code> and alarms if <code>State</code> is anything other than <code>ACTIVE</code> costs you twenty minutes and covers the entire failure family. Put it next to your other synthetic checks, not inside the pipeline it is watching.</p>



<p class="wp-block-paragraph">The second half of that check is a heartbeat on volume. If a bus that normally sees a few thousand events a day sees zero for an hour, that is an incident even when every component reports healthy. Alarm on <code>MatchedEvents</code> hitting zero, not just on errors.</p>



<h2 class="wp-block-heading">Failure family two: events that arrive twice, or backwards</h2>



<p class="wp-block-paragraph">EventBridge is at-least-once. Shopify&#8217;s webhooks are at-least-once. Neither one promises ordering. Put those together and you get two distinct bugs that people usually try to fix with one patch.</p>



<p class="wp-block-paragraph">The duplicate is the obvious one. The same <code>orders/create</code> arrives twice, and if your handler posts to a fulfilment provider or sends a customer email, you have just done it twice. The dedupe key is sitting in the envelope: Shopify puts <code>X-Shopify-Webhook-Id</code> into <code>detail.metadata</code>, and it identifies the delivery. Write it into DynamoDB with a conditional put and a TTL of a few days, and drop the event if the write fails.</p>



<p class="wp-block-paragraph">The out-of-order case is the one that costs you money quietly. An <code>orders/updated</code> carrying a cancelled status arrives before the <code>orders/updated</code> carrying the address change, and your database ends up holding the older state because it was written last. Nothing failed. The row is just wrong, and it will stay wrong until someone complains.</p>



<p class="wp-block-paragraph">The envelope carries what you need for this too. <code>detail.metadata</code> includes <code>X-Shopify-Triggered-At</code>, and the resource in <code>detail.payload</code> carries its own <code>updated_at</code>. Compare before you write, and refuse to apply an update whose timestamp is older than the one already stored.</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow"><p>The dedupe key stops you from doing the work twice. The version check stops you from doing the work backwards. They solve different problems and you need both.</p></blockquote>



<p class="wp-block-paragraph">One thing you can skip on this path: HMAC verification. On the HTTPS delivery method you must verify the signature, because anyone can POST to your endpoint. On the EventBridge path, only the partner account behind the event source is permitted to publish to that bus, and the AWS docs are explicit that adding your own resource policy to a partner bus is rejected. The signature header still rides along in the metadata, but the trust boundary is enforced by AWS rather than by your code.</p>



<h2 class="wp-block-heading">Failure family three: rules that match nothing</h2>



<p class="wp-block-paragraph">Every Shopify event, regardless of topic, arrives with the same <code>detail-type</code>. That single fact invalidates the routing instinct most people bring from AWS service events.</p>



<pre class="wp-block-code"><code>{
  "version": "0",
  "id": "1b8e2e75-b771-e964-f0e6-fbca6a21dad8",
  "detail-type": "shopifyWebhook",
  "source": "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;",
  "account": "123456789012",
  "time": "2022-07-02T12:47:58Z",
  "region": "eu-west-1",
  "resources": [],
  "detail": {
    "payload": {
      "id": 1234567890,
      "title": "Columbia Las Hermosas"
    },
    "metadata": {
      "Content-Type": "application/json",
      "X-Shopify-Topic": "products/update",
      "X-Shopify-Shop-Domain": "example.myshopify.com",
      "X-Shopify-Hmac-SHA256": "...",
      "X-Shopify-Webhook-Id": "...",
      "X-Shopify-API-Version": "...",
      "X-Shopify-Triggered-At": "2022-07-02T12:47:57.989779121Z"
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Two things to take from that envelope. The resource body is nested under <code>detail.payload</code>, not at the top of <code>detail</code>, so a pattern copied from an HTTPS handler will match nothing. And the topic lives in <code>detail.metadata</code>, which is where all your routing has to happen.</p>



<pre class="wp-block-code"><code>// Exact topic match
{
  "detail-type": ["shopifyWebhook"],
  "detail": {
    "metadata": {
      "X-Shopify-Topic": ["orders/create"]
    }
  }
}

// Every orders topic, one rule
{
  "detail-type": ["shopifyWebhook"],
  "detail": {
    "metadata": {
      "X-Shopify-Topic": [{ "prefix": "orders/" }]
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Do not deploy a pattern you have not tested against a real envelope. <code>test-event-pattern</code> answers in a second and saves an afternoon:</p>



<pre class="wp-block-code"><code>aws events test-event-pattern 
  --event-pattern file://pattern.json 
  --event file://sample-event.json</code></pre>



<h3 class="wp-block-heading">One rule or thirty?</h3>



<p class="wp-block-paragraph">There is a real argument for a single catch-all rule that pushes everything into one queue and lets your consumer branch on the topic. It is less infrastructure, it deploys faster, and adding a topic does not require a Terraform run.</p>



<p class="wp-block-paragraph">What you give up is per-topic visibility. <code>MatchedEvents</code>, <code>FailedInvocations</code> and the dead-letter queue are all scoped to the rule. Collapse thirty topics into one rule and you can no longer tell that inventory events stopped three days ago, because the aggregate number still looks fine.</p>



<p class="wp-block-paragraph">The split I reach for first: a dedicated rule for each topic that touches money or fulfilment, and one catch-all for everything else. You get precise alarms where the cost of being wrong is high and low overhead everywhere else.</p>



<h2 class="wp-block-heading">Failure family four: events that arrive and die at the target</h2>



<p class="wp-block-paragraph">By default EventBridge keeps retrying a failed target invocation for up to a day, with exponential backoff and jitter. That is generous, and it is also the reason people assume they do not need a dead-letter queue. They do, for two reasons.</p>



<p class="wp-block-paragraph">First, a whole class of errors gets <em>no</em> retries at all. Missing permissions on the target, a target that no longer exists, an address that will not resolve. EventBridge does not retry those, because retrying cannot help. It sends them straight to the DLQ if one is configured, and drops them if one is not.</p>



<p class="wp-block-paragraph">Second, a day of retries is not much when the failure is a bad deploy discovered on a Friday evening.</p>



<pre class="wp-block-code"><code>aws events put-targets 
  --rule shopify-orders-create 
  --event-bus-name "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;" 
  --targets '[{
    "Id": "order-processor",
    "Arn": "arn:aws:lambda:eu-west-1:123456789012:function:order-processor",
    "RetryPolicy": {
      "MaximumRetryAttempts": 20,
      "MaximumEventAgeInSeconds": 3600
    },
    "DeadLetterConfig": {
      "Arn": "arn:aws:sqs:eu-west-1:123456789012:shopify-orders-dlq"
    }
  }]'</code></pre>



<p class="wp-block-paragraph">Lowering the retry window is deliberate here. Twenty-four hours of retries against a genuinely broken consumer buys you nothing and hides the problem; a shorter window pushes failures into the DLQ where they are visible and countable.</p>



<h3 class="wp-block-heading">The DLQ permission trap</h3>



<p class="wp-block-paragraph">This one catches almost everyone who manages infrastructure as code. Configure a DLQ through the console and AWS attaches the queue policy for you. Configure it through <code>PutTargets</code> — which is what Terraform, CloudFormation and the CLI all do — and you must attach it yourself. Miss it, and you have a dead-letter queue that cannot receive dead letters.</p>



<pre class="wp-block-code"><code>{
  "Sid": "Dead-letter queue permissions",
  "Effect": "Allow",
  "Principal": { "Service": "events.amazonaws.com" },
  "Action": "sqs:SendMessage",
  "Resource": "arn:aws:sqs:eu-west-1:123456789012:shopify-orders-dlq",
  "Condition": {
    "ArnEquals": {
      "aws:SourceArn": "arn:aws:events:eu-west-1:123456789012:rule/shopify-orders-create"
    }
  }
}</code></pre>



<p class="wp-block-paragraph">The metric that catches this is <code>InvocationsFailedToBeSentToDlq</code>. If it is ever non-zero, your safety net has a hole in it and events are being lost at the exact moment you were counting on it. Alarm on it at a threshold of one. It only reports when it is non-zero, so it costs nothing the rest of the time.</p>



<p class="wp-block-paragraph">Two more constraints worth knowing before you design around a DLQ: it must be a standard SQS queue, not FIFO, and it must live in the same region as the rule. Each message carries the error code, the exhausted retry condition, the retry count and both ARNs as message attributes, which is usually enough to triage without opening the payload.</p>



<h2 class="wp-block-heading">Failure family five: the bill</h2>



<p class="wp-block-paragraph">EventBridge does not meter one event as one event. It meters in 64 KB chunks, so an event larger than that bills as multiple events. Rates change and vary by region, so check the current pricing page rather than trusting any number you read in a blog post, but the mechanism is stable and it is what determines your bill.</p>



<p class="wp-block-paragraph">This matters more for commerce than for most event sources. A product update is small. An order with thirty line items, per-item discount allocations, tax lines, shipping lines, note attributes and a stack of metafields is not. Wholesale and subscription stores routinely produce order payloads that cross the chunk boundary, and the same order updated eight times through its lifecycle multiplies that.</p>



<p class="wp-block-paragraph">Three levers, roughly in order of how much they return:</p>



<ul class="wp-block-list"><li><strong>Trim at the subscription.</strong> Shopify&#8217;s webhook subscription API lets you restrict which fields are included in the payload and which metafield namespaces come along. Fields you never read cost you at ingestion, at archive and again at replay. This is the only lever that stops paying for the data before it enters AWS.</li><li><strong>Subscribe to fewer topics.</strong> Broad topics like <code>orders/updated</code> fire on changes you do not care about. If you only act on fulfilment state, subscribe to the fulfilment topics instead of filtering a firehose after you have paid for it.</li><li><strong>Archive selectively, and set retention.</strong> Archives bill for processing, for storage and again for replay. An archive with no retention period grows forever. Archive the topics you would genuinely replay and let the rest go.</li></ul>



<p class="wp-block-paragraph">One structural limit to design around: EventBridge caps the total size of a single event. A payload that exceeds it does not get truncated in a helpful way — the publish fails. Trimming at the subscription protects you here as well as on cost.</p>



<h2 class="wp-block-heading">The layer nobody builds until they need it</h2>



<p class="wp-block-paragraph">Archive and replay is genuinely useful, and it is also routinely misunderstood. Replay re-delivers events that <em>reached the bus</em>. It does nothing at all for the failure family at the top of this post, where the events never reached the bus in the first place. Replay fixes bugs in your consumer. It does not fix gaps in your ingestion.</p>



<p class="wp-block-paragraph">For that you need reconciliation: a scheduled job that queries the Shopify Admin API for resources changed since a stored watermark and compares them against what you hold. It is unglamorous, it is the thing that catches the outage you did not know about, and it is worth building before you need it rather than during the incident.</p>



<ul class="wp-block-list"><li>Run it hourly for orders and fulfilments, daily for products and customers. The cadence should track how expensive being wrong is, not how much data there is.</li><li>Store a watermark per topic and advance it only after a successful full pass. A partial pass that advances the watermark creates the exact gap you built the job to find.</li><li>Compare counts first, records second. A count mismatch is cheap to compute and tells you whether to bother with the expensive comparison.</li><li>Emit the drift as a metric, not just a log line. &#8220;Orders in Shopify but not in our store, last hour&#8221; is a graph worth putting on a dashboard, and it should normally read zero.</li></ul>



<p class="wp-block-paragraph">The reconciliation worker does not need to live in Lambda. It is a long, paginated, rate-limited crawl, which is an awkward fit for a function timeout and a comfortable fit for a small VPS you already run. If you have a box at InterServer or Hetzner sitting there for other jobs, a cron entry and a script is a perfectly respectable answer.</p>



<h2 class="wp-block-heading">Troubleshooting by symptom</h2>



<p class="wp-block-paragraph">Work these in order. Each one is cheap and rules out a whole branch.</p>



<h3 class="wp-block-heading">Nothing is arriving at all</h3>



<ol class="wp-block-list"><li>Run <code>describe-event-source</code>. If <code>State</code> is not <code>ACTIVE</code>, stop here. Everything published so far is gone and you need the reconciliation path.</li><li>Confirm the region of the bus matches the region in the source ARN.</li><li>List your webhook subscriptions through the Admin API and confirm the address is the event-source ARN, not the bus ARN.</li><li>Confirm the subscriptions belong to the app whose access token you are using. Registering with a token from a different app is a common and confusing dead end.</li><li>Check <code>MatchedEvents</code> on the bus with no rule dimension. Non-zero means events are landing and your rules are the problem, not the plumbing.</li></ol>



<h3 class="wp-block-heading">Some topics arrive, order or customer topics do not</h3>



<p class="wp-block-paragraph">This is almost always scopes rather than infrastructure. Order and customer topics sit behind protected customer data access, which is a separate approval in the app configuration on top of the read scopes. Without it, product events flow perfectly and order events silently do not — which looks exactly like a broken rule and is not.</p>



<h3 class="wp-block-heading">The rule matches but the target does nothing</h3>



<p class="wp-block-paragraph">Compare <code>MatchedEvents</code> against <code>SuccessfulInvocationAttempts</code> on the rule. A gap sends you to <code>FailedInvocations</code> and to the DLQ. Check the target&#8217;s resource policy, and check <code>InvocationsFailedToBeSentToDlq</code> before you trust that the DLQ is catching anything.</p>



<h3 class="wp-block-heading">Events arrive, but late</h3>



<p class="wp-block-paragraph">Look at <code>ThrottledRules</code> and at <code>IngestionToInvocationSuccessLatency</code>. Sustained throttling usually means an invocation quota rather than a rule problem, and it shows up first during flash sales, which is the worst possible time to discover it. Load-test the path before a peak event, not after.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list"><li>Creating the partner event source and never associating it with a bus. Silent, total, unrecoverable data loss for the whole window.</li><li>Registering the event bus ARN instead of the event source ARN, then debugging Shopify&#8217;s rejection for an hour.</li><li>Writing rule patterns against the resource shape from an HTTPS webhook, forgetting that the body sits under <code>detail.payload</code>.</li><li>Routing on <code>detail-type</code>. Every Shopify event carries the same one, so a pattern that matches on it alone matches everything.</li><li>Configuring a DLQ through Terraform without the queue policy, and only finding out when you needed it.</li><li>Assuming replay covers ingestion gaps. It replays what reached the bus and nothing else.</li><li>Deduplicating on the resource ID instead of the webhook ID, so legitimate subsequent updates get discarded as duplicates.</li><li>Skipping reconciliation because the pipeline &#8220;works&#8221;. It works right up until it doesn&#8217;t, and that is precisely when you need the other path.</li></ul>



<h2 class="wp-block-heading">Best practices for streaming Shopify events into AWS</h2>



<ul class="wp-block-list"><li>Alarm on the event source state and on <code>MatchedEvents</code> reaching zero. Absence of events is a signal, and it is the only signal you get for the worst failure.</li><li>Dedupe on <code>X-Shopify-Webhook-Id</code> and version-check on <code>X-Shopify-Triggered-At</code>. Two mechanisms, two problems.</li><li>Give every target a DLQ and a retry window you chose deliberately, rather than inheriting the default.</li><li>Keep dedicated rules for money and fulfilment topics so their metrics stay legible; batch the rest behind a catch-all.</li><li>Trim payloads at the Shopify subscription rather than in a Lambda. Filtering after ingestion means you already paid for the bytes.</li><li>Define the whole thing in Terraform or CloudFormation, including the queue policies. This stack has too many one-time console clicks to survive being hand-built twice.</li><li>Point your observability platform at the same bus. Datadog and New Relic are both EventBridge partners, so business events and infrastructure telemetry can share one pipeline instead of two.</li><li>Build reconciliation before your first peak trading period, not after your first missing-order ticket.</li></ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I still need to verify the HMAC signature on the EventBridge path?</h3>



<p class="wp-block-paragraph">No. Only the partner account behind the event source can publish to a partner event bus, and AWS actively rejects attempts to add your own resource policy granting anyone else access. The signature header is still present in the metadata, but the trust boundary is enforced by AWS rather than by your handler. On the HTTPS delivery method, verification remains mandatory.</p>



<h3 class="wp-block-heading">Should I use EventBridge or plain HTTPS webhooks?</h3>



<p class="wp-block-paragraph">HTTPS is simpler, works with any host, and is easier to debug because you can curl your own endpoint. It also puts you on the hook for absorbing burst traffic within a short response deadline, and for keeping the endpoint up well enough that Shopify does not remove the subscription after persistent failures. EventBridge moves that burst absorption to AWS and gives you native fan-out. If your consumers already live in AWS, the operational maths favours EventBridge. If they do not, a dedicated reliability layer such as Hookdeck in front of an HTTPS endpoint is a reasonable alternative and a much smaller change.</p>



<h3 class="wp-block-heading">Can I use one partner event source for multiple stores?</h3>



<p class="wp-block-paragraph">Events from every shop that installed your app flow through the source associated with that app, and the shop is identified by <code>X-Shopify-Shop-Domain</code> in the metadata. You can route per-shop with rule patterns matching that field. For genuine tenant isolation — separate accounts, separate blast radius — you want separate apps and separate sources, because a single bus is a single failure domain.</p>



<h3 class="wp-block-heading">Why do I get duplicate order events even though nothing failed?</h3>



<p class="wp-block-paragraph">Because at-least-once means exactly that. Duplicates are normal operation, not a fault to be investigated. Separately, an order genuinely does change several times shortly after creation — payment capture, risk assessment, post-purchase upsells — so several <code>orders/updated</code> events for one order are expected and are not duplicates at all. Deduplicate on the webhook ID to tell the two apart.</p>



<h3 class="wp-block-heading">What happens to events published while my consumer is broken?</h3>



<p class="wp-block-paragraph">They reach the bus, match your rules, and EventBridge retries the target within your configured window. Once that window is exhausted they go to the DLQ if you have one and are discarded if you do not. The events themselves are not lost at the bus level as long as the source is associated — this is the failure family you can actually engineer your way out of.</p>



<h3 class="wp-block-heading">Can I archive and replay Shopify events?</h3>



<p class="wp-block-paragraph">Yes, with an archive on the partner event bus and an event pattern controlling what gets archived. Budget for three separate charges — processing into the archive, storage while it sits there, and the replay itself — and always set an explicit retention period, because an archive without one grows indefinitely.</p>



<h3 class="wp-block-heading">Does this work the same way for BigCommerce or commercetools?</h3>



<p class="wp-block-paragraph">The AWS half is identical: partner source, association step, bus, rules, targets, and every failure family in this post. What differs is the envelope shape and how you register subscriptions on the vendor side. The association gap in particular bites the same way regardless of which platform is publishing.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Almost everything about streaming Shopify events into AWS degrades loudly. Targets throw errors, retries show up as metrics, dead letters pile up in a queue you can see. Those are the failures you will handle correctly, because they announce themselves.</p>



<p class="wp-block-paragraph">The one that will actually hurt you is the one that reports success on both sides while dropping every event on the floor. Association state and event volume are the two signals that catch it, and neither one appears on any dashboard by default. Add them on day one, before you write the first rule. Everything else in this post can be fixed after the fact; that one cannot.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a hand with your event pipeline?</h2>



<p class="wp-block-paragraph">Most of my work on this stack is either standing it up properly the first time or working out where records went after someone else stood it up. Things I can help with:</p>



<ul class="wp-block-list"><li>Building the Shopify-to-EventBridge path end to end in Terraform, including the queue policies and retry configuration that the console quietly does for you.</li><li>Auditing an existing pipeline for silent loss: source association, region drift, missing DLQ permissions, rules that have been matching nothing since the day they shipped.</li><li>Designing the idempotency and ordering layer — dedupe store, TTLs, version checks — so replays and duplicates stop corrupting downstream state.</li><li>Writing the reconciliation job against the Admin API, with watermarks, drift metrics and alarms that fire before a customer does.</li><li>Cutting EventBridge spend by trimming payloads at the subscription and rationalising archive retention, without losing anything you actually query.</li><li>Load-testing the whole path ahead of a peak trading period so throttling shows up in a test window rather than on the day.</li></ul>



<p class="wp-block-paragraph">If you have a rule pattern that isn&#8217;t matching, a DLQ that&#8217;s mysteriously empty, or a bill that grew faster than your order volume, send me the pattern, the metric graph or the line item and I&#8217;ll tell you what I&#8217;d look at first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/streaming-shopify-events-into-aws/">Streaming Shopify Events into AWS Without Losing Orders</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>CloudWatch Data Pipeline Monitoring: Catching the Runs That Succeed and Deliver Nothing</title>
		<link>https://john-nessime.com/blog/devops/cloudwatch-data-pipeline-monitoring/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 15 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Anomaly Detection]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Freshness]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Observability]]></category>
		<category><![CDATA[Embedded Metric Format]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[Step Functions]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=225</guid>

					<description><![CDATA[<p>Your SaaS pipeline will fail far more often by succeeding at nothing than by throwing an exception, and every CloudWatch default treats an absent metric as a non-event. Here are the four signals worth alarming on: liveness, volume, freshness and shape, plus the missing-data traps that leave alarms permanently green.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/cloudwatch-data-pipeline-monitoring/">CloudWatch Data Pipeline Monitoring: Catching the Runs That Succeed and Deliver Nothing</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said the Salesforce numbers looked wrong on the executive dashboard. I opened the pipeline. Every run for the past several days was green: the state machine succeeded, no failed states, no Lambda errors, no alarms in the account. The pipeline had been running flawlessly and writing an empty file every single time.</p>



<p class="wp-block-paragraph">That gap is the whole problem with CloudWatch data pipeline monitoring, and it is why most teams find out about broken SaaS integrations from a business user rather than from a page. A web server that breaks returns 500s, and your error-rate alarm fires. A pipeline that breaks frequently returns exit code 0, writes a zero-byte object, and reports success. The connected app got revoked, a scope was dropped during an admin cleanup, or a filter clause quietly started matching nothing. None of the AWS-native metrics have an opinion about any of that.</p>



<p class="wp-block-paragraph">This post is about what to measure instead: the four signals worth alarming on, the CloudWatch defaults that leave those alarms silently in the wrong state, how to emit custom metrics without turning your bill into a line item someone asks about, and what to check when a metric exists but the alarm still refuses to fire.</p>



<h2 class="wp-block-heading">&#8220;Did the job fail?&#8221; is the weakest question you can ask</h2>



<p class="wp-block-paragraph">Job status is a proxy for pipeline health, and a bad one. It tells you the orchestrator finished its instructions. It says nothing about whether those instructions produced data.</p>



<p class="wp-block-paragraph">SaaS sources make this worse than usual. A REST API returning HTTP 200 with an empty result array is, to your extraction code, a completely successful call. Pagination that stops on the first page because a cursor was reset is successful. An OAuth refresh that returns a token with fewer scopes than yesterday is successful right up until one object starts coming back empty. Every one of those is a green run.</p>



<p class="wp-block-paragraph">Build the monitoring around four questions the job outcome cannot answer:</p>



<ol class="wp-block-list"><li><strong>Liveness</strong> &#8211; did the run happen at all?</li><li><strong>Volume</strong> &#8211; did it move a plausible amount of data?</li><li><strong>Freshness</strong> &#8211; how old is the newest record that actually landed?</li><li><strong>Shape</strong> &#8211; are the fields and values still what downstream expects?</li></ol>



<p class="wp-block-paragraph">Each needs a different kind of metric, and each fails silently in its own way.</p>



<h2 class="wp-block-heading">Signal 1: liveness, and the missing-data trap that hides it</h2>



<p class="wp-block-paragraph">Step Functions publishes execution metrics to the <code>AWS/States</code> namespace, including <code>ExecutionsStarted</code>, <code>ExecutionsSucceeded</code>, <code>ExecutionsFailed</code>, <code>ExecutionsTimedOut</code>, <code>ExecutionsAborted</code> and <code>ExecutionTime</code>. The obvious move is an alarm on <code>ExecutionsFailed</code>. Most teams stop there, and that alarm is close to useless on its own.</p>



<p class="wp-block-paragraph">Here is why. Those metrics are published as a result of executions. If somebody disables the EventBridge schedule rule during an incident and forgets to re-enable it, no execution starts, so no data point is published for that dimension at all. Your alarm has nothing to evaluate.</p>



<p class="wp-block-paragraph">What happens next depends entirely on a parameter most people never set. CloudWatch alarms accept a <code>TreatMissingData</code> setting with four values: <code>breaching</code>, <code>notBreaching</code>, <code>ignore</code> and <code>missing</code>. The default is <code>missing</code>, which means the alarm keeps its current state when there is nothing to evaluate. An alarm that was sitting in OK stays in OK. Forever. It is green because it is blind, and there is no visual difference between the two.</p>



<p class="wp-block-paragraph">The fix is not to change <code>TreatMissingData</code> on the failure alarm. It is to add a separate heartbeat metric that your pipeline emits on completion, and alarm on the <em>absence</em> of that heartbeat:</p>



<pre class="wp-block-code"><code># Emitted at the very end of a successful run, per source system.
aws cloudwatch put-metric-data 
  --namespace "Pipelines/Ingest" 
  --metric-name "RunCompleted" 
  --value 1 
  --unit Count 
  --dimensions Source=salesforce,Stage=extract</code></pre>



<p class="wp-block-paragraph">Then the alarm that actually catches a dead scheduler:</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-alarm 
  --alarm-name "ingest-salesforce-no-heartbeat" 
  --namespace "Pipelines/Ingest" 
  --metric-name "RunCompleted" 
  --dimensions Name=Source,Value=salesforce Name=Stage,Value=extract 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 2 
  --datapoints-to-alarm 2 
  --threshold 1 
  --comparison-operator LessThanThreshold 
  --treat-missing-data breaching 
  --alarm-actions arn:aws:sns:REGION:ACCOUNT_ID:pipeline-alerts</code></pre>



<p class="wp-block-paragraph">The two flags doing the real work are <code>--comparison-operator LessThanThreshold</code> and <code>--treat-missing-data breaching</code>. The first catches a run that completes but emits nothing. The second catches a run that never happens at all. Without the second flag, silence reads as health.</p>



<p class="wp-block-paragraph">Set <code>--period</code> comfortably longer than your schedule interval so one slow run does not page anyone, and use <code>--datapoints-to-alarm</code> to require consecutive breaches. One caveat: <code>breaching</code> is right for a heartbeat you control, but a poor blanket default across the account, because plenty of AWS metrics are only emitted when non-zero. Throttling counters are legitimately sparse, and forcing them to breach on silence just makes noise.</p>



<h2 class="wp-block-heading">Signal 2: volume, and why a static threshold beats anomaly detection at first</h2>



<p class="wp-block-paragraph">A heartbeat proves the run happened. It does not prove the run did anything. Emit a row count, per source, per object:</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-data 
  --namespace "Pipelines/Ingest" 
  --metric-name "RecordsWritten" 
  --value 14238 
  --unit Count 
  --dimensions Source=salesforce,Object=opportunity</code></pre>



<p class="wp-block-paragraph">Alarm on that being unexpectedly low rather than on it being zero. Zero-row alarms feel precise, but the more common failure is partial: a cursor that only advances through the first page, a rate limit that truncates the pull, an incremental sync that silently narrows its window. Those deliver a fraction of the expected rows, not none of them.</p>



<p class="wp-block-paragraph">There is a subtlety here that trips people. If a run fails before emitting <code>RecordsWritten</code> at all, a low-volume alarm on that metric sees missing data, not a zero. Metric math fixes it cleanly:</p>



<pre class="wp-block-code"><code>FILL(m1, 0)</code></pre>



<p class="wp-block-paragraph">Wrapping the metric in <code>FILL</code> with a value of 0 substitutes zero for every empty period, so an absent run and an empty run both breach the same alarm. You get one alert instead of two competing ones, and it means what you think it means.</p>



<p class="wp-block-paragraph">CloudWatch anomaly detection is the obvious upgrade for seasonal volume. It models expected behaviour and alarms on deviation from the band, which handles a Monday-morning spike that a static threshold cannot. I still would not reach for it first: it needs a stretch of representative history before the band is trustworthy, and during that stretch it will be confidently wrong. Start with a static floor well below your observed minimum, and move to anomaly detection once you know what normal looks like and can see the fixed number fighting you.</p>



<h2 class="wp-block-heading">Signal 3: freshness, the metric almost nobody emits</h2>



<p class="wp-block-paragraph">This is the one that separates a pipeline you can trust from one you merely watch. Liveness and volume can both look perfect while the data is stale, because a job that re-reads the same window every run produces rows, produces a heartbeat, and produces nothing new.</p>



<p class="wp-block-paragraph">The metric to emit is lag: the difference in seconds between now and the maximum source-side modification timestamp in the batch you just landed. Not your own processing timestamp, which will always look current. The source system&#8217;s timestamp, the one that tells you how far behind the truth you are.</p>



<pre class="wp-block-code"><code>import time
import boto3

cw = boto3.client("cloudwatch")

def emit_freshness(source, obj, max_source_ts_epoch):
    """max_source_ts_epoch: max LastModifiedDate in this batch, as epoch seconds."""
    lag_seconds = int(time.time()) - int(max_source_ts_epoch)
    cw.put_metric_data(
        Namespace="Pipelines/Ingest",
        MetricData=[{
            "MetricName": "SourceLagSeconds",
            "Value": lag_seconds,
            "Unit": "Seconds",
            "Dimensions": [
                {"Name": "Source", "Value": source},
                {"Name": "Object", "Value": obj},
            ],
        }],
    )</code></pre>



<p class="wp-block-paragraph">Alarm with <code>GreaterThanThreshold</code>, and set the threshold to roughly two schedule intervals plus your worst observed run duration. An hourly pipeline that takes twelve minutes on a bad day gets a threshold somewhere around two and a half hours. Tighter than that and you page on ordinary jitter.</p>



<p class="wp-block-paragraph">Freshness is also the one metric worth putting in front of non-engineers. &#8220;Your Salesforce data is currently 40 minutes behind&#8221; is a sentence a stakeholder can act on; &#8220;ExecutionsSucceeded is 1&#8221; is not. If you surface dashboards to a business audience through Grafana Cloud or QuickSight over the CloudWatch data source, lag is the panel that earns its space.</p>



<h2 class="wp-block-heading">Signal 4: shape, using logs instead of counters</h2>



<p class="wp-block-paragraph">Shape problems are schema drift and value drift. A SaaS admin renames a custom field, a picklist gains a value your mapping does not handle, a required field starts arriving null for a subset of records. The rows keep flowing. The row count looks fine. Downstream joins quietly drop records.</p>



<p class="wp-block-paragraph">Counters alone are the wrong shape here, because what you need at 2am is not &#8220;the null rate rose&#8221;, it is &#8220;which records, and what changed&#8221;. Embedded Metric Format is built for exactly that split: you write one structured JSON log line, CloudWatch extracts the numbers as real metrics you can alarm on, and the full event stays queryable in Logs Insights for the investigation.</p>



<pre class="wp-block-code"><code>{
  "_aws": {
    "Timestamp": 1700000000000,
    "CloudWatchMetrics": [
      {
        "Namespace": "Pipelines/Quality",
        "Dimensions": [["Source", "Object"]],
        "Metrics": [
          { "Name": "NullRequiredFields", "Unit": "Count" },
          { "Name": "UnmappedFields", "Unit": "Count" }
        ]
      }
    ]
  },
  "Source": "salesforce",
  "Object": "opportunity",
  "NullRequiredFields": 412,
  "UnmappedFields": 2,
  "unmapped_field_names": ["Renewal_Tier__c", "Partner_Code__c"],
  "batch_id": "2f8c1a9e"
}</code></pre>



<p class="wp-block-paragraph">Two things to get right. Every name listed in <code>Dimensions</code> must also exist as a top-level property in the same JSON object, or extraction silently produces nothing. And note that <code>unmapped_field_names</code> and <code>batch_id</code> are properties, not dimensions: they ride along in the log event for querying but never become metric dimensions, which is exactly what you want.</p>



<p class="wp-block-paragraph">When the alarm fires, the investigation is a Logs Insights query against the same log group:</p>



<pre class="wp-block-code"><code>fields @timestamp, Object, UnmappedFields, unmapped_field_names
| filter Source = "salesforce" and UnmappedFields &gt; 0
| sort @timestamp desc
| limit 50</code></pre>



<p class="wp-block-paragraph">That hands you the new field names in seconds, which is the difference between a five-minute fix and an hour of guessing.</p>



<h2 class="wp-block-heading">Keeping CloudWatch data pipeline monitoring off the top of your bill</h2>



<p class="wp-block-paragraph">Custom metrics and log ingestion are consistently the two largest CloudWatch line items for data teams. Neither has a flat fee, so the cost is a direct function of choices you make in code. Three matter far more than the rest.</p>



<h3 class="wp-block-heading">Cardinality is the cost lever</h3>



<p class="wp-block-paragraph">A custom metric is billed per unique combination of namespace, metric name and dimension values. <code>Source</code> with five values and <code>Object</code> with twenty gives you a hundred metrics per metric name. Add <code>Tenant</code> with two hundred values and you have twenty thousand. Multiply by four metric names and the number stops being a rounding error.</p>



<p class="wp-block-paragraph">Never put a run ID, batch ID, record ID or raw error string in a dimension. Those belong in the log event. Dimensions are for things you would actually build an alarm on, and you are not going to build an alarm per batch.</p>



<h3 class="wp-block-heading">EMF over PutMetricData for anything high-frequency</h3>



<p class="wp-block-paragraph"><code>PutMetricData</code> is a synchronous API call on your critical path. In Lambda that means duration you pay for, plus a request that can throttle and needs handling. EMF avoids both: you write JSON to stdout, CloudWatch Logs extracts the metrics asynchronously, your function returns. For a per-run summary either is fine. For anything per batch or per record, EMF is the one I reach for.</p>



<h3 class="wp-block-heading">The Infrequent Access log class will break this silently</h3>



<p class="wp-block-paragraph">This one has caught competent people. CloudWatch Logs offers a Standard and an Infrequent Access log class, and IA has a lower ingestion price, which makes it an obvious target during a cost review. According to the AWS documentation, the IA class does not support embedded metric format, metric filters or subscription filters. Logs Insights queries still work.</p>



<p class="wp-block-paragraph">So if somebody moves your pipeline log group to IA to save money, every EMF-derived metric stops being extracted. The alarms built on those metrics go to <code>INSUFFICIENT_DATA</code>, or worse, sit in OK if they use the default missing-data behaviour. The logs still arrive. The dashboard still exists. The metrics are just gone.</p>



<p class="wp-block-paragraph">Log class cannot be changed after a log group is created, so recovery means a new Standard log group and redirecting output to it. Keep metric-backing log groups on Standard and point IA at the genuinely archival stuff: verbose debug output, raw payload dumps, anything you only read during a forensic investigation.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Routing: one page, not four</h2>



<p class="wp-block-paragraph">Build all four signals and one broken pipeline fires the heartbeat, volume and freshness alarms within minutes of each other. Three pages, one problem. That is how alert fatigue starts.</p>



<p class="wp-block-paragraph">Composite alarms fix it. Build one composite alarm per pipeline whose rule combines the child alarms with OR, put the notification action there, and leave the child alarms with no actions at all. They become diagnostic detail you read after the page rather than three separate interruptions.</p>



<p class="wp-block-paragraph">SNS to email is fine for one person and poor past that, since it has no deduplication, escalation or acknowledgement. With more than one person on call, route through something with a schedule such as PagerDuty, Opsgenie or Better Stack. If your team already lives in Grafana Cloud or Datadog for application monitoring, pulling CloudWatch alarm state in through their AWS integration beats maintaining two alerting surfaces.</p>



<h2 class="wp-block-heading">Troubleshooting: the metric exists but the alarm never fires</h2>



<p class="wp-block-paragraph">Roughly in order of how often I have seen each one.</p>



<ul class="wp-block-list"><li><strong>Dimension mismatch.</strong> The alarm&#8217;s dimension set must match the published metric&#8217;s dimension set exactly, all of them, in the same values. A metric published with <code>Source</code> and <code>Object</code> is a different metric from one published with only <code>Source</code>. Run <code>aws cloudwatch list-metrics --namespace "Pipelines/Ingest"</code> and compare the output against your alarm definition character by character.</li><li><strong>Alarm stuck in INSUFFICIENT_DATA.</strong> The metric is not being published at all for that dimension set, or the period is shorter than the publishing interval so most periods are empty. Lengthen the period to at least the emission interval.</li><li><strong>Alarm stuck in OK during a real outage.</strong> Classic default <code>missing</code> behaviour. Any alarm whose purpose is to detect absence needs <code>--treat-missing-data breaching</code>.</li><li><strong>EMF metrics never appear.</strong> Check the log group&#8217;s class first, then confirm every dimension name in the <code>Dimensions</code> array exists as a top-level property in the same JSON object, then confirm the JSON is a single line with no wrapping prefix from your logger.</li><li><strong>Freshness spikes on a schedule.</strong> Almost always a timezone mismatch between how the SaaS API reports modification timestamps and how you convert them. Normalise everything to UTC epoch seconds at the boundary and stop converting anywhere else.</li><li><strong>Row counts double intermittently.</strong> Retries emitting the metric twice for one logical run. Emit volume metrics once, after the write is confirmed, not inside the retried block.</li></ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list"><li>Alarming only on job failure, which misses every silent-success failure mode.</li><li>Leaving <code>TreatMissingData</code> at its default on alarms designed to detect absence.</li><li>Alarming on zero rows instead of on unexpectedly low rows, so partial extractions slip through.</li><li>Emitting a processing timestamp as the freshness metric, which is always current and therefore always useless.</li><li>Putting high-cardinality identifiers into metric dimensions and then being surprised by the bill.</li><li>Moving pipeline log groups to the Infrequent Access class without checking what depends on metric extraction.</li><li>Wiring notification actions to every child alarm instead of to one composite alarm per pipeline.</li><li>Building dashboards nobody looks at instead of alarms that reach a person.</li></ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list"><li>Emit liveness, volume, freshness and a shape metric for every source. Four metrics per source is a small amount of code and covers most of what actually breaks.</li><li>Define alarms in Terraform or CloudFormation alongside the pipeline, so a new source cannot ship without monitoring.</li><li>Use consistent dimension names across every pipeline. <code>Source</code> and <code>Object</code> everywhere makes search expressions and dashboards trivial; a mix of <code>source</code>, <code>system</code> and <code>connector</code> makes them impossible.</li><li>Set alarm descriptions to a one-line runbook. The person paged at 3am should not have to guess what the alarm means.</li><li>Test the absence case deliberately. Disable the schedule in a non-production account and confirm the heartbeat alarm fires. An untested absence alarm is an assumption.</li><li>Set explicit log retention on every pipeline log group. Never-expire is the default and it compounds quietly.</li><li>Keep an eye on the source side too, not just yours. Most SaaS APIs publish their own rate limit or API usage counters, and pulling those into a metric is often the earliest possible warning.</li></ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Is CloudWatch enough on its own for data pipeline monitoring?</h3>



<p class="wp-block-paragraph">For alerting, yes, if your workloads are on AWS. It has metrics, alarms, composite alarms and log querying, and it is already collecting service metrics you would otherwise have to ship somewhere. Where it gets weak is exploratory analysis and cross-source correlation, which is where teams add Grafana, Datadog or a dedicated data-observability tool on top. Run CloudWatch for the paging path first and add the analysis layer when you have an actual gap.</p>



<h3 class="wp-block-heading">How do I monitor a pipeline that runs on a managed service like Glue or AppFlow?</h3>



<p class="wp-block-paragraph">The same four signals apply; only the emission point changes. Managed services publish their own metrics, but those describe the service&#8217;s execution, not your data. Where you cannot add code inside the job, hook the completion event through EventBridge to a small Lambda function that inspects what landed and emits the volume and freshness metrics. It is a handful of lines and it is the only way to get data-level signals out of a job you do not control.</p>



<h3 class="wp-block-heading">Should each tenant get its own metric in a multi-tenant pipeline?</h3>



<p class="wp-block-paragraph">Usually not as a dimension. Tenant counts grow, and cardinality is billed per unique dimension combination. Emit aggregate metrics with <code>Source</code> and <code>Object</code> dimensions, put the tenant identifier in the log event as a property, and use Logs Insights to slice by tenant when you need to. Break out a dimension per tenant only for the small number of accounts where a contractual SLA makes it worth the cost.</p>



<h3 class="wp-block-heading">What is a sensible freshness threshold?</h3>



<p class="wp-block-paragraph">Start at two schedule intervals plus your worst observed run duration, then adjust after watching real data for a couple of weeks. The threshold that matters is the one the business would notice, so if a stakeholder can tolerate half a day of staleness on a nightly export, do not page at four hours. Match the alarm to the actual consequence.</p>



<h3 class="wp-block-heading">Why does my alarm sit in INSUFFICIENT_DATA instead of ALARM?</h3>



<p class="wp-block-paragraph">CloudWatch has no data points to evaluate for that metric and dimension set. Either nothing is publishing, the dimensions do not match what is published, or the alarm period is shorter than the interval at which you emit. Confirm the metric exists with <code>list-metrics</code> first, and only then look at the alarm configuration.</p>



<h3 class="wp-block-heading">How much does this cost to run?</h3>



<p class="wp-block-paragraph">It depends on three things you control: the number of unique custom metrics, the volume of log data ingested, and how much data your Logs Insights queries scan. Alarms and dashboards are minor by comparison. Keep dimension cardinality low, set retention deliberately, and query narrow time windows, and the monitoring stays well below the cost of the pipeline it protects. Check current rates on the AWS pricing page rather than trusting any number in a blog post, including this one.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Effective CloudWatch data pipeline monitoring is built around absence, not around errors. Your SaaS integrations will fail far more often by succeeding at nothing than by throwing an exception, and every default in CloudWatch treats an absent metric as a non-event.</p>



<p class="wp-block-paragraph">If you only change one thing after reading this, add a heartbeat metric with <code>--treat-missing-data breaching</code> to your most important pipeline, then go and disable its schedule in a test account and confirm the alarm actually fires. If it does not, you have just found out on your own terms instead of from a stakeholder asking why the numbers look wrong.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your pipeline monitoring?</h2>



<p class="wp-block-paragraph">I work with teams running SaaS-to-warehouse pipelines on AWS, usually at the point where something broke quietly and nobody wants that to happen twice. Things I can help with here:</p>



<ul class="wp-block-list"><li>Auditing existing CloudWatch alarms for silent-failure gaps, particularly missing-data configuration and dimension mismatches that leave alarms permanently green.</li><li>Instrumenting liveness, volume, freshness and data-quality metrics into pipelines built on Lambda, Glue, Step Functions or self-hosted schedulers.</li><li>Designing EMF logging and metric namespaces that stay useful as source and object counts grow, without cardinality running away.</li><li>Building composite alarms and routing so one broken pipeline produces one actionable page rather than a cluster of them.</li><li>Cutting CloudWatch spend on log ingestion and custom metrics without losing the signals you actually rely on.</li><li>Writing Terraform modules that ship monitoring alongside every new source connector, so coverage is structural rather than remembered.</li></ul>



<p class="wp-block-paragraph">If you have a pipeline that has gone quiet, send me the alarm configuration and a Logs Insights output from the run in question and I will tell you what it is missing.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/cloudwatch-data-pipeline-monitoring/">CloudWatch Data Pipeline Monitoring: Catching the Runs That Succeed and Deliver Nothing</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
