<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Ecommerce Analytics | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/ecommerce-analytics/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/ecommerce-analytics/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Mon, 26 Oct 2026 06:37:42 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Ecommerce Analytics | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/ecommerce-analytics/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Building an Ecommerce Recommendation Pipeline on AWS: The Parts That Bite</title>
		<link>https://john-nessime.com/blog/cloud-computing/aws-ecommerce-recommendation-pipeline/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 26 Oct 2026 06:37:42 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Machine Learning]]></category>
		<category><![CDATA[Retail Technology]]></category>
		<category><![CDATA[Amazon Personalize]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Batch Processing]]></category>
		<category><![CDATA[Clickstream Ingestion]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Ecommerce Analytics]]></category>
		<category><![CDATA[Idle Resources]]></category>
		<category><![CDATA[Item Cold Start]]></category>
		<category><![CDATA[Kinesis Data Streams]]></category>
		<category><![CDATA[Model Serving]]></category>
		<category><![CDATA[Recommendation Systems]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Training-Serving Skew]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=894</guid>

					<description><![CDATA[<p>An idle Amazon Personalize setup does not throw errors. It just bills. A practical walkthrough of the five decisions that shape an AWS ecommerce recommendation pipeline: domain recommenders versus custom recipes, bulk versus streaming ingestion, schema limits, training cadence, and real-time versus batch serving, with the failure modes that stay invisible until the invoice arrives.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/aws-ecommerce-recommendation-pipeline/">Building an Ecommerce Recommendation Pipeline on AWS: The Parts That Bite</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question usually arrives from finance rather than from engineering. Something along the lines of: why is there an Amazon Personalize charge on an account where nobody has shipped a recommendation carousel yet?</p>



<p class="wp-block-paragraph">The answer is almost always the same. Someone followed a tutorial, created a dataset group, trained a solution, deployed a campaign to see if the output looked reasonable, then got pulled onto something else. The campaign is still holding a throughput floor. The solution is still retraining on a schedule. Neither is wired to a storefront. Both are billing.</p>



<p class="wp-block-paragraph">I want that failure mode first because it is invisible. A broken AWS ecommerce recommendation pipeline throws errors and gets fixed by Friday. An idle one throws nothing. It sits there being technically correct and quietly expensive.</p>



<p class="wp-block-paragraph">This post covers the five decisions that actually shape the pipeline: which resource type to train, how events get in, what the schema can hold, how fresh the model needs to be, and how a recommendation reaches the page. I will flag the spots where the default behavior is not the behavior you want.</p>



<h2 class="wp-block-heading">The bill arrives before the carousel does</h2>



<p class="wp-block-paragraph">Two things in Personalize cost money on a clock rather than on demand, and both default toward spending.</p>



<p class="wp-block-paragraph"><strong>Solution training.</strong> New custom solutions use automatic training by default. Personalize creates a new solution version every seven days unless you say otherwise, and the cadence is configurable between one and thirty days. The first run starts within an hour of the solution going active. AWS is explicit that you incur training costs while the solution is active, and that switching automatic training off is how you stop them.</p>



<p class="wp-block-paragraph"><strong>Inference capacity.</strong> A campaign has <code>minProvisionedTPS</code>, a domain recommender has <code>minRecommendationRequestsPerSecond</code>. Both default to 1. Both set a billing floor. Personalize scales above the floor as traffic climbs but never below it, and you are billed on the higher of the floor and your actual rate. A campaign serving zero requests still bills its floor for every hour it stays active.</p>



<p class="wp-block-paragraph">Two habits fix most of this. While you are evaluating rather than serving, create solutions with automatic training off and trigger training yourself. And delete campaigns you are not calling instead of leaving them parked.</p>



<pre class="wp-block-code"><code>import boto3

personalize = boto3.client('personalize')

# Automatic training off while evaluating.
# You then call create_solution_version() yourself when you want a model.
response = personalize.create_solution(
    name='storefront-user-personalization',
    datasetGroupArn='arn:aws:personalize:REGION:ACCOUNT:dataset-group/NAME',
    recipeArn='arn:aws:personalize:::recipe/aws-user-personalization-v2',
    performAutoTraining=False
)
print(response['solutionArn'])</code></pre>



<h2 class="wp-block-heading">Decision one: domain recommender or custom solution</h2>



<p class="wp-block-paragraph">A domain dataset group created with the ECOMMERCE domain exposes pre-configured use cases. A custom dataset group makes you pick a recipe and configure the solution yourself.</p>



<p class="wp-block-paragraph">The ECOMMERCE use cases map onto the modules you were going to build anyway: Recommended for you, Frequently bought together, Customers who viewed X also viewed, Best sellers, Most viewed. Each carries an event-type requirement. The purchase-driven ones need at minimum a thousand Purchase events; the view-driven ones need at minimum a thousand View events. That is a useful early check on your event taxonomy, before you spend a training run finding out.</p>



<p class="wp-block-paragraph">Where the domain route wins: you skip the recipe argument, the defaults are sensible, and you reach a testable result faster. For a catalog under a few thousand SKUs with a normal retail event mix, this is the one I reach for first.</p>



<p class="wp-block-paragraph">Where it hurts: you are constrained to the shape of the provided use cases, you get less control over hyperparameters, and you cannot create next best action resources inside a domain dataset group at all. If you need to recommend actions rather than products, you are in custom territory.</p>



<p class="wp-block-paragraph">On the custom side the recipes worth knowing are User-Personalization-v2 for a personalized feed, Similar-Items for a related-products rail, Personalized-Ranking-v2 for reordering a category or search page, Trending-Now for what is moving, and Popularity-Count as the baseline you should measure against. The v2 recipes use a transformer architecture, handle much larger catalogs, and return item metadata in the inference response, saving a catalog lookup on the render path. They also bill on a different model from the older recipes, so do not carry a cost estimate across.</p>



<p class="wp-block-paragraph">One thing to watch while reading around: the legacy HRNN recipes are no longer available. Their documentation stays online for reference and plenty of third-party tutorials still use them. If a guide points a solution at <code>aws-hrnn</code>, its advice about cold items and training cadence is stale too.</p>



<h2 class="wp-block-heading">Decision two: how interaction data gets in</h2>



<h3 class="wp-block-heading">Bulk imports replace unless you tell them not to</h3>



<p class="wp-block-paragraph">This is the one that costs people a month of confused model quality.</p>



<p class="wp-block-paragraph">A dataset import job defaults to <code>FULL</code>, which overwrites all existing bulk data in the dataset. It leaves records you sent individually through the event tracker alone, but everything previously loaded from S3 is replaced by whatever is in this job. The alternative is <code>INCREMENTAL</code>, which appends and only replaces records sharing an ID.</p>



<p class="wp-block-paragraph">Picture a nightly export producing a rolling thirty-day extract, fed into an import job left on the default. Every night the dataset is truncated to thirty days. Nothing errors. The job reports success. Model quality just degrades as the training window shrinks, and it is not obvious why, because the pipeline is green.</p>



<pre class="wp-block-code"><code>import boto3

personalize = boto3.client('personalize')

# INCREMENTAL appends to what is already there.
# FULL, the default, replaces all previously bulk-imported records.
personalize.create_dataset_import_job(
    jobName='interactions-nightly',
    datasetArn='arn:aws:personalize:REGION:ACCOUNT:dataset/NAME/INTERACTIONS',
    dataSource={'dataLocation': 's3://your-bucket/interactions/'},
    roleArn='arn:aws:iam::ACCOUNT:role/PersonalizeS3Role',
    importMode='INCREMENTAL'
)</code></pre>



<p class="wp-block-paragraph">Two limits shape the schedule. An incremental interactions import is capped well below a full one, so treat it as a delta mechanism and not a backfill. And an interactions import job needs a minimum number of new records to be accepted, so a low-traffic store cannot run one every fifteen minutes and expect success. Batch the deltas into a cadence that clears the floor.</p>



<h3 class="wp-block-heading">Streaming events through an event tracker</h3>



<p class="wp-block-paragraph">The live path is a separate API. Create an event tracker, take the tracking ID it returns, and call PutEvents against the <code>personalize-events</code> client.</p>



<pre class="wp-block-code"><code>import boto3
import time

personalize_events = boto3.client('personalize-events')

personalize_events.put_events(
    trackingId='YOUR_TRACKING_ID',
    userId='user-8842',
    sessionId='sess-19f3ac',
    eventList=[{
        'eventId': 'evt-1',
        'eventType': 'View',
        'itemId': 'SKU-4471',
        'sentAt': int(time.time())
    }]
)</code></pre>



<p class="wp-block-paragraph">Do not put that call inline in the request path of a product page. Recommendation telemetry is not worth a network round trip on your time to first byte, and it is certainly not worth a page error when the API has a bad minute. Push events onto Kinesis Data Streams or Amazon Data Firehose, drain with Lambda, call PutEvents from there.</p>



<p class="wp-block-paragraph">That indirection buys a second thing you want anyway: Firehose can land a durable copy of the same events in S3, which becomes the source for bulk imports and for analysis outside Personalize. Events that only ever existed inside a managed service are events you cannot re-examine when the model does something odd.</p>



<p class="wp-block-paragraph">The <code>sessionId</code> field matters more than it looks. It lets you record anonymous browsing before login and have Personalize associate that session with the user afterward. Skip it and you train on your smallest, least representative slice of traffic.</p>



<h2 class="wp-block-heading">Decision three: schema and event types, decided once</h2>



<p class="wp-block-paragraph">Personalize schemas are Avro. An interactions schema needs a user ID, an item ID and a timestamp. An event type field is optional, but the domain use cases key off it, so you want it.</p>



<pre class="wp-block-code"><code>{
  "type": "record",
  "name": "Interactions",
  "namespace": "com.amazonaws.personalize.schema",
  "fields": [
    { "name": "USER_ID",    "type": "string" },
    { "name": "ITEM_ID",    "type": "string" },
    { "name": "EVENT_TYPE", "type": "string" },
    { "name": "TIMESTAMP",  "type": "long" }
  ],
  "version": "1.0"
}</code></pre>



<p class="wp-block-paragraph">The budget for extra fields is tighter than people expect. An interactions dataset allows only a handful of metadata columns beyond the reserved fields, and the count of distinct event types plus those metadata columns is itself capped. That is a hard design constraint. You cannot ship twenty-eight granular event types and also carry rich per-event context.</p>



<p class="wp-block-paragraph">Spend it deliberately. View, AddToCart and Purchase carry most retail pipelines. Wishlist adds earn a slot if customers use them. Hover, scroll depth and image zoom do not, because they inflate volume without changing what the model learns about intent. Once you have several event types you can weight them differently at training time, which beats the alternative people reach for first, which is dropping views entirely.</p>



<p class="wp-block-paragraph">Before any of that, settle the item ID question. Variants and parent products are different objects in Shopify and WooCommerce alike, and a model trained on variant IDs will happily suggest the blue medium of a shirt the customer already owns in blue medium. Pick a level, write it down, use the same one in your bulk extract and your live payload. Getting those out of step is the most common reason a technically healthy pipeline produces recommendations that feel broken.</p>



<p class="wp-block-paragraph">On volume: the documented floor is a thousand item interactions from at least twenty-five unique users with two or more interactions each, plus a minimum share of your user base clearing that two-interaction bar. AWS recommends considerably more for quality. Near the minimum, the honest read is that you are not building a recommendation pipeline yet. You are building the event capture that lets you build one later, and a popularity-ranked rail will serve customers better meanwhile.</p>



<p class="wp-block-paragraph">Timestamps are Unix time. Mixed timezones, or milliseconds where seconds are expected, will not throw. They will scramble the ordering the model learns from, which is the entire signal.</p>



<h2 class="wp-block-heading">Decision four: freshness without a retraining bill</h2>



<p class="wp-block-paragraph">Two mechanisms get conflated constantly, and the difference sets your monthly cost.</p>



<p class="wp-block-paragraph"><strong>Automatic training</strong> creates a new solution version. The model learns from your latest data. It costs a training run each time.</p>



<p class="wp-block-paragraph"><strong>Automatic updates</strong> are something else entirely. For User-Personalization-v2, User-Personalization and Next-Best-Action, and for the Top picks for you and Recommended for you domain use cases, Personalize refreshes the deployed model every two hours so new items can surface through exploration. That is not retraining. The model does not learn from your recent interactions. It just stops new products from being structurally invisible.</p>



<p class="wp-block-paragraph">So automatic updates solve the cold item problem and do nothing about model drift. AWS&#8217;s own guidance is that even with them, training cadence should stay at least weekly.</p>



<p class="wp-block-paragraph">Trending-Now is the interesting exception. It evaluates interactions over a configurable interval and identifies what is climbing, without waiting for a training run. For a store with promotional spikes or seasonal churn, a Trending-Now rail beside a personalized rail covers a lot of ground for very little retraining.</p>



<p class="wp-block-paragraph">The trap is Similar-Items, which has no automatic updates. Add products daily, run a related-products rail on Similar-Items, and new products only appear after the next training. Same-day visibility means daily training. That is a real budget line, and it is the point where catalog velocity stops being a merchandising detail and becomes an infrastructure decision.</p>



<h2 class="wp-block-heading">Decision five: how the recommendation reaches the page</h2>



<p class="wp-block-paragraph">Real-time means a campaign or a recommender answering GetRecommendations. Batch means an inference job reading input from S3 and writing recommendations back to S3, with no campaign involved and therefore no throughput floor.</p>



<p class="wp-block-paragraph">Most small and mid-sized stores reach for real-time by reflex and do not need it. Ask what the module actually is. A homepage rail refreshed daily, an email send, a merchandising export: those are batch problems. Run the batch overnight, load results into DynamoDB or a Redis instance, serve the carousel from a key lookup. Single-digit millisecond reads, no per-request inference charge, and nothing sitting idle at three in the morning.</p>



<p class="wp-block-paragraph">Real-time earns its cost where the recommendation has to react inside the session, which in practice means product detail and cart pages.</p>



<p class="wp-block-paragraph">Two things belong in the serving layer from day one either way. Filters, so you can exclude out-of-stock items and things the customer already bought; note that Personalize considers a bounded number of interactions per user per event type when filtering, and a user with no imported interactions for several months stops counting toward filtering at all. And a timeout with a fallback, because a carousel must never be able to delay a product page. Fall back to a cached bestseller list and log the fallback rate as a metric, so a rise shows up in CloudWatch or Grafana rather than in a conversion report a fortnight later.</p>



<h2 class="wp-block-heading">Troubleshooting an AWS ecommerce recommendation pipeline</h2>



<ul class="wp-block-list">
<li><strong>Everyone gets the same recommendations.</strong> Usually too few interactions per user, or user IDs that are not stable across sessions. Check that anonymous activity is recorded with a session ID and reconciled at login.</li>



<li><strong>New products never show up.</strong> Check that your recipe supports automatic updates. If it does not, this is a training cadence problem, not a data problem.</li>



<li><strong>Training fails on data volume.</strong> Compare against the documented minimums. The share of users with two or more interactions catches people out: plenty of users is not the same as plenty of returning users.</li>



<li><strong>The dataset shrank and nothing errored.</strong> A bulk import running in FULL mode against a rolling extract. Check the import mode on every scheduled job.</li>



<li><strong>Recommendations ignore what the user just clicked.</strong> Confirm events reach the event tracker, and that the user ID sent to PutEvents matches the one sent to GetRecommendations exactly.</li>



<li><strong>The bill rose during a quiet week.</strong> Automatic training runs on a calendar, not on traffic. Check cadence, and check for campaigns nobody is calling.</li>



<li><strong>Filters are not excluding purchased items.</strong> Filtering looks at a bounded slice of each user&#8217;s history per event type. Heavy users push older purchases outside it.</li>
</ul>



<p class="wp-block-paragraph">Auditing an account you did not build starts with listing what exists, not with reading the application code.</p>



<pre class="wp-block-code"><code>aws personalize list-dataset-groups
aws personalize list-solutions --dataset-group-arn ARN
aws personalize list-campaigns --solution-arn ARN
aws personalize list-recommenders --dataset-group-arn ARN

# Then see what each campaign is holding open
aws personalize describe-campaign --campaign-arn ARN</code></pre>



<p class="wp-block-paragraph">Orphaned campaigns from an abandoned proof of concept show up in that output in about ninety seconds.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Leaving automatic training on for a solution that exists only to satisfy curiosity.</li>



<li>Raising the provisioned throughput floor for a traffic spike, then never lowering it afterward.</li>



<li>Modelling on variant IDs in the bulk extract and product IDs in the live events, or the reverse.</li>



<li>Recording events only for authenticated users, starving the model of most of your traffic.</li>



<li>Designing a rich event taxonomy that collides with the cap on event types plus metadata columns.</li>



<li>Calling PutEvents synchronously on the render path.</li>



<li>Following an implementation guide that still references the retired HRNN recipes.</li>



<li>Shipping with no fallback, so the first slow inference call becomes a slow product page.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Keep a raw copy of every interaction event in S3, independent of Personalize. Managed services are easier to reason about when you can replay what you sent them.</li>



<li>Start with a domain dataset group. Move to custom recipes when you can name the specific thing the domain route will not do.</li>



<li>Measure against a popularity baseline before celebrating anything. If a personalized rail cannot beat bestsellers, the pipeline is not earning its cost yet.</li>



<li>Set training cadence from catalog velocity, not from habit. A stable catalog of a few hundred products does not need what a weekly fashion drop needs.</li>



<li>Prefer batch inference plus a cache for anything that does not have to react within the session.</li>



<li>Alarm on idle inference resources. Utilization near zero for days is a cost bug, and it is the only bug here that never pages anyone.</li>



<li>Keep the staging storefront on cheap infrastructure. The application side of this runs fine on a modest VPS from a provider like Contabo or InterServer while you are still shaping the event schema, and it keeps experiments off the production account.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How much data do I need before Personalize is worth it?</h3>



<p class="wp-block-paragraph">The documented training floor is a thousand item interactions from at least twenty-five unique users with two or more interactions each, and AWS recommends substantially more for quality recommendations. Below that, popularity ranking and hand-curated rails serve customers better and cost nothing to run.</p>



<h3 class="wp-block-heading">Can I build this without Amazon Personalize?</h3>



<p class="wp-block-paragraph">Yes, and for some shops you should. Item-to-item collaborative filtering over purchase history is a modest amount of code and runs fine as a scheduled job writing to a cache. You give up cold-start handling, exploration and session-level personalization. You gain a near-zero cost and full visibility into why an item was recommended.</p>



<h3 class="wp-block-heading">What does an idle Personalize setup cost?</h3>



<p class="wp-block-paragraph">More than zero, which is the surprising part. An active campaign bills its provisioned throughput floor whether or not anyone calls it, and a solution with automatic training on bills a training run on its cadence regardless of traffic. Rates change, so apply current pricing yourself, but the shape is a floor plus a recurring training charge.</p>



<h3 class="wp-block-heading">Real-time or batch recommendations?</h3>



<p class="wp-block-paragraph">Batch unless the module has to respond to the current session. Homepage rails, email and merchandising exports are batch problems. Product detail and cart pages are where real-time inference genuinely changes what the customer sees.</p>



<h3 class="wp-block-heading">Do new products appear automatically?</h3>



<p class="wp-block-paragraph">Only with recipes that support automatic updates, or with Trending-Now. Automatic updates refresh the deployed model every couple of hours so new items can be explored, but they do not retrain it. With a recipe like Similar-Items, a new product waits for the next training run.</p>



<h3 class="wp-block-heading">How do I connect a Shopify or WooCommerce store?</h3>



<p class="wp-block-paragraph">There is no first-party connector. Emit events from the storefront to your own endpoint, push them onto a stream, and call PutEvents from a consumer. For history, export orders and product views to S3 and load them with a dataset import job. Ordinary plumbing; the hard part is settling the item ID question first.</p>



<h3 class="wp-block-heading">How often should I retrain?</h3>



<p class="wp-block-paragraph">At least weekly is the documented baseline, even for recipes with automatic updates. Go faster only when your catalog turns over faster, and price the extra training runs before committing to a daily cadence.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">An AWS ecommerce recommendation pipeline runs two clocks, and they are easy to mistake for one. The freshness clock decides whether a customer can see the product you added this morning. The billing clock runs on training cadence and provisioned throughput, and it keeps running when nobody is looking at the site at all.</p>



<p class="wp-block-paragraph">Get the event schema right, keep a raw copy of everything in S3, stay on batch until the session genuinely demands otherwise, and set training cadence from how fast your catalog actually moves. Do that and the rest is straightforward. Skip it and you end up with something that works, costs more than the revenue it lifts, and nobody can say why.</p>



<h2 class="wp-block-heading">Need a second pair of eyes on your recommendation pipeline?</h2>



<p class="wp-block-paragraph">This is the kind of work I do for ecommerce teams building on AWS:</p>



<ul class="wp-block-list">
<li>Auditing an existing Personalize setup for orphaned campaigns, over-provisioned throughput floors and training cadences that outrun the catalog.</li>



<li>Designing the event schema and item ID strategy before you commit, including the variant-versus-product decision for Shopify and WooCommerce catalogs.</li>



<li>Building the ingestion path: storefront events through Kinesis or Firehose into Lambda, with a durable S3 copy for bulk imports and replay.</li>



<li>Choosing between domain recommenders and custom recipes for your catalog size and event mix, and modelling what each costs to run.</li>



<li>Moving eligible modules from real-time campaigns to batch inference plus a cache layer, then measuring what that takes off the bill.</li>



<li>Setting up monitoring so idle inference capacity and rising fallback rates surface as alerts rather than as a surprise invoice.</li>
</ul>



<p class="wp-block-paragraph">If something is already running, send me your dataset import job configuration and a list of your active campaigns, and I will tell you what I would change first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/aws-ecommerce-recommendation-pipeline/">Building an Ecommerce Recommendation Pipeline on AWS: The Parts That Bite</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>One Customer, Four Systems: Unifying Shopify, Stripe, CRM and Support Data</title>
		<link>https://john-nessime.com/blog/technical-guides/unify-shopify-stripe-crm-support-data/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 29 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Customer 360]]></category>
		<category><![CDATA[Customer Support Analytics]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Data Warehouse]]></category>
		<category><![CDATA[dbt]]></category>
		<category><![CDATA[Ecommerce Analytics]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Event-Driven Architecture]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Identity Resolution]]></category>
		<category><![CDATA[PII Redaction]]></category>
		<category><![CDATA[Reverse ETL]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[Stripe]]></category>
		<category><![CDATA[Webhooks]]></category>
		<category><![CDATA[Zendesk]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=350</guid>

					<description><![CDATA[<p>Three CRM records, one person, and nothing in the logs to explain it. A practical walkthrough of what actually breaks when you unify Shopify, Stripe, CRM and support data: guest-checkout identity, duplicate and out-of-order webhooks, revenue that never ties to payouts, deletion requests that miss half your copies, and reverse ETL write loops. Organized by failure family, with schema and handler patterns you can apply directly.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/unify-shopify-stripe-crm-support-data/">One Customer, Four Systems: Unifying Shopify, Stripe, CRM and Support Data</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The project usually starts with a screenshot. Someone in support pastes three CRM records into a channel, all carrying the same person&#8217;s name, and asks which one is real. One has the order history. One has the subscription. The third has four tickets attached and no orders at all.</p>



<p class="wp-block-paragraph">Nobody broke anything. Shopify, Stripe, the CRM and the helpdesk each created a customer record using whatever identifier they had at the moment they needed one, and none of those identifiers agreed. The systems are all correct in isolation. The person is the thing that got lost.</p>



<p class="wp-block-paragraph">This post covers what actually breaks when you unify Shopify, Stripe, CRM and support data, and how to build the join so it survives guest checkouts, duplicate webhooks, refunds, currency conversion and deletion requests. It&#8217;s organized by failure family rather than by tool, because the tools get acquired and renamed every couple of years and the failures never change.</p>



<h2 class="wp-block-heading">Failure family one: the join key is not the email address</h2>



<p class="wp-block-paragraph">Every one of these systems will happily give you an email address, which is exactly why so many pipelines join on it and quietly produce garbage.</p>



<p class="wp-block-paragraph">Start with where identity actually lives in each system:</p>



<ul class="wp-block-list">
<li><strong>Shopify</strong> has a customer ID, but an order does not have to be attached to one. Guest checkout produces an order with an email on it and no durable customer record behind it.</li>

<li><strong>Stripe</strong> has a customer ID, but only if a Customer object was created. Checkout Sessions that don&#8217;t create one are associated with guest customers instead, which Stripe groups in the Dashboard based on the same card, email or phone. That grouping is read-only and it is not an ID you can join on.</li>

<li><strong>The CRM</strong> keys on email in almost every default configuration, and cheerfully creates a second contact when the email differs by one character.</li>

<li><strong>The helpdesk</strong> keys on the requester&#8217;s email, which is whatever the person typed into a contact form or sent the ticket from. It is very often not the address on the order.</li>
</ul>



<p class="wp-block-paragraph">Here&#8217;s the part that stays invisible until someone finally checks: nothing errors. Every dashboard still returns a number. Lifetime value gets silently split across three profiles, so your LTV looks lower and your repeat-purchase rate looks worse than it is. Support metrics look <em>better</em> than they are, because a customer coming back with the same unresolved problem shows up as a brand new contact. You can run like that for a year without a single failed job.</p>



<h3 class="wp-block-heading">Build an identifier table, not a fuzzy match</h3>



<p class="wp-block-paragraph">The fix is boring and it works: a surrogate person key, and a separate table of every identifier ever observed for that person, with the source that reported it.</p>



<pre class="wp-block-code"><code>-- One row per real human, generated by you, owned by you.
CREATE TABLE person (
  person_key      uuid PRIMARY KEY,
  created_at      timestamptz NOT NULL DEFAULT now(),
  merged_into     uuid NULL REFERENCES person(person_key)
);

-- One row per identifier per source. This is the whole trick.
CREATE TABLE person_identifier (
  person_key      uuid NOT NULL REFERENCES person(person_key),
  source          text NOT NULL,   -- shopify | stripe | crm | helpdesk
  id_type         text NOT NULL,   -- email | phone | shopify_customer_id | stripe_customer_id
  id_value        text NOT NULL,   -- already normalized
  first_seen_at   timestamptz NOT NULL,
  last_seen_at    timestamptz NOT NULL,
  PRIMARY KEY (source, id_type, id_value)
);</code></pre>



<p class="wp-block-paragraph">Normalization happens before the row is written, not in the join. Lowercase and trim emails, store phone numbers in E.164, strip whitespace from external IDs. Resist the urge to get clever with Gmail&#8217;s dot and plus handling. Stripping <code>+tags</code> is a business policy decision with real consequences, not a technical detail, and the same rule applied to a domain that treats plus addresses as distinct mailboxes will merge two genuinely different people.</p>



<p class="wp-block-paragraph">Deterministic matching first, always. Exact matches on shared identifiers, transitively closed. Probabilistic matching on name plus address plus device is worth reaching for only when you have exhausted the deterministic keys, and it has an asymmetric failure mode: a household sharing one card gets collapsed into one person, and unmerging afterwards is expensive because every downstream system has already copied the merged key into its own records. Merges are cheap to make and painful to undo. Bias toward leaving two records separate.</p>



<p class="wp-block-paragraph">One thing worth doing early on the Stripe side: when you create a Customer, write your own identifier into <code>metadata</code>. Stripe allows fifty key-value pairs per object, with keys up to forty characters and values up to five hundred, and it only returns metadata for requests made with a secret key. That&#8217;s enough room for a person key and an order reference, which turns a future fuzzy join into an exact one. Never put anything sensitive there.</p>



<h2 class="wp-block-heading">Failure family two: webhooks arrive twice, out of order, or not at all</h2>



<p class="wp-block-paragraph">Both platforms are explicit about this, and both get treated as if they weren&#8217;t.</p>



<p class="wp-block-paragraph">Stripe delivers at least once, does not guarantee ordering, and retries failed deliveries with exponential backoff for up to three days. The event <code>id</code> (the one starting <code>evt_</code>) stays identical across every retry, which makes it your deduplication key. Shopify sends <code>X-Shopify-Webhook-Id</code> for the same purpose, signs the raw body into <code>X-Shopify-Hmac-Sha256</code>, and states plainly that ordering isn&#8217;t guaranteed within a topic or across topics for the same resource. Shopify&#8217;s own documentation goes further and tells you not to rely on webhooks as your only source of data.</p>



<p class="wp-block-paragraph">For a customer-data pipeline specifically, out-of-order delivery is nastier than duplication. Consider a <code>customers/update</code> that changes someone&#8217;s email arriving before the <code>orders/create</code> that was placed under the old address. Process those in arrival order and you attach the order to a person who doesn&#8217;t exist yet, or worse, to the wrong one. Order your writes by the event&#8217;s own timestamp, using <code>X-Shopify-Triggered-At</code> or the <code>updated_at</code> in the payload, and make profile updates last-writer-wins on that timestamp rather than on insertion time.</p>



<h3 class="wp-block-heading">Do almost nothing in the handler</h3>



<p class="wp-block-paragraph">Verify the signature, claim the event ID, write the raw body to durable storage, return 2xx. That&#8217;s it. Identity resolution, CRM lookups and enrichment calls all belong in a worker reading from that store. A handler that tries to resolve identity inline will eventually be slow enough to time out, which triggers a retry, which arrives while the first attempt is still running.</p>



<p class="wp-block-paragraph">Make the claim atomic rather than check-then-act, because two retries genuinely can land concurrently:</p>



<pre class="wp-block-code"><code>-- Atomic claim. If this inserts zero rows, someone already has it.
INSERT INTO webhook_event (source, event_id, topic, received_at, payload)
VALUES ($1, $2, $3, now(), $4)
ON CONFLICT (source, event_id) DO NOTHING
RETURNING event_id;</code></pre>



<p class="wp-block-paragraph">Return a 2xx for a duplicate. A duplicate is a success from the sender&#8217;s point of view, and returning an error just schedules more of them.</p>



<p class="wp-block-paragraph">Signature verification has one recurring trap on the Shopify side: it runs against the <em>raw</em> body. Any JSON body parser that runs first will change the bytes and every signature will fail. Capture the raw body before parsing, and compare in constant time.</p>



<pre class="wp-block-code"><code>import base64, hashlib, hmac

def verify_shopify(raw_body: bytes, header_value: str, client_secret: str) -&gt; bool:
    digest = hmac.new(
        client_secret.encode("utf-8"),
        raw_body,
        hashlib.sha256,
    ).digest()
    expected = base64.b64encode(digest).decode("utf-8")
    return hmac.compare_digest(expected, header_value)</code></pre>



<p class="wp-block-paragraph">Worth knowing before you spend an afternoon on it: after rotating an app&#8217;s client secret, Shopify notes it can take up to an hour before signatures are generated with the new one. If verification starts failing right after a rotation, that&#8217;s usually why.</p>



<p class="wp-block-paragraph">And because delivery isn&#8217;t guaranteed, webhooks alone are never enough. Run a reconciliation job that pulls orders and customers modified since the last watermark and fills the gaps. This is the job that saves you during an outage on your side, and it&#8217;s the one people skip. If you&#8217;d rather not build the buffering and replay layer, managed webhook infrastructure like Hookdeck exists for exactly this, and self-hosting a small queue and worker on a modest VPS from somewhere like Contabo or InterServer is perfectly reasonable too. The point is that something has to catch what the webhook missed.</p>



<h2 class="wp-block-heading">Failure family three: &#8220;revenue&#8221; means four different things</h2>



<p class="wp-block-paragraph">This is the one that turns into a meeting. Finance says the number is wrong. Analytics says the number is right. Both are correct, because they&#8217;re measuring different things and nobody wrote down which.</p>



<ul class="wp-block-list">
<li>Shopify reports gross sales at the moment the order is placed, in the store&#8217;s currency and the store&#8217;s timezone.</li>

<li>Stripe pays out in batches, net of processing fees, refunds and disputes, on its own schedule, in the settlement currency of the account.</li>

<li>A refund issued in Shopify and a refund issued in Stripe are two different events that may or may not be the same underlying money movement.</li>

<li>A dispute lands days or weeks after the order and reverses revenue in a period that has already been reported.</li>
</ul>



<p class="wp-block-paragraph">There&#8217;s a structural trap here too. If the store runs Shopify Payments, there is no Stripe account for you to query. Shopify Payments is built on Stripe&#8217;s processing infrastructure, but it&#8217;s Shopify&#8217;s gateway, and payouts, fees and disputes live in the Shopify admin rather than behind a Stripe API key. Plenty of businesses end up with Shopify Payments handling storefront orders and a separate Stripe account handling subscriptions or invoicing, which means two payment systems, two payout schedules and one customer who has no idea any of this exists.</p>



<p class="wp-block-paragraph">Do not build a single <code>revenue</code> column. Build separate fact tables at their natural grain (orders, order line items, payments, refunds, disputes, payouts) plus a bridging table that links a payout back to the transactions inside it. Then let each team define its own metric on top. Finance gets net settled cash. Marketing gets gross booked revenue. Neither has to be wrong for the other to be right.</p>



<p class="wp-block-paragraph">Store every timestamp in UTC and keep the original timezone as a separate column. Store every amount in minor units as an integer alongside its currency code, and store the FX rate that was actually applied rather than recalculating it later from a rates table. Recalculated FX is the single most common source of &#8220;the number moved and nobody changed anything.&#8221;</p>



<p class="wp-block-paragraph">On ingestion, Stripe Data Pipeline can push Stripe data directly into Snowflake, Redshift or Databricks as a data share, which removes a connector from the picture if you&#8217;re already on one of those. Otherwise a managed ELT tool like Fivetran or a self-hosted Airbyte will handle Shopify, Stripe, the CRM and the helpdesk with off-the-shelf connectors. Both routes are fine. The connector is genuinely the easy part.</p>



<h2 class="wp-block-heading">Failure family four: deletion is the requirement nobody models</h2>



<p class="wp-block-paragraph">If you&#8217;re building this as a Shopify app, three compliance webhooks are mandatory: <code>customers/data_request</code>, <code>customers/redact</code> and <code>shop/redact</code>. You acknowledge with a 200 and complete the action within thirty days. <code>shop/redact</code> fires forty-eight hours after uninstall and covers the shop&#8217;s data. These are enforced at app review, so you&#8217;ll find out about them either way.</p>



<p class="wp-block-paragraph">The interesting problem isn&#8217;t receiving the webhook. It&#8217;s that by the time it arrives, that person&#8217;s email exists in your raw landing zone, your staging tables, your identity table, your customer view, a BI extract, the CRM, the helpdesk, and a Slack message from a debugging session six months ago. A delete that only touches the golden record is not a delete.</p>



<p class="wp-block-paragraph">The design decision that makes this tractable is to hold personal data in exactly one place. Every downstream table carries <code>person_key</code> and nothing else. Facts, aggregates, models, extracts and syncs all reference the key. When a redaction request lands, you redact one row set in the identifier table and the rest of the warehouse is already anonymous. Getting this right at schema-design time costs an afternoon; retrofitting it costs a quarter.</p>



<p class="wp-block-paragraph">Practical pieces that make it real:</p>



<ul class="wp-block-list">
<li>Tag every table and column that can contain an identifier, in your catalog or in dbt metadata, and treat the tag list as the deletion scope.</li>

<li>Partition the raw landing zone so you can rewrite a partition rather than scan an entire bucket looking for one email.</li>

<li>Keep a deletion queue with a status per downstream system, including the CRM and the helpdesk, which have their own APIs and their own idea of what deletion means.</li>

<li>Write down what you retain and why. Financial records generally have to survive a deletion request for tax and dispute reasons. Say so rather than promising a hard delete you can&#8217;t perform.</li>
</ul>



<h2 class="wp-block-heading">Failure family five: pushing the unified view back out</h2>



<p class="wp-block-paragraph">A customer view nobody can act on is a report. The value shows up when a support agent opens a ticket and can see subscription status and lifetime order count without switching tabs.</p>



<p class="wp-block-paragraph">That&#8217;s reverse ETL, and the market is well served: Hightouch, Census (now part of Fivetran), Polytomic, RudderStack, and iPaaS-style tools like Workato if you need branching logic rather than field syncs. The warehouse-native argument is straightforward and mostly correct: your models already live there, and you avoid maintaining a second copy of the customer profile inside a vendor&#8217;s rigid schema. The counter-argument is also real. Reverse ETL is batch by nature, so if you need sub-minute reaction to an event, a packaged CDP or a direct event pipeline will serve you better, and you&#8217;ll be running several vendors plus warehouse compute plus the people to maintain the models.</p>



<p class="wp-block-paragraph">Three things go wrong here more or less every time.</p>



<p class="wp-block-paragraph"><strong>The write loop.</strong> You compute a churn-risk field and sync it into the CRM. Your ELT connector reads the CRM back into the warehouse. Now the computed field is an input to the model that computes it. This is genuinely hard to spot because nothing errors, the value just drifts. Namespace every synced field with a consistent prefix and explicitly exclude that namespace from ingestion.</p>



<p class="wp-block-paragraph"><strong>The resync stampede.</strong> Incremental syncs push diffs and stay small. Change a model definition and the next run wants to update every record, which will meet a helpdesk API rate limit at speed. Cap the batch size, and treat a full resync as a planned operation rather than something a merge to main can trigger.</p>



<p class="wp-block-paragraph"><strong>Syncing logic instead of keys.</strong> Push <code>person_key</code> into a custom field on the CRM contact and the helpdesk user. It makes the key searchable by agents, it gives the next pipeline a deterministic join, and it stops three teams from each rebuilding their own version of your matching rules.</p>



<h2 class="wp-block-heading">A pipeline shape that holds up</h2>



<p class="wp-block-paragraph">Put together, the order of operations matters more than the tool choice:</p>



<ol class="wp-block-list">
<li>Land raw. Every webhook body and every API page goes to immutable storage first, keyed by source and event ID, before anything interprets it.</li>

<li>Deduplicate on the provider&#8217;s event ID, atomically, with a retention window longer than the provider&#8217;s retry window.</li>

<li>Normalize identifiers into the identifier table as rows arrive, with the source recorded.</li>

<li>Resolve identity deterministically into <code>person_key</code>, as a scheduled job, never inside a request handler.</li>

<li>Build fact tables per system at their own grain, keyed on <code>person_key</code>, with no raw PII.</li>

<li>Model the customer view on top, in dbt or whatever your team already uses. The Fivetran <code>dbt_customer360</code> package is a reasonable starting point if you want to see how someone else structured the identity step.</li>

<li>Activate a small, named, namespaced set of fields back into the CRM and helpdesk.</li>

<li>Run reconciliation and freshness checks on a schedule, and alert on them like you would any other production job.</li>
</ol>



<h2 class="wp-block-heading">Troubleshooting: symptom to root cause</h2>



<p class="wp-block-paragraph">These are the ones that come up repeatedly, and the check that usually settles them.</p>



<ul class="wp-block-list">
<li><strong>Warehouse customer count is much higher than Shopify&#8217;s.</strong> Guest orders are minting a new person on every purchase. Check how many <code>person</code> rows have exactly one identifier and one order.</li>

<li><strong>Warehouse count is much lower.</strong> An over-eager merge rule. Look for person keys with an implausible number of distinct emails or shipping addresses attached.</li>

<li><strong>BI revenue doesn&#8217;t match Stripe payouts.</strong> Grain mismatch, not a data loss. Reconcile a single payout against its constituent transactions before touching anything else.</li>

<li><strong>A CRM field keeps flipping between two values.</strong> Write loop. Check whether the field&#8217;s namespace is excluded from your ELT source configuration.</li>

<li><strong>Recent orders missing, older ones fine.</strong> Webhook delivery gap. Confirm the reconciliation job ran, and check its watermark rather than trusting its exit code.</li>

<li><strong>Every Shopify signature suddenly fails.</strong> Either a body parser is running before verification, or the client secret was rotated within the last hour.</li>

<li><strong>Tickets attached to the wrong customer after a merge.</strong> The merge happened in the warehouse but the synced key in the helpdesk still points at the old person. Merges need to propagate outward, not just inward.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Joining on email in the model layer instead of resolving identity once, upstream.</li>

<li>Treating webhooks as the system of record instead of as a low-latency hint, with a reconciliation pull behind them.</li>

<li>Doing identity resolution, enrichment or CRM writes inside the webhook handler.</li>

<li>Ordering profile updates by arrival time rather than by the event&#8217;s own timestamp.</li>

<li>One <code>revenue</code> column that finance and marketing are both told to use.</li>

<li>Storing amounts as floats, or recalculating historical FX from a rates table.</li>

<li>Copying PII into every downstream table because it was convenient at the time.</li>

<li>Reaching for probabilistic matching before the deterministic keys have been exhausted.</li>

<li>Assuming a Shopify store on Shopify Payments has a Stripe account you can query.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>Own the surrogate key. Never let a vendor&#8217;s internal ID become the primary identity of a person in your systems.</li>

<li>Write your <code>person_key</code> into Stripe <code>metadata</code> and Shopify customer metafields at creation time, so future joins are exact rather than inferred.</li>

<li>Keep raw payloads immutable and replayable. Every model bug you&#8217;ll ever have is cheaper to fix if you can reprocess from raw.</li>

<li>Version your matching rules and record which version produced each merge, so you can audit and reverse a bad rule.</li>

<li>Alert on freshness and on row-count deltas per source, not just on job success. A connector returning zero rows exits cleanly.</li>

<li>Reconcile one payout end to end by hand before you trust the automated version. It&#8217;s the fastest way to find a grain error.</li>

<li>Document the metric definitions next to the models. Most &#8220;the data is wrong&#8221; tickets are definition disputes.</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Do I need a CDP to unify Shopify, Stripe, CRM and support data?</h3>



<p class="wp-block-paragraph">Not necessarily. If you already run a warehouse and have someone comfortable with dbt, ELT plus modelling plus reverse ETL covers the same ground with more control over the data model. A packaged CDP earns its cost when you need real-time behavioural segmentation, when nobody on the team can maintain models, or when marketing needs to ship changes without waiting on data engineering.</p>



<h3 class="wp-block-heading">How do I handle guest checkout customers?</h3>



<p class="wp-block-paragraph">Create a person from the order&#8217;s email anyway, and mark the record as identity-weak. When that email later appears attached to a real Shopify customer or Stripe Customer object, the deterministic match links them and the order history follows. The mistake is discarding guest orders because they have no customer ID, which is exactly the revenue you most want attributed.</p>



<h3 class="wp-block-heading">Should I sync everything into the CRM or keep it in the warehouse?</h3>



<p class="wp-block-paragraph">Keep the full picture in the warehouse and sync a deliberately small set of fields outward. The test is whether an agent or a rep will act on the field during a conversation. Lifetime value, subscription status, open ticket count and churn risk pass. A hundred-column profile does not, and it turns every model change into a rate-limit incident.</p>



<h3 class="wp-block-heading">Why don&#8217;t my Shopify sales match my Stripe payouts?</h3>



<p class="wp-block-paragraph">Because they&#8217;re measuring different things at different times. Shopify shows gross sales at order time; Stripe pays out batched amounts net of fees, refunds and disputes, and the timing differs by days. Add currency conversion and a timezone mismatch and the numbers will never tie without an explicit bridging table linking payouts to the transactions inside them.</p>



<h3 class="wp-block-heading">Can I use email as the primary key if my store requires account creation?</h3>



<p class="wp-block-paragraph">You can get away with it longer, but it&#8217;s still a bad primary key. People change email addresses, use one address for orders and another for support, and share addresses within a household. A surrogate key with an identifier table costs very little up front and means an email change is a new row rather than a broken join and a duplicate customer.</p>



<h3 class="wp-block-heading">How do I test the identity resolution before trusting it?</h3>



<p class="wp-block-paragraph">Run it in shadow mode. Compute person keys without activating them, then have support pick twenty customers they know are duplicates and twenty they know are distinct, and check what the resolver did. That exercise finds normalization bugs and over-eager merge rules faster than any aggregate metric, because the people who answer the tickets already know where the duplicates are.</p>



<h3 class="wp-block-heading">What&#8217;s the minimum viable version of this?</h3>



<p class="wp-block-paragraph">Raw landing with deduplication, an identifier table with deterministic matching, one fact table per source, and a single synced field into the helpdesk. That&#8217;s a couple of weeks of work and it removes most of the pain. Everything above it is refinement.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Conclusion: unify Shopify, Stripe, CRM and support data around a key you own</h2>



<p class="wp-block-paragraph">If you take one thing from this: the hard part of unifying Shopify, Stripe, CRM and support data is not moving the data. Connectors are commodity. The hard part is deciding what a customer <em>is</em>, writing that decision down as a key you own, and then refusing to let any downstream system quietly invent its own version of it.</p>



<p class="wp-block-paragraph">Do the identity work first, land everything raw so you can reprocess when you get it wrong, and keep personal data in one place so deletion stays a single operation. The dashboards will follow. Skip that order and you get exactly the screenshot this post opened with, eighteen months later and with a lot more rows behind it.</p>



<h2 class="wp-block-heading">Need help wiring this together?</h2>



<p class="wp-block-paragraph">I work on exactly this kind of plumbing: the unglamorous layer between an ecommerce platform, a payment processor and the tools your team actually opens all day. Things I can help with here:</p>



<ul class="wp-block-list">
<li>Designing the identity model (surrogate keys, identifier tables, deterministic matching rules) and auditing an existing one for over-merging.</li>

<li>Building idempotent, signature-verified webhook intake for Shopify and Stripe, with a dead-letter path and a reconciliation job behind it.</li>

<li>Setting up ELT into Snowflake, BigQuery, Redshift or Postgres, whether with Fivetran, self-hosted Airbyte, or Stripe Data Pipeline where it fits.</li>

<li>Modelling orders, payments, refunds, disputes and payouts so finance and marketing can each get a number they trust.</li>

<li>Reverse ETL back into the CRM and helpdesk without write loops, rate-limit incidents or a hundred junk fields.</li>

<li>Making deletion and data-subject requests actually work across the warehouse, the raw landing zone and every downstream copy.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;re stuck on a specific piece, send me the concrete thing: a webhook payload, a schema dump, the query that disagrees with your Stripe payout report. It&#8217;s much easier to say something useful about a real artifact than about a diagram.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/unify-shopify-stripe-crm-support-data/">One Customer, Four Systems: Unifying Shopify, Stripe, CRM and Support Data</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Streaming Shopify Events into AWS Without Losing Orders</title>
		<link>https://john-nessime.com/blog/devops/streaming-shopify-events-into-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 16 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon SQS]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Dead Letter Queue]]></category>
		<category><![CDATA[DynamoDB]]></category>
		<category><![CDATA[Ecommerce Analytics]]></category>
		<category><![CDATA[Event-Driven Architecture]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Partner Event Source]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Reliability Engineering]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[Terraform]]></category>
		<category><![CDATA[Webhooks]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=221</guid>

					<description><![CDATA[<p>Wiring Shopify webhooks into Amazon EventBridge takes an afternoon. Keeping every order is the hard part. A walk through the five failure families that actually bite when streaming Shopify events into AWS: the partner source that silently drops everything, duplicate and out-of-order deliveries, rule patterns that match nothing, targets that fail without a dead-letter queue, and the 64 KB metering rule that quietly inflates the bill.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/streaming-shopify-events-into-aws/">Streaming Shopify Events into AWS Without Losing Orders</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The partner event source in the EventBridge console said <code>Pending</code>. It had said <code>Pending</code> for six days.</p>



<p class="wp-block-paragraph">Nobody noticed, because nothing errored. No 5xx in a log. No failed delivery in Shopify&#8217;s dashboard. No alarm. Shopify had been publishing order events the entire time, and AWS had been throwing every single one of them on the floor.</p>



<p class="wp-block-paragraph">That behaviour is documented, in one short note in the AWS docs: events published to a partner event source that has not been associated with an event bus are dropped immediately and are not persisted at rest. There is no retry for that. There is no buffer. The events are gone, and the only way to get the data back is to go ask the Shopify Admin API for it after the fact.</p>



<p class="wp-block-paragraph">That is the shape of most of the pain in this integration. Streaming Shopify events into AWS is easy to stand up and easy to get quietly wrong, and every one of the quiet failures looks identical from the outside: everything is green, and some of your data isn&#8217;t there.</p>



<p class="wp-block-paragraph">This post walks the five failure families that actually cost you records, plus the reconciliation layer that most teams only build after the first incident. It assumes you can read a rule pattern and an IAM policy. It does not assume you have shipped this before.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">What the pipe actually looks like</h2>



<p class="wp-block-paragraph">Four moving parts, and only two of them live in your account.</p>



<ol class="wp-block-list"><li>A Shopify app holds the webhook subscriptions. Each subscription has a topic and a delivery method. For this path the delivery method is EventBridge and the address is an ARN, not a URL.</li><li>Shopify creates a <strong>partner event source</strong> inside your AWS account, in the region you nominated.</li><li>You associate that source with a <strong>partner event bus</strong>. This is the step everyone forgets.</li><li>Rules on that bus match events and push them at targets: Lambda, SQS, Step Functions, Firehose, whatever fits.</li></ol>



<p class="wp-block-paragraph">The ARN trips people up more than anything else in the setup. Shopify wants the <em>event source</em> ARN, not the event bus ARN. They look similar and only one of them works:</p>



<pre class="wp-block-code"><code># Correct - the event source ARN. Note the empty account field.
arn:aws:events:eu-west-1::event-source/aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;

# Wrong - this is the bus, and Shopify will reject it
arn:aws:events:eu-west-1:123456789012:event-bus/aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;</code></pre>



<p class="wp-block-paragraph">Associating the source is a single call, and both the name and the source name are the same string:</p>



<pre class="wp-block-code"><code># Create the partner event bus that accepts the source
aws events create-event-bus 
  --name "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;" 
  --event-source-name "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;" 
  --region eu-west-1

# Confirm it flipped from PENDING to ACTIVE
aws events describe-event-source 
  --name "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;" 
  --region eu-west-1</code></pre>



<p class="wp-block-paragraph">The same architecture applies if you are not on Shopify. BigCommerce and commercetools both publish to EventBridge as partner sources, and the failure families below are identical because they come from EventBridge&#8217;s semantics, not the store&#8217;s.</p>



<h2 class="wp-block-heading">Failure family one: events that never existed</h2>



<p class="wp-block-paragraph">This is the one from the opening, and it is the most expensive because it is completely silent on both sides.</p>



<p class="wp-block-paragraph">Shopify considers the delivery successful. It handed the event to the partner source, which is its contract. AWS considers nothing to have happened, because an unassociated source has no bus to write to, and EventBridge does not persist events at rest before a bus exists. Your CloudWatch metrics show nothing, because metrics are emitted per bus and per rule, and you have neither.</p>



<p class="wp-block-paragraph">The same class of hole opens up in two other ways:</p>



<ul class="wp-block-list"><li><strong>Region mismatch.</strong> The source is created in the region you gave Shopify. Your bus, your rules, your targets and your dead-letter queues all have to be in that region. A rule in the right account but the wrong region matches nothing, forever, without complaint.</li><li><strong>Environment drift.</strong> A staging store pointed at a production source, or a source created against an account ID that belonged to an old sandbox. Nothing errors. Events just land somewhere you are not looking.</li></ul>



<p class="wp-block-paragraph">The fix is boring and it works: treat the source state as a monitored asset. A scheduled job that calls <code>describe-event-source</code> and alarms if <code>State</code> is anything other than <code>ACTIVE</code> costs you twenty minutes and covers the entire failure family. Put it next to your other synthetic checks, not inside the pipeline it is watching.</p>



<p class="wp-block-paragraph">The second half of that check is a heartbeat on volume. If a bus that normally sees a few thousand events a day sees zero for an hour, that is an incident even when every component reports healthy. Alarm on <code>MatchedEvents</code> hitting zero, not just on errors.</p>



<h2 class="wp-block-heading">Failure family two: events that arrive twice, or backwards</h2>



<p class="wp-block-paragraph">EventBridge is at-least-once. Shopify&#8217;s webhooks are at-least-once. Neither one promises ordering. Put those together and you get two distinct bugs that people usually try to fix with one patch.</p>



<p class="wp-block-paragraph">The duplicate is the obvious one. The same <code>orders/create</code> arrives twice, and if your handler posts to a fulfilment provider or sends a customer email, you have just done it twice. The dedupe key is sitting in the envelope: Shopify puts <code>X-Shopify-Webhook-Id</code> into <code>detail.metadata</code>, and it identifies the delivery. Write it into DynamoDB with a conditional put and a TTL of a few days, and drop the event if the write fails.</p>



<p class="wp-block-paragraph">The out-of-order case is the one that costs you money quietly. An <code>orders/updated</code> carrying a cancelled status arrives before the <code>orders/updated</code> carrying the address change, and your database ends up holding the older state because it was written last. Nothing failed. The row is just wrong, and it will stay wrong until someone complains.</p>



<p class="wp-block-paragraph">The envelope carries what you need for this too. <code>detail.metadata</code> includes <code>X-Shopify-Triggered-At</code>, and the resource in <code>detail.payload</code> carries its own <code>updated_at</code>. Compare before you write, and refuse to apply an update whose timestamp is older than the one already stored.</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow"><p>The dedupe key stops you from doing the work twice. The version check stops you from doing the work backwards. They solve different problems and you need both.</p></blockquote>



<p class="wp-block-paragraph">One thing you can skip on this path: HMAC verification. On the HTTPS delivery method you must verify the signature, because anyone can POST to your endpoint. On the EventBridge path, only the partner account behind the event source is permitted to publish to that bus, and the AWS docs are explicit that adding your own resource policy to a partner bus is rejected. The signature header still rides along in the metadata, but the trust boundary is enforced by AWS rather than by your code.</p>



<h2 class="wp-block-heading">Failure family three: rules that match nothing</h2>



<p class="wp-block-paragraph">Every Shopify event, regardless of topic, arrives with the same <code>detail-type</code>. That single fact invalidates the routing instinct most people bring from AWS service events.</p>



<pre class="wp-block-code"><code>{
  "version": "0",
  "id": "1b8e2e75-b771-e964-f0e6-fbca6a21dad8",
  "detail-type": "shopifyWebhook",
  "source": "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;",
  "account": "123456789012",
  "time": "2022-07-02T12:47:58Z",
  "region": "eu-west-1",
  "resources": [],
  "detail": {
    "payload": {
      "id": 1234567890,
      "title": "Columbia Las Hermosas"
    },
    "metadata": {
      "Content-Type": "application/json",
      "X-Shopify-Topic": "products/update",
      "X-Shopify-Shop-Domain": "example.myshopify.com",
      "X-Shopify-Hmac-SHA256": "...",
      "X-Shopify-Webhook-Id": "...",
      "X-Shopify-API-Version": "...",
      "X-Shopify-Triggered-At": "2022-07-02T12:47:57.989779121Z"
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Two things to take from that envelope. The resource body is nested under <code>detail.payload</code>, not at the top of <code>detail</code>, so a pattern copied from an HTTPS handler will match nothing. And the topic lives in <code>detail.metadata</code>, which is where all your routing has to happen.</p>



<pre class="wp-block-code"><code>// Exact topic match
{
  "detail-type": ["shopifyWebhook"],
  "detail": {
    "metadata": {
      "X-Shopify-Topic": ["orders/create"]
    }
  }
}

// Every orders topic, one rule
{
  "detail-type": ["shopifyWebhook"],
  "detail": {
    "metadata": {
      "X-Shopify-Topic": [{ "prefix": "orders/" }]
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Do not deploy a pattern you have not tested against a real envelope. <code>test-event-pattern</code> answers in a second and saves an afternoon:</p>



<pre class="wp-block-code"><code>aws events test-event-pattern 
  --event-pattern file://pattern.json 
  --event file://sample-event.json</code></pre>



<h3 class="wp-block-heading">One rule or thirty?</h3>



<p class="wp-block-paragraph">There is a real argument for a single catch-all rule that pushes everything into one queue and lets your consumer branch on the topic. It is less infrastructure, it deploys faster, and adding a topic does not require a Terraform run.</p>



<p class="wp-block-paragraph">What you give up is per-topic visibility. <code>MatchedEvents</code>, <code>FailedInvocations</code> and the dead-letter queue are all scoped to the rule. Collapse thirty topics into one rule and you can no longer tell that inventory events stopped three days ago, because the aggregate number still looks fine.</p>



<p class="wp-block-paragraph">The split I reach for first: a dedicated rule for each topic that touches money or fulfilment, and one catch-all for everything else. You get precise alarms where the cost of being wrong is high and low overhead everywhere else.</p>



<h2 class="wp-block-heading">Failure family four: events that arrive and die at the target</h2>



<p class="wp-block-paragraph">By default EventBridge keeps retrying a failed target invocation for up to a day, with exponential backoff and jitter. That is generous, and it is also the reason people assume they do not need a dead-letter queue. They do, for two reasons.</p>



<p class="wp-block-paragraph">First, a whole class of errors gets <em>no</em> retries at all. Missing permissions on the target, a target that no longer exists, an address that will not resolve. EventBridge does not retry those, because retrying cannot help. It sends them straight to the DLQ if one is configured, and drops them if one is not.</p>



<p class="wp-block-paragraph">Second, a day of retries is not much when the failure is a bad deploy discovered on a Friday evening.</p>



<pre class="wp-block-code"><code>aws events put-targets 
  --rule shopify-orders-create 
  --event-bus-name "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;" 
  --targets '[{
    "Id": "order-processor",
    "Arn": "arn:aws:lambda:eu-west-1:123456789012:function:order-processor",
    "RetryPolicy": {
      "MaximumRetryAttempts": 20,
      "MaximumEventAgeInSeconds": 3600
    },
    "DeadLetterConfig": {
      "Arn": "arn:aws:sqs:eu-west-1:123456789012:shopify-orders-dlq"
    }
  }]'</code></pre>



<p class="wp-block-paragraph">Lowering the retry window is deliberate here. Twenty-four hours of retries against a genuinely broken consumer buys you nothing and hides the problem; a shorter window pushes failures into the DLQ where they are visible and countable.</p>



<h3 class="wp-block-heading">The DLQ permission trap</h3>



<p class="wp-block-paragraph">This one catches almost everyone who manages infrastructure as code. Configure a DLQ through the console and AWS attaches the queue policy for you. Configure it through <code>PutTargets</code> — which is what Terraform, CloudFormation and the CLI all do — and you must attach it yourself. Miss it, and you have a dead-letter queue that cannot receive dead letters.</p>



<pre class="wp-block-code"><code>{
  "Sid": "Dead-letter queue permissions",
  "Effect": "Allow",
  "Principal": { "Service": "events.amazonaws.com" },
  "Action": "sqs:SendMessage",
  "Resource": "arn:aws:sqs:eu-west-1:123456789012:shopify-orders-dlq",
  "Condition": {
    "ArnEquals": {
      "aws:SourceArn": "arn:aws:events:eu-west-1:123456789012:rule/shopify-orders-create"
    }
  }
}</code></pre>



<p class="wp-block-paragraph">The metric that catches this is <code>InvocationsFailedToBeSentToDlq</code>. If it is ever non-zero, your safety net has a hole in it and events are being lost at the exact moment you were counting on it. Alarm on it at a threshold of one. It only reports when it is non-zero, so it costs nothing the rest of the time.</p>



<p class="wp-block-paragraph">Two more constraints worth knowing before you design around a DLQ: it must be a standard SQS queue, not FIFO, and it must live in the same region as the rule. Each message carries the error code, the exhausted retry condition, the retry count and both ARNs as message attributes, which is usually enough to triage without opening the payload.</p>



<h2 class="wp-block-heading">Failure family five: the bill</h2>



<p class="wp-block-paragraph">EventBridge does not meter one event as one event. It meters in 64 KB chunks, so an event larger than that bills as multiple events. Rates change and vary by region, so check the current pricing page rather than trusting any number you read in a blog post, but the mechanism is stable and it is what determines your bill.</p>



<p class="wp-block-paragraph">This matters more for commerce than for most event sources. A product update is small. An order with thirty line items, per-item discount allocations, tax lines, shipping lines, note attributes and a stack of metafields is not. Wholesale and subscription stores routinely produce order payloads that cross the chunk boundary, and the same order updated eight times through its lifecycle multiplies that.</p>



<p class="wp-block-paragraph">Three levers, roughly in order of how much they return:</p>



<ul class="wp-block-list"><li><strong>Trim at the subscription.</strong> Shopify&#8217;s webhook subscription API lets you restrict which fields are included in the payload and which metafield namespaces come along. Fields you never read cost you at ingestion, at archive and again at replay. This is the only lever that stops paying for the data before it enters AWS.</li><li><strong>Subscribe to fewer topics.</strong> Broad topics like <code>orders/updated</code> fire on changes you do not care about. If you only act on fulfilment state, subscribe to the fulfilment topics instead of filtering a firehose after you have paid for it.</li><li><strong>Archive selectively, and set retention.</strong> Archives bill for processing, for storage and again for replay. An archive with no retention period grows forever. Archive the topics you would genuinely replay and let the rest go.</li></ul>



<p class="wp-block-paragraph">One structural limit to design around: EventBridge caps the total size of a single event. A payload that exceeds it does not get truncated in a helpful way — the publish fails. Trimming at the subscription protects you here as well as on cost.</p>



<h2 class="wp-block-heading">The layer nobody builds until they need it</h2>



<p class="wp-block-paragraph">Archive and replay is genuinely useful, and it is also routinely misunderstood. Replay re-delivers events that <em>reached the bus</em>. It does nothing at all for the failure family at the top of this post, where the events never reached the bus in the first place. Replay fixes bugs in your consumer. It does not fix gaps in your ingestion.</p>



<p class="wp-block-paragraph">For that you need reconciliation: a scheduled job that queries the Shopify Admin API for resources changed since a stored watermark and compares them against what you hold. It is unglamorous, it is the thing that catches the outage you did not know about, and it is worth building before you need it rather than during the incident.</p>



<ul class="wp-block-list"><li>Run it hourly for orders and fulfilments, daily for products and customers. The cadence should track how expensive being wrong is, not how much data there is.</li><li>Store a watermark per topic and advance it only after a successful full pass. A partial pass that advances the watermark creates the exact gap you built the job to find.</li><li>Compare counts first, records second. A count mismatch is cheap to compute and tells you whether to bother with the expensive comparison.</li><li>Emit the drift as a metric, not just a log line. &#8220;Orders in Shopify but not in our store, last hour&#8221; is a graph worth putting on a dashboard, and it should normally read zero.</li></ul>



<p class="wp-block-paragraph">The reconciliation worker does not need to live in Lambda. It is a long, paginated, rate-limited crawl, which is an awkward fit for a function timeout and a comfortable fit for a small VPS you already run. If you have a box at InterServer or Hetzner sitting there for other jobs, a cron entry and a script is a perfectly respectable answer.</p>



<h2 class="wp-block-heading">Troubleshooting by symptom</h2>



<p class="wp-block-paragraph">Work these in order. Each one is cheap and rules out a whole branch.</p>



<h3 class="wp-block-heading">Nothing is arriving at all</h3>



<ol class="wp-block-list"><li>Run <code>describe-event-source</code>. If <code>State</code> is not <code>ACTIVE</code>, stop here. Everything published so far is gone and you need the reconciliation path.</li><li>Confirm the region of the bus matches the region in the source ARN.</li><li>List your webhook subscriptions through the Admin API and confirm the address is the event-source ARN, not the bus ARN.</li><li>Confirm the subscriptions belong to the app whose access token you are using. Registering with a token from a different app is a common and confusing dead end.</li><li>Check <code>MatchedEvents</code> on the bus with no rule dimension. Non-zero means events are landing and your rules are the problem, not the plumbing.</li></ol>



<h3 class="wp-block-heading">Some topics arrive, order or customer topics do not</h3>



<p class="wp-block-paragraph">This is almost always scopes rather than infrastructure. Order and customer topics sit behind protected customer data access, which is a separate approval in the app configuration on top of the read scopes. Without it, product events flow perfectly and order events silently do not — which looks exactly like a broken rule and is not.</p>



<h3 class="wp-block-heading">The rule matches but the target does nothing</h3>



<p class="wp-block-paragraph">Compare <code>MatchedEvents</code> against <code>SuccessfulInvocationAttempts</code> on the rule. A gap sends you to <code>FailedInvocations</code> and to the DLQ. Check the target&#8217;s resource policy, and check <code>InvocationsFailedToBeSentToDlq</code> before you trust that the DLQ is catching anything.</p>



<h3 class="wp-block-heading">Events arrive, but late</h3>



<p class="wp-block-paragraph">Look at <code>ThrottledRules</code> and at <code>IngestionToInvocationSuccessLatency</code>. Sustained throttling usually means an invocation quota rather than a rule problem, and it shows up first during flash sales, which is the worst possible time to discover it. Load-test the path before a peak event, not after.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list"><li>Creating the partner event source and never associating it with a bus. Silent, total, unrecoverable data loss for the whole window.</li><li>Registering the event bus ARN instead of the event source ARN, then debugging Shopify&#8217;s rejection for an hour.</li><li>Writing rule patterns against the resource shape from an HTTPS webhook, forgetting that the body sits under <code>detail.payload</code>.</li><li>Routing on <code>detail-type</code>. Every Shopify event carries the same one, so a pattern that matches on it alone matches everything.</li><li>Configuring a DLQ through Terraform without the queue policy, and only finding out when you needed it.</li><li>Assuming replay covers ingestion gaps. It replays what reached the bus and nothing else.</li><li>Deduplicating on the resource ID instead of the webhook ID, so legitimate subsequent updates get discarded as duplicates.</li><li>Skipping reconciliation because the pipeline &#8220;works&#8221;. It works right up until it doesn&#8217;t, and that is precisely when you need the other path.</li></ul>



<h2 class="wp-block-heading">Best practices for streaming Shopify events into AWS</h2>



<ul class="wp-block-list"><li>Alarm on the event source state and on <code>MatchedEvents</code> reaching zero. Absence of events is a signal, and it is the only signal you get for the worst failure.</li><li>Dedupe on <code>X-Shopify-Webhook-Id</code> and version-check on <code>X-Shopify-Triggered-At</code>. Two mechanisms, two problems.</li><li>Give every target a DLQ and a retry window you chose deliberately, rather than inheriting the default.</li><li>Keep dedicated rules for money and fulfilment topics so their metrics stay legible; batch the rest behind a catch-all.</li><li>Trim payloads at the Shopify subscription rather than in a Lambda. Filtering after ingestion means you already paid for the bytes.</li><li>Define the whole thing in Terraform or CloudFormation, including the queue policies. This stack has too many one-time console clicks to survive being hand-built twice.</li><li>Point your observability platform at the same bus. Datadog and New Relic are both EventBridge partners, so business events and infrastructure telemetry can share one pipeline instead of two.</li><li>Build reconciliation before your first peak trading period, not after your first missing-order ticket.</li></ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I still need to verify the HMAC signature on the EventBridge path?</h3>



<p class="wp-block-paragraph">No. Only the partner account behind the event source can publish to a partner event bus, and AWS actively rejects attempts to add your own resource policy granting anyone else access. The signature header is still present in the metadata, but the trust boundary is enforced by AWS rather than by your handler. On the HTTPS delivery method, verification remains mandatory.</p>



<h3 class="wp-block-heading">Should I use EventBridge or plain HTTPS webhooks?</h3>



<p class="wp-block-paragraph">HTTPS is simpler, works with any host, and is easier to debug because you can curl your own endpoint. It also puts you on the hook for absorbing burst traffic within a short response deadline, and for keeping the endpoint up well enough that Shopify does not remove the subscription after persistent failures. EventBridge moves that burst absorption to AWS and gives you native fan-out. If your consumers already live in AWS, the operational maths favours EventBridge. If they do not, a dedicated reliability layer such as Hookdeck in front of an HTTPS endpoint is a reasonable alternative and a much smaller change.</p>



<h3 class="wp-block-heading">Can I use one partner event source for multiple stores?</h3>



<p class="wp-block-paragraph">Events from every shop that installed your app flow through the source associated with that app, and the shop is identified by <code>X-Shopify-Shop-Domain</code> in the metadata. You can route per-shop with rule patterns matching that field. For genuine tenant isolation — separate accounts, separate blast radius — you want separate apps and separate sources, because a single bus is a single failure domain.</p>



<h3 class="wp-block-heading">Why do I get duplicate order events even though nothing failed?</h3>



<p class="wp-block-paragraph">Because at-least-once means exactly that. Duplicates are normal operation, not a fault to be investigated. Separately, an order genuinely does change several times shortly after creation — payment capture, risk assessment, post-purchase upsells — so several <code>orders/updated</code> events for one order are expected and are not duplicates at all. Deduplicate on the webhook ID to tell the two apart.</p>



<h3 class="wp-block-heading">What happens to events published while my consumer is broken?</h3>



<p class="wp-block-paragraph">They reach the bus, match your rules, and EventBridge retries the target within your configured window. Once that window is exhausted they go to the DLQ if you have one and are discarded if you do not. The events themselves are not lost at the bus level as long as the source is associated — this is the failure family you can actually engineer your way out of.</p>



<h3 class="wp-block-heading">Can I archive and replay Shopify events?</h3>



<p class="wp-block-paragraph">Yes, with an archive on the partner event bus and an event pattern controlling what gets archived. Budget for three separate charges — processing into the archive, storage while it sits there, and the replay itself — and always set an explicit retention period, because an archive without one grows indefinitely.</p>



<h3 class="wp-block-heading">Does this work the same way for BigCommerce or commercetools?</h3>



<p class="wp-block-paragraph">The AWS half is identical: partner source, association step, bus, rules, targets, and every failure family in this post. What differs is the envelope shape and how you register subscriptions on the vendor side. The association gap in particular bites the same way regardless of which platform is publishing.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Almost everything about streaming Shopify events into AWS degrades loudly. Targets throw errors, retries show up as metrics, dead letters pile up in a queue you can see. Those are the failures you will handle correctly, because they announce themselves.</p>



<p class="wp-block-paragraph">The one that will actually hurt you is the one that reports success on both sides while dropping every event on the floor. Association state and event volume are the two signals that catch it, and neither one appears on any dashboard by default. Add them on day one, before you write the first rule. Everything else in this post can be fixed after the fact; that one cannot.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a hand with your event pipeline?</h2>



<p class="wp-block-paragraph">Most of my work on this stack is either standing it up properly the first time or working out where records went after someone else stood it up. Things I can help with:</p>



<ul class="wp-block-list"><li>Building the Shopify-to-EventBridge path end to end in Terraform, including the queue policies and retry configuration that the console quietly does for you.</li><li>Auditing an existing pipeline for silent loss: source association, region drift, missing DLQ permissions, rules that have been matching nothing since the day they shipped.</li><li>Designing the idempotency and ordering layer — dedupe store, TTLs, version checks — so replays and duplicates stop corrupting downstream state.</li><li>Writing the reconciliation job against the Admin API, with watermarks, drift metrics and alarms that fire before a customer does.</li><li>Cutting EventBridge spend by trimming payloads at the subscription and rationalising archive retention, without losing anything you actually query.</li><li>Load-testing the whole path ahead of a peak trading period so throttling shows up in a test window rather than on the day.</li></ul>



<p class="wp-block-paragraph">If you have a rule pattern that isn&#8217;t matching, a DLQ that&#8217;s mysteriously empty, or a bill that grew faster than your order volume, send me the pattern, the metric graph or the line item and I&#8217;ll tell you what I&#8217;d look at first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/streaming-shopify-events-into-aws/">Streaming Shopify Events into AWS Without Losing Orders</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Shopify Sales Dashboard with AWS: Build One That Actually Reconciles</title>
		<link>https://john-nessime.com/blog/technical-guides/shopify-sales-dashboard-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 04 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Business Intelligence]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Ecommerce Analytics]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[Partition Projection]]></category>
		<category><![CDATA[QuickSight]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[SQL]]></category>
		<category><![CDATA[Webhooks]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=142</guid>

					<description><![CDATA[<p>Most Shopify dashboards built on AWS work perfectly for about three weeks, then quietly drift away from the numbers in the Shopify admin. Here is why that happens, and how to design the ingestion, storage and query layers so your totals still reconcile six months in.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/shopify-sales-dashboard-aws/">Shopify Sales Dashboard with AWS: Build One That Actually Reconciles</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone in the finance channel posts two screenshots side by side. On the left, the dashboard you built. On the right, the Shopify admin. The totals don&#8217;t match, and they&#8217;re not off by a rounding error either. They&#8217;re off by enough that nobody wants to use your dashboard for anything that matters.</p>



<p class="wp-block-paragraph">The frustrating part is that the pipeline is fine. Webhooks are arriving, Lambda is running clean, S3 has the files, Athena returns rows. Nothing is broken in the way monitoring understands &#8220;broken&#8221;. The pipeline is faithfully recording something that is no longer true.</p>



<p class="wp-block-paragraph">This post is about building a Shopify sales dashboard with AWS that survives that conversation. Not the wiring, which is well documented and mostly straightforward, but the design decisions that determine whether your numbers still hold up six months in. I&#8217;ll cover the three ingestion paths and when each one is the right call, why append-only pipelines drift, how to lay out S3 and Athena so recomputation is cheap, and what to do when the totals are already wrong.</p>



<h2 class="wp-block-heading">Why a Shopify sales dashboard with AWS drifts from the admin</h2>



<p class="wp-block-paragraph">Here&#8217;s the thing that catches almost everyone: <strong>a Shopify order is not an event, it&#8217;s a mutable record.</strong></p>



<p class="wp-block-paragraph">An event pipeline assumes facts are immutable once written. A payment happened. A shipment left. You append it, you never touch it again, and the sum of the log is the truth. That model is why streaming architectures are so clean, and it&#8217;s exactly wrong for order data.</p>



<p class="wp-block-paragraph">An order created on Monday can be edited on Tuesday, partially refunded on Friday, and fully refunded three weeks later. Every one of those changes belongs, financially, to Monday. If your pipeline appends the <code>orders/create</code> payload and never revisits it, Monday&#8217;s revenue is frozen at the moment of checkout and it will only ever be too high.</p>



<p class="wp-block-paragraph">This is the invisible failure. Nothing alerts. No queue backs up. Your dashboard is confidently wrong, and the gap widens roughly in proportion to your return rate. A store with a two percent return rate takes a long time to notice. A fashion store running thirty percent returns notices in about a month, usually via an angry accountant.</p>



<h3 class="wp-block-heading">The four adjustments that move historical numbers</h3>



<ul class="wp-block-list">
<li><strong>Refunds.</strong> Full or partial. A refund carries its own <code>created_at</code>, which is when the money moved back. The order it belongs to has a different, earlier date. You need both, and which one you attribute to depends on whether finance wants cash-basis or order-basis reporting. Ask before you build.</li>

<li><strong>Order edits.</strong> A merchant adds a line item or adjusts a quantity after the fact. The original payload is now stale. Shopify exposes both the original and the current totals precisely because of this.</li>

<li><strong>Cancellations.</strong> A cancelled order keeps existing in the API. If you filter only on payment status you will happily keep counting it.</li>

<li><strong>Test and draft orders.</strong> Test orders carry a flag marking them as such. Nobody remembers to filter these until a QA run during a quiet week produces a suspicious spike.</li>
</ul>



<p class="wp-block-paragraph">The design consequence is simple to state and annoying to implement: <strong>your pipeline must be able to recompute any past day.</strong> Every storage and partitioning decision below follows from that one requirement.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Getting data out of Shopify: three paths, three trade-offs</h2>



<p class="wp-block-paragraph">Before anything else: new Shopify apps are built on the GraphQL Admin API. The REST Admin API has been designated a legacy API and new public apps must use GraphQL. If you&#8217;re starting fresh, start there. If you inherited a REST integration, it probably still runs, but you&#8217;re on borrowed time and you should plan the migration rather than discover the deadline.</p>



<h3 class="wp-block-heading">Path 1: EventBridge partner event source</h3>



<p class="wp-block-paragraph">Shopify can deliver webhooks straight into an Amazon EventBridge partner event bus in your account. No public endpoint, no API Gateway, no HMAC verification code, because verification only applies to HTTPS deliveries. Shopify&#8217;s own docs confirm EventBridge and Pub/Sub deliveries skip it.</p>



<p class="wp-block-paragraph">You create the source in the Shopify app configuration using your AWS account ID, region and a source name, then associate it with an event bus in the EventBridge console and write rules to route it. The address you register with Shopify is the <em>partner event source</em> ARN, not the event bus ARN. That distinction accounts for a large share of the &#8220;I set it up and nothing arrives&#8221; threads on the Shopify forums.</p>



<p class="wp-block-paragraph">A rule matching everything from the Shopify partner source looks like this. Start broad, then narrow once you&#8217;ve seen the real shape of an event:</p>



<pre class="wp-block-code"><code>{
  "source": [ { "prefix": "aws.partner/shopify.com" } ]
}</code></pre>



<p class="wp-block-paragraph">Send that to an SQS queue with a dead-letter queue attached rather than straight to Lambda. Buffering gives you a replay buffer when a downstream deploy goes wrong, and the DLQ means a bad payload parks itself instead of poisoning the whole rule. This is the path I reach for first for anything already on AWS.</p>



<h3 class="wp-block-heading">Path 2: HTTPS webhooks into API Gateway and Lambda</h3>



<p class="wp-block-paragraph">The conventional route, and the right one if you need webhook delivery outside AWS too, or you want the payloads to pass through something you fully control. The cost is that you now own an internet-facing endpoint and the HMAC verification on it.</p>



<p class="wp-block-paragraph">Verify against the <strong>raw request body</strong>, before any JSON parsing. Re-serialising the payload changes byte-for-byte content and the signature will never match. Use a constant-time comparison so the check doesn&#8217;t leak timing information:</p>



<pre class="wp-block-code"><code>import base64, hashlib, hmac

def verify(raw_body: bytes, header_hmac: str, secret: str) -&gt; bool:
    digest = hmac.new(secret.encode(), raw_body, hashlib.sha256).digest()
    computed = base64.b64encode(digest).decode()
    return hmac.compare_digest(computed, header_hmac)</code></pre>



<p class="wp-block-paragraph">Shopify sends the signature in the <code>X-Shopify-Hmac-SHA256</code> header, base64-encoded, computed with your app&#8217;s client secret over the raw body. Store that secret in Secrets Manager or as an SSM SecureString parameter, not in a Lambda environment variable.</p>



<p class="wp-block-paragraph">Acknowledge fast. Shopify&#8217;s timeout is short and it retries with backoff over a finite window, so a handler that does real work inline will generate a wall of duplicate deliveries during a flash sale, exactly when you can least afford it. Return 2xx immediately, do the work asynchronously.</p>



<h3 class="wp-block-heading">Path 3: scheduled GraphQL bulk pull</h3>



<p class="wp-block-paragraph">Webhooks give you low latency. They do not give you completeness. Anything that fails past its retry window is gone, and Shopify will eventually remove a subscription that keeps failing. That&#8217;s a silent data loss mode with no local symptom at all.</p>



<p class="wp-block-paragraph">So run a scheduled reconciliation pull alongside the stream. Shopify&#8217;s GraphQL bulk operations are built for this: you submit a query, it runs asynchronously, and you fetch a JSONL result file when it finishes. That&#8217;s the right tool for backfills and nightly catch-up, rather than paginating thousands of pages against a points-based rate limiter and getting throttled halfway through.</p>



<p class="wp-block-paragraph">A nightly job that re-pulls the last seven to fourteen days and overwrites those partitions costs almost nothing and quietly fixes every category of drift described above. If you build one thing from this post, build that.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Decide what &#8220;revenue&#8221; means before you write a line of SQL</h2>



<p class="wp-block-paragraph">This is where most reconciliation arguments actually live, and it isn&#8217;t an engineering problem at all until you&#8217;ve had the conversation.</p>



<p class="wp-block-paragraph">Shopify&#8217;s own sales reporting builds total sales from gross sales, minus discounts, minus returns, plus taxes and shipping. Gift card sales sit outside that in a separate finance report. If your dashboard sums order totals and calls it revenue, you have built a different metric with the same name, and it will disagree with the admin forever no matter how good your pipeline is.</p>



<p class="wp-block-paragraph">Write the definition down. Put it in the dashboard as a tooltip. When someone challenges a number, you want the argument to be about the definition, not about whether your infrastructure works.</p>



<h3 class="wp-block-heading">The currency trap</h3>



<p class="wp-block-paragraph">If the store sells in more than one currency, the money fields split in two. Shopify exposes totals as a set containing both <code>shop_money</code> and <code>presentment_money</code>: the amount in the store&#8217;s base currency, and the amount the customer actually saw and paid.</p>



<p class="wp-block-paragraph">Sum the presentment amounts across a multi-currency store and you get a number with no meaning at all, euros and yen added together as if they were the same unit. For a single reporting figure you want the shop-currency side. Keep the presentment amount and its currency code in the table anyway, because the day someone asks &#8220;how much did we actually sell in Germany&#8221;, you&#8217;ll want it and it is painful to backfill.</p>



<p class="wp-block-paragraph">One caveat worth knowing: orders created through the API rather than through checkout can behave differently from native multi-currency checkout orders. If your store takes orders from an ERP or a marketplace integration, spot-check a few of those specifically.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Storage layout: partition by order date, never by arrival date</h2>



<p class="wp-block-paragraph">Two layers in S3. Keep them separate and keep them honest about what they are.</p>



<ol class="wp-block-list">
<li><strong>Raw.</strong> Every payload exactly as received, partitioned by ingestion date. Append-only, never edited. This is your audit trail and your rebuild source. Lifecycle it to a colder storage class after a few months, don&#8217;t delete it.</li>

<li><strong>Curated.</strong> One row per order representing current state, in Parquet, partitioned by <em>order date</em>. This is what the dashboard queries. It is derived, disposable and rewritable.</li>
</ol>



<p class="wp-block-paragraph">The partitioning choice on the curated layer is the load-bearing decision in the whole design. If you partition by arrival date, which is what Amazon Data Firehose does by default because it buckets on the moment it writes the file, then a refund that arrives three weeks late lands in today&#8217;s partition. Correcting Monday now means finding and rewriting fragments scattered across twenty other partitions. Partitioned by order date, correcting Monday means overwriting exactly one prefix.</p>



<p class="wp-block-paragraph">Firehose can do this with dynamic partitioning, which routes records by keys inside the payload rather than by write time. If you&#8217;re not using Firehose, extract the order date in your Lambda and write the prefix yourself.</p>



<h3 class="wp-block-heading">Use partition projection so Athena stops guessing</h3>



<p class="wp-block-paragraph">The default Glue Data Catalog approach means running a crawler or issuing <code>MSCK REPAIR TABLE</code> to register new partitions. Forget one and you get a query that silently returns nothing for recent days. Nobody notices until Monday.</p>



<p class="wp-block-paragraph">Partition projection removes the metastore lookup entirely. You tell Athena the shape of the partition keys and it calculates the prefixes at query time:</p>



<pre class="wp-block-code"><code>CREATE EXTERNAL TABLE shop_orders (
  order_id             bigint,
  order_number         string,
  created_at           timestamp,
  financial_status     string,
  cancelled_at         timestamp,
  is_test              boolean,
  total_shop           decimal(12,2),
  shop_currency        string,
  total_presentment    decimal(12,2),
  presentment_currency string
)
PARTITIONED BY (order_date string)
STORED AS PARQUET
LOCATION 's3://your-bucket/curated/orders/'
TBLPROPERTIES (
  'projection.enabled' = 'true',
  'projection.order_date.type' = 'date',
  'projection.order_date.format' = 'yyyy-MM-dd',
  'projection.order_date.range' = '2019-01-01,NOW',
  'projection.order_date.interval' = '1',
  'projection.order_date.interval.unit' = 'DAYS',
  'storage.location.template' =
    's3://your-bucket/curated/orders/order_date=${order_date}/'
);</code></pre>



<p class="wp-block-paragraph">Set the range start to your store&#8217;s actual first order month. Projection generates every prefix in the range, so a range starting a decade too early makes wide scans slower for no benefit.</p>



<p class="wp-block-paragraph">Parquet matters here for the same reason. Athena bills on bytes scanned, so a columnar format with good compression cuts the bill directly, and a dashboard that only ever selects six columns from a forty-column table never touches the rest.</p>



<h3 class="wp-block-heading">Net sales in one query</h3>



<p class="wp-block-paragraph">With refunds in their own table keyed by order and carrying their own date, attributing them back to the original order day is a left join and a subtraction:</p>



<pre class="wp-block-code"><code>SELECT
    o.order_date,
    SUM(o.total_shop)                                AS gross_shop,
    SUM(COALESCE(r.refunded_shop, 0))                AS refunded_shop,
    SUM(o.total_shop - COALESCE(r.refunded_shop, 0)) AS net_shop
FROM shop_orders o
LEFT JOIN (
    SELECT order_id, SUM(amount_shop) AS refunded_shop
    FROM shop_refunds
    GROUP BY order_id
) r ON r.order_id = o.order_id
WHERE o.order_date BETWEEN '2025-01-01' AND '2025-01-31'
  AND o.is_test = false
  AND o.cancelled_at IS NULL
GROUP BY o.order_date
ORDER BY o.order_date;</code></pre>



<p class="wp-block-paragraph">Note the two filters doing quiet work at the bottom. Those two lines are the difference between a number finance accepts and a number they don&#8217;t.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Choosing the dashboard layer</h2>



<p class="wp-block-paragraph">Once the data is correct, this part is genuinely a preference. All of these work.</p>



<ul class="wp-block-list">
<li><strong>Amazon QuickSight</strong>, now delivered as part of Amazon Quick Suite, is the least-friction option if you&#8217;re already in AWS. Its in-memory SPICE layer means viewers aren&#8217;t firing an Athena query per chart interaction, which controls both latency and scan cost. Per-viewer pricing tends to be the deciding factor either way, so model it for your actual audience size before committing.</li>

<li><strong>Grafana</strong> with the Athena data source is a good fit if you&#8217;re already running Grafana for infrastructure and want commercial and operational panels on one screen. Grafana Cloud removes the hosting question if you&#8217;d rather not run it.</li>

<li><strong>Power BI</strong> makes sense when the finance team already lives in Microsoft 365 and models in DAX. The cross-cloud hop is real but manageable.</li>

<li><strong>Metabase</strong> or a self-hosted alternative on a small VPS from a provider like InterServer or Hetzner is the pragmatic answer for a handful of internal viewers, where per-seat BI licensing costs more than the entire pipeline.</li>
</ul>



<p class="wp-block-paragraph">The honest trade-off: managed BI costs more per month and saves you from becoming the person who patches the reporting server. Self-hosting inverts that. Neither is wrong, but pick deliberately rather than by inertia.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting: symptom to cause</h2>



<h3 class="wp-block-heading">Totals are consistently higher than the Shopify admin</h3>



<p class="wp-block-paragraph">Almost always refunds, cancellations or test orders. Check in that order. If the gap grows with the age of the reporting window, it&#8217;s refunds. If it&#8217;s a fixed offset on specific days, look for test orders or a QA run.</p>



<h3 class="wp-block-heading">Totals are lower, and recent days are missing rows</h3>



<p class="wp-block-paragraph">Either partitions aren&#8217;t registered, which projection fixes permanently, or the webhook subscription has been dropped after repeated delivery failures. Check the subscription still exists before you go digging through Lambda logs. A nightly bulk pull would have masked this, which is another argument for having one.</p>



<h3 class="wp-block-heading">Orders appear twice</h3>



<p class="wp-block-paragraph">Shopify&#8217;s delivery model is at-least-once, not exactly-once, and you may also have more than one subscription on the same topic. Deduplicate on the delivery ID header before you touch anything else, and make the write itself idempotent so a duplicate is a no-op rather than a second row.</p>



<h3 class="wp-block-heading">Numbers are right on the daily view, wrong on the monthly</h3>



<p class="wp-block-paragraph">Timezone. Order timestamps carry an offset; your partition key is a date string. If you derive the date in UTC and the store reports in a local timezone, orders near midnight land on the wrong day. That averages out over a month, which is exactly why the discrepancy hides until month boundaries.</p>



<h3 class="wp-block-heading">Athena costs jumped without more data</h3>



<p class="wp-block-paragraph">Someone built a dashboard with a filter that doesn&#8217;t hit the partition column, so every panel refresh scans the full table. Look at bytes scanned per query and check whether the BI tool is caching results or re-querying on every interaction.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating orders as immutable events and never revisiting a past day.</li>

<li>Partitioning on arrival time because that&#8217;s the default, then discovering corrections are expensive.</li>

<li>Verifying the HMAC against a re-serialised body instead of the raw bytes.</li>

<li>Doing real work inside the webhook handler, generating duplicates under load.</li>

<li>Summing presentment amounts across currencies.</li>

<li>Registering the event bus ARN with Shopify instead of the partner event source ARN.</li>

<li>Relying on webhooks alone with no scheduled reconciliation.</li>

<li>Shipping a &#8220;revenue&#8221; number without ever defining what it includes.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Keep raw and curated layers separate. Raw is append-only; curated is rewritable.</li>

<li>Make every partition idempotently rebuildable from raw. Test that path deliberately, before you need it.</li>

<li>Buffer through SQS with a dead-letter queue. Free replay, free isolation of bad payloads.</li>

<li>Run a nightly bulk pull over a rolling window and overwrite those partitions.</li>

<li>Use partition projection. It removes an entire category of silent failure.</li>

<li>Alarm on the absence of events, not just on errors. A CloudWatch alarm on zero orders processed in an hour during business hours catches broken subscriptions the same day.</li>

<li>Store the API secret in Secrets Manager and scope the Lambda role to the exact prefixes it writes.</li>

<li>Publish a reconciliation panel comparing your total to the admin&#8217;s for the same window. Surfacing the gap builds more trust than hiding it.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need a data warehouse, or is S3 and Athena enough?</h3>



<p class="wp-block-paragraph">For a single store&#8217;s order data, S3 with Athena is almost certainly enough, and it&#8217;s cheaper because you pay per query rather than for a running cluster. Redshift starts to earn its place when you&#8217;re joining Shopify data against several other large sources, or when concurrent query load makes Athena&#8217;s queue times noticeable.</p>



<h3 class="wp-block-heading">How near-real-time can this be?</h3>



<p class="wp-block-paragraph">Events land within seconds. The practical floor is your buffering window, since writing one tiny file per order gives you a small-files problem that ruins query performance. A few minutes of buffering is the usual compromise. If you genuinely need sub-minute order counts, put a live counter in DynamoDB alongside the analytical pipeline rather than trying to make the data lake do both jobs.</p>



<h3 class="wp-block-heading">Which webhook topics should I subscribe to?</h3>



<p class="wp-block-paragraph">At minimum, order creation, order update, order cancellation and refund creation. Update and refund topics are the ones people skip, and they&#8217;re exactly the ones carrying the corrections. Subscribe to fewer topics than you think you need and add rather than subscribing to everything, since every extra topic is volume you pay to store and process.</p>



<h3 class="wp-block-heading">Can I skip AWS and use a connector tool?</h3>



<p class="wp-block-paragraph">Yes, and for many stores that&#8217;s the right answer. A managed connector into a hosted warehouse gets you a working dashboard in an afternoon. You&#8217;re paying a monthly fee to avoid owning any of this, and trading away control over the data model. Building it on AWS wins when you need Shopify data joined to systems the connector doesn&#8217;t cover, or when row-based connector pricing outgrows the infrastructure cost.</p>



<h3 class="wp-block-heading">How do I backfill historical orders?</h3>



<p class="wp-block-paragraph">Use a GraphQL bulk operation rather than paginating the API. Submit the query, poll for completion, then stream the JSONL result into your raw bucket and run the same transformation your live pipeline uses. If backfill and live processing use different code paths, they will diverge, and you&#8217;ll spend an afternoon working out which one is lying.</p>



<h3 class="wp-block-heading">What does a setup like this cost to run?</h3>



<p class="wp-block-paragraph">For a typical single store, the pipeline itself is small money: Lambda invocations, a few gigabytes in S3, and Athena billed on bytes scanned, which partitioning and Parquet keep low. The BI seats are usually the largest line item, which is why the dashboard layer decision deserves more thought than the ingestion one. Model it against current published rates rather than trusting any figure you read in a blog post, including this one.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">A Shopify sales dashboard with AWS doesn&#8217;t fail because the pipeline breaks. It fails because the pipeline keeps working perfectly on data that has since changed underneath it.</p>



<p class="wp-block-paragraph">Design for correction from the first commit. Partition by order date, keep the raw layer so you can always rebuild, run a scheduled pull to catch what the stream missed, and agree on what revenue means before anyone builds a chart. Do that and the Monday morning screenshot comparison becomes a non-event, which is the highest praise a reporting pipeline ever gets.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with your Shopify data pipeline on AWS?</h2>



<p class="wp-block-paragraph">I design and build ecommerce data pipelines and reporting stacks on AWS. Typical engagements look like:</p>



<ul class="wp-block-list">
<li>Working out why an existing Shopify dashboard disagrees with the admin, and fixing the root cause rather than patching the query</li>

<li>Building the ingestion layer end to end: EventBridge or API Gateway, Lambda, SQS with dead-letter handling, and a scheduled GraphQL bulk reconciliation job</li>

<li>Designing the S3 layout, Glue schema and Athena tables so past days can be recomputed cheaply and partitions never go missing</li>

<li>Migrating REST Admin API integrations to GraphQL before the deadline forces the issue</li>

<li>Building the dashboard itself in QuickSight, Grafana or Metabase, including the metric definitions finance will actually sign off on</li>

<li>Cutting Athena scan costs and BI licensing on a reporting stack that has grown more expensive than anyone planned</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;re in the middle of one of these, send me the actual thing: the Athena query, the S3 prefix layout, the two totals that don&#8217;t match. It&#8217;s a much faster conversation than describing it in the abstract.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/shopify-sales-dashboard-aws/">Shopify Sales Dashboard with AWS: Build One That Actually Reconciles</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
