<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Shopify | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/shopify/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/shopify/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sun, 20 Sep 2026 10:43:53 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Shopify | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/shopify/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>One Customer, Four Systems: Unifying Shopify, Stripe, CRM and Support Data</title>
		<link>https://john-nessime.com/blog/technical-guides/unify-shopify-stripe-crm-support-data/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 29 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Customer 360]]></category>
		<category><![CDATA[Customer Support Analytics]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Data Warehouse]]></category>
		<category><![CDATA[dbt]]></category>
		<category><![CDATA[Ecommerce Analytics]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Event-Driven Architecture]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Identity Resolution]]></category>
		<category><![CDATA[PII Redaction]]></category>
		<category><![CDATA[Reverse ETL]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[Stripe]]></category>
		<category><![CDATA[Webhooks]]></category>
		<category><![CDATA[Zendesk]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=350</guid>

					<description><![CDATA[<p>Three CRM records, one person, and nothing in the logs to explain it. A practical walkthrough of what actually breaks when you unify Shopify, Stripe, CRM and support data: guest-checkout identity, duplicate and out-of-order webhooks, revenue that never ties to payouts, deletion requests that miss half your copies, and reverse ETL write loops. Organized by failure family, with schema and handler patterns you can apply directly.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/unify-shopify-stripe-crm-support-data/">One Customer, Four Systems: Unifying Shopify, Stripe, CRM and Support Data</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The project usually starts with a screenshot. Someone in support pastes three CRM records into a channel, all carrying the same person&#8217;s name, and asks which one is real. One has the order history. One has the subscription. The third has four tickets attached and no orders at all.</p>



<p class="wp-block-paragraph">Nobody broke anything. Shopify, Stripe, the CRM and the helpdesk each created a customer record using whatever identifier they had at the moment they needed one, and none of those identifiers agreed. The systems are all correct in isolation. The person is the thing that got lost.</p>



<p class="wp-block-paragraph">This post covers what actually breaks when you unify Shopify, Stripe, CRM and support data, and how to build the join so it survives guest checkouts, duplicate webhooks, refunds, currency conversion and deletion requests. It&#8217;s organized by failure family rather than by tool, because the tools get acquired and renamed every couple of years and the failures never change.</p>



<h2 class="wp-block-heading">Failure family one: the join key is not the email address</h2>



<p class="wp-block-paragraph">Every one of these systems will happily give you an email address, which is exactly why so many pipelines join on it and quietly produce garbage.</p>



<p class="wp-block-paragraph">Start with where identity actually lives in each system:</p>



<ul class="wp-block-list">
<li><strong>Shopify</strong> has a customer ID, but an order does not have to be attached to one. Guest checkout produces an order with an email on it and no durable customer record behind it.</li>

<li><strong>Stripe</strong> has a customer ID, but only if a Customer object was created. Checkout Sessions that don&#8217;t create one are associated with guest customers instead, which Stripe groups in the Dashboard based on the same card, email or phone. That grouping is read-only and it is not an ID you can join on.</li>

<li><strong>The CRM</strong> keys on email in almost every default configuration, and cheerfully creates a second contact when the email differs by one character.</li>

<li><strong>The helpdesk</strong> keys on the requester&#8217;s email, which is whatever the person typed into a contact form or sent the ticket from. It is very often not the address on the order.</li>
</ul>



<p class="wp-block-paragraph">Here&#8217;s the part that stays invisible until someone finally checks: nothing errors. Every dashboard still returns a number. Lifetime value gets silently split across three profiles, so your LTV looks lower and your repeat-purchase rate looks worse than it is. Support metrics look <em>better</em> than they are, because a customer coming back with the same unresolved problem shows up as a brand new contact. You can run like that for a year without a single failed job.</p>



<h3 class="wp-block-heading">Build an identifier table, not a fuzzy match</h3>



<p class="wp-block-paragraph">The fix is boring and it works: a surrogate person key, and a separate table of every identifier ever observed for that person, with the source that reported it.</p>



<pre class="wp-block-code"><code>-- One row per real human, generated by you, owned by you.
CREATE TABLE person (
  person_key      uuid PRIMARY KEY,
  created_at      timestamptz NOT NULL DEFAULT now(),
  merged_into     uuid NULL REFERENCES person(person_key)
);

-- One row per identifier per source. This is the whole trick.
CREATE TABLE person_identifier (
  person_key      uuid NOT NULL REFERENCES person(person_key),
  source          text NOT NULL,   -- shopify | stripe | crm | helpdesk
  id_type         text NOT NULL,   -- email | phone | shopify_customer_id | stripe_customer_id
  id_value        text NOT NULL,   -- already normalized
  first_seen_at   timestamptz NOT NULL,
  last_seen_at    timestamptz NOT NULL,
  PRIMARY KEY (source, id_type, id_value)
);</code></pre>



<p class="wp-block-paragraph">Normalization happens before the row is written, not in the join. Lowercase and trim emails, store phone numbers in E.164, strip whitespace from external IDs. Resist the urge to get clever with Gmail&#8217;s dot and plus handling. Stripping <code>+tags</code> is a business policy decision with real consequences, not a technical detail, and the same rule applied to a domain that treats plus addresses as distinct mailboxes will merge two genuinely different people.</p>



<p class="wp-block-paragraph">Deterministic matching first, always. Exact matches on shared identifiers, transitively closed. Probabilistic matching on name plus address plus device is worth reaching for only when you have exhausted the deterministic keys, and it has an asymmetric failure mode: a household sharing one card gets collapsed into one person, and unmerging afterwards is expensive because every downstream system has already copied the merged key into its own records. Merges are cheap to make and painful to undo. Bias toward leaving two records separate.</p>



<p class="wp-block-paragraph">One thing worth doing early on the Stripe side: when you create a Customer, write your own identifier into <code>metadata</code>. Stripe allows fifty key-value pairs per object, with keys up to forty characters and values up to five hundred, and it only returns metadata for requests made with a secret key. That&#8217;s enough room for a person key and an order reference, which turns a future fuzzy join into an exact one. Never put anything sensitive there.</p>



<h2 class="wp-block-heading">Failure family two: webhooks arrive twice, out of order, or not at all</h2>



<p class="wp-block-paragraph">Both platforms are explicit about this, and both get treated as if they weren&#8217;t.</p>



<p class="wp-block-paragraph">Stripe delivers at least once, does not guarantee ordering, and retries failed deliveries with exponential backoff for up to three days. The event <code>id</code> (the one starting <code>evt_</code>) stays identical across every retry, which makes it your deduplication key. Shopify sends <code>X-Shopify-Webhook-Id</code> for the same purpose, signs the raw body into <code>X-Shopify-Hmac-Sha256</code>, and states plainly that ordering isn&#8217;t guaranteed within a topic or across topics for the same resource. Shopify&#8217;s own documentation goes further and tells you not to rely on webhooks as your only source of data.</p>



<p class="wp-block-paragraph">For a customer-data pipeline specifically, out-of-order delivery is nastier than duplication. Consider a <code>customers/update</code> that changes someone&#8217;s email arriving before the <code>orders/create</code> that was placed under the old address. Process those in arrival order and you attach the order to a person who doesn&#8217;t exist yet, or worse, to the wrong one. Order your writes by the event&#8217;s own timestamp, using <code>X-Shopify-Triggered-At</code> or the <code>updated_at</code> in the payload, and make profile updates last-writer-wins on that timestamp rather than on insertion time.</p>



<h3 class="wp-block-heading">Do almost nothing in the handler</h3>



<p class="wp-block-paragraph">Verify the signature, claim the event ID, write the raw body to durable storage, return 2xx. That&#8217;s it. Identity resolution, CRM lookups and enrichment calls all belong in a worker reading from that store. A handler that tries to resolve identity inline will eventually be slow enough to time out, which triggers a retry, which arrives while the first attempt is still running.</p>



<p class="wp-block-paragraph">Make the claim atomic rather than check-then-act, because two retries genuinely can land concurrently:</p>



<pre class="wp-block-code"><code>-- Atomic claim. If this inserts zero rows, someone already has it.
INSERT INTO webhook_event (source, event_id, topic, received_at, payload)
VALUES ($1, $2, $3, now(), $4)
ON CONFLICT (source, event_id) DO NOTHING
RETURNING event_id;</code></pre>



<p class="wp-block-paragraph">Return a 2xx for a duplicate. A duplicate is a success from the sender&#8217;s point of view, and returning an error just schedules more of them.</p>



<p class="wp-block-paragraph">Signature verification has one recurring trap on the Shopify side: it runs against the <em>raw</em> body. Any JSON body parser that runs first will change the bytes and every signature will fail. Capture the raw body before parsing, and compare in constant time.</p>



<pre class="wp-block-code"><code>import base64, hashlib, hmac

def verify_shopify(raw_body: bytes, header_value: str, client_secret: str) -&gt; bool:
    digest = hmac.new(
        client_secret.encode("utf-8"),
        raw_body,
        hashlib.sha256,
    ).digest()
    expected = base64.b64encode(digest).decode("utf-8")
    return hmac.compare_digest(expected, header_value)</code></pre>



<p class="wp-block-paragraph">Worth knowing before you spend an afternoon on it: after rotating an app&#8217;s client secret, Shopify notes it can take up to an hour before signatures are generated with the new one. If verification starts failing right after a rotation, that&#8217;s usually why.</p>



<p class="wp-block-paragraph">And because delivery isn&#8217;t guaranteed, webhooks alone are never enough. Run a reconciliation job that pulls orders and customers modified since the last watermark and fills the gaps. This is the job that saves you during an outage on your side, and it&#8217;s the one people skip. If you&#8217;d rather not build the buffering and replay layer, managed webhook infrastructure like Hookdeck exists for exactly this, and self-hosting a small queue and worker on a modest VPS from somewhere like Contabo or InterServer is perfectly reasonable too. The point is that something has to catch what the webhook missed.</p>



<h2 class="wp-block-heading">Failure family three: &#8220;revenue&#8221; means four different things</h2>



<p class="wp-block-paragraph">This is the one that turns into a meeting. Finance says the number is wrong. Analytics says the number is right. Both are correct, because they&#8217;re measuring different things and nobody wrote down which.</p>



<ul class="wp-block-list">
<li>Shopify reports gross sales at the moment the order is placed, in the store&#8217;s currency and the store&#8217;s timezone.</li>

<li>Stripe pays out in batches, net of processing fees, refunds and disputes, on its own schedule, in the settlement currency of the account.</li>

<li>A refund issued in Shopify and a refund issued in Stripe are two different events that may or may not be the same underlying money movement.</li>

<li>A dispute lands days or weeks after the order and reverses revenue in a period that has already been reported.</li>
</ul>



<p class="wp-block-paragraph">There&#8217;s a structural trap here too. If the store runs Shopify Payments, there is no Stripe account for you to query. Shopify Payments is built on Stripe&#8217;s processing infrastructure, but it&#8217;s Shopify&#8217;s gateway, and payouts, fees and disputes live in the Shopify admin rather than behind a Stripe API key. Plenty of businesses end up with Shopify Payments handling storefront orders and a separate Stripe account handling subscriptions or invoicing, which means two payment systems, two payout schedules and one customer who has no idea any of this exists.</p>



<p class="wp-block-paragraph">Do not build a single <code>revenue</code> column. Build separate fact tables at their natural grain (orders, order line items, payments, refunds, disputes, payouts) plus a bridging table that links a payout back to the transactions inside it. Then let each team define its own metric on top. Finance gets net settled cash. Marketing gets gross booked revenue. Neither has to be wrong for the other to be right.</p>



<p class="wp-block-paragraph">Store every timestamp in UTC and keep the original timezone as a separate column. Store every amount in minor units as an integer alongside its currency code, and store the FX rate that was actually applied rather than recalculating it later from a rates table. Recalculated FX is the single most common source of &#8220;the number moved and nobody changed anything.&#8221;</p>



<p class="wp-block-paragraph">On ingestion, Stripe Data Pipeline can push Stripe data directly into Snowflake, Redshift or Databricks as a data share, which removes a connector from the picture if you&#8217;re already on one of those. Otherwise a managed ELT tool like Fivetran or a self-hosted Airbyte will handle Shopify, Stripe, the CRM and the helpdesk with off-the-shelf connectors. Both routes are fine. The connector is genuinely the easy part.</p>



<h2 class="wp-block-heading">Failure family four: deletion is the requirement nobody models</h2>



<p class="wp-block-paragraph">If you&#8217;re building this as a Shopify app, three compliance webhooks are mandatory: <code>customers/data_request</code>, <code>customers/redact</code> and <code>shop/redact</code>. You acknowledge with a 200 and complete the action within thirty days. <code>shop/redact</code> fires forty-eight hours after uninstall and covers the shop&#8217;s data. These are enforced at app review, so you&#8217;ll find out about them either way.</p>



<p class="wp-block-paragraph">The interesting problem isn&#8217;t receiving the webhook. It&#8217;s that by the time it arrives, that person&#8217;s email exists in your raw landing zone, your staging tables, your identity table, your customer view, a BI extract, the CRM, the helpdesk, and a Slack message from a debugging session six months ago. A delete that only touches the golden record is not a delete.</p>



<p class="wp-block-paragraph">The design decision that makes this tractable is to hold personal data in exactly one place. Every downstream table carries <code>person_key</code> and nothing else. Facts, aggregates, models, extracts and syncs all reference the key. When a redaction request lands, you redact one row set in the identifier table and the rest of the warehouse is already anonymous. Getting this right at schema-design time costs an afternoon; retrofitting it costs a quarter.</p>



<p class="wp-block-paragraph">Practical pieces that make it real:</p>



<ul class="wp-block-list">
<li>Tag every table and column that can contain an identifier, in your catalog or in dbt metadata, and treat the tag list as the deletion scope.</li>

<li>Partition the raw landing zone so you can rewrite a partition rather than scan an entire bucket looking for one email.</li>

<li>Keep a deletion queue with a status per downstream system, including the CRM and the helpdesk, which have their own APIs and their own idea of what deletion means.</li>

<li>Write down what you retain and why. Financial records generally have to survive a deletion request for tax and dispute reasons. Say so rather than promising a hard delete you can&#8217;t perform.</li>
</ul>



<h2 class="wp-block-heading">Failure family five: pushing the unified view back out</h2>



<p class="wp-block-paragraph">A customer view nobody can act on is a report. The value shows up when a support agent opens a ticket and can see subscription status and lifetime order count without switching tabs.</p>



<p class="wp-block-paragraph">That&#8217;s reverse ETL, and the market is well served: Hightouch, Census (now part of Fivetran), Polytomic, RudderStack, and iPaaS-style tools like Workato if you need branching logic rather than field syncs. The warehouse-native argument is straightforward and mostly correct: your models already live there, and you avoid maintaining a second copy of the customer profile inside a vendor&#8217;s rigid schema. The counter-argument is also real. Reverse ETL is batch by nature, so if you need sub-minute reaction to an event, a packaged CDP or a direct event pipeline will serve you better, and you&#8217;ll be running several vendors plus warehouse compute plus the people to maintain the models.</p>



<p class="wp-block-paragraph">Three things go wrong here more or less every time.</p>



<p class="wp-block-paragraph"><strong>The write loop.</strong> You compute a churn-risk field and sync it into the CRM. Your ELT connector reads the CRM back into the warehouse. Now the computed field is an input to the model that computes it. This is genuinely hard to spot because nothing errors, the value just drifts. Namespace every synced field with a consistent prefix and explicitly exclude that namespace from ingestion.</p>



<p class="wp-block-paragraph"><strong>The resync stampede.</strong> Incremental syncs push diffs and stay small. Change a model definition and the next run wants to update every record, which will meet a helpdesk API rate limit at speed. Cap the batch size, and treat a full resync as a planned operation rather than something a merge to main can trigger.</p>



<p class="wp-block-paragraph"><strong>Syncing logic instead of keys.</strong> Push <code>person_key</code> into a custom field on the CRM contact and the helpdesk user. It makes the key searchable by agents, it gives the next pipeline a deterministic join, and it stops three teams from each rebuilding their own version of your matching rules.</p>



<h2 class="wp-block-heading">A pipeline shape that holds up</h2>



<p class="wp-block-paragraph">Put together, the order of operations matters more than the tool choice:</p>



<ol class="wp-block-list">
<li>Land raw. Every webhook body and every API page goes to immutable storage first, keyed by source and event ID, before anything interprets it.</li>

<li>Deduplicate on the provider&#8217;s event ID, atomically, with a retention window longer than the provider&#8217;s retry window.</li>

<li>Normalize identifiers into the identifier table as rows arrive, with the source recorded.</li>

<li>Resolve identity deterministically into <code>person_key</code>, as a scheduled job, never inside a request handler.</li>

<li>Build fact tables per system at their own grain, keyed on <code>person_key</code>, with no raw PII.</li>

<li>Model the customer view on top, in dbt or whatever your team already uses. The Fivetran <code>dbt_customer360</code> package is a reasonable starting point if you want to see how someone else structured the identity step.</li>

<li>Activate a small, named, namespaced set of fields back into the CRM and helpdesk.</li>

<li>Run reconciliation and freshness checks on a schedule, and alert on them like you would any other production job.</li>
</ol>



<h2 class="wp-block-heading">Troubleshooting: symptom to root cause</h2>



<p class="wp-block-paragraph">These are the ones that come up repeatedly, and the check that usually settles them.</p>



<ul class="wp-block-list">
<li><strong>Warehouse customer count is much higher than Shopify&#8217;s.</strong> Guest orders are minting a new person on every purchase. Check how many <code>person</code> rows have exactly one identifier and one order.</li>

<li><strong>Warehouse count is much lower.</strong> An over-eager merge rule. Look for person keys with an implausible number of distinct emails or shipping addresses attached.</li>

<li><strong>BI revenue doesn&#8217;t match Stripe payouts.</strong> Grain mismatch, not a data loss. Reconcile a single payout against its constituent transactions before touching anything else.</li>

<li><strong>A CRM field keeps flipping between two values.</strong> Write loop. Check whether the field&#8217;s namespace is excluded from your ELT source configuration.</li>

<li><strong>Recent orders missing, older ones fine.</strong> Webhook delivery gap. Confirm the reconciliation job ran, and check its watermark rather than trusting its exit code.</li>

<li><strong>Every Shopify signature suddenly fails.</strong> Either a body parser is running before verification, or the client secret was rotated within the last hour.</li>

<li><strong>Tickets attached to the wrong customer after a merge.</strong> The merge happened in the warehouse but the synced key in the helpdesk still points at the old person. Merges need to propagate outward, not just inward.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Joining on email in the model layer instead of resolving identity once, upstream.</li>

<li>Treating webhooks as the system of record instead of as a low-latency hint, with a reconciliation pull behind them.</li>

<li>Doing identity resolution, enrichment or CRM writes inside the webhook handler.</li>

<li>Ordering profile updates by arrival time rather than by the event&#8217;s own timestamp.</li>

<li>One <code>revenue</code> column that finance and marketing are both told to use.</li>

<li>Storing amounts as floats, or recalculating historical FX from a rates table.</li>

<li>Copying PII into every downstream table because it was convenient at the time.</li>

<li>Reaching for probabilistic matching before the deterministic keys have been exhausted.</li>

<li>Assuming a Shopify store on Shopify Payments has a Stripe account you can query.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>Own the surrogate key. Never let a vendor&#8217;s internal ID become the primary identity of a person in your systems.</li>

<li>Write your <code>person_key</code> into Stripe <code>metadata</code> and Shopify customer metafields at creation time, so future joins are exact rather than inferred.</li>

<li>Keep raw payloads immutable and replayable. Every model bug you&#8217;ll ever have is cheaper to fix if you can reprocess from raw.</li>

<li>Version your matching rules and record which version produced each merge, so you can audit and reverse a bad rule.</li>

<li>Alert on freshness and on row-count deltas per source, not just on job success. A connector returning zero rows exits cleanly.</li>

<li>Reconcile one payout end to end by hand before you trust the automated version. It&#8217;s the fastest way to find a grain error.</li>

<li>Document the metric definitions next to the models. Most &#8220;the data is wrong&#8221; tickets are definition disputes.</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Do I need a CDP to unify Shopify, Stripe, CRM and support data?</h3>



<p class="wp-block-paragraph">Not necessarily. If you already run a warehouse and have someone comfortable with dbt, ELT plus modelling plus reverse ETL covers the same ground with more control over the data model. A packaged CDP earns its cost when you need real-time behavioural segmentation, when nobody on the team can maintain models, or when marketing needs to ship changes without waiting on data engineering.</p>



<h3 class="wp-block-heading">How do I handle guest checkout customers?</h3>



<p class="wp-block-paragraph">Create a person from the order&#8217;s email anyway, and mark the record as identity-weak. When that email later appears attached to a real Shopify customer or Stripe Customer object, the deterministic match links them and the order history follows. The mistake is discarding guest orders because they have no customer ID, which is exactly the revenue you most want attributed.</p>



<h3 class="wp-block-heading">Should I sync everything into the CRM or keep it in the warehouse?</h3>



<p class="wp-block-paragraph">Keep the full picture in the warehouse and sync a deliberately small set of fields outward. The test is whether an agent or a rep will act on the field during a conversation. Lifetime value, subscription status, open ticket count and churn risk pass. A hundred-column profile does not, and it turns every model change into a rate-limit incident.</p>



<h3 class="wp-block-heading">Why don&#8217;t my Shopify sales match my Stripe payouts?</h3>



<p class="wp-block-paragraph">Because they&#8217;re measuring different things at different times. Shopify shows gross sales at order time; Stripe pays out batched amounts net of fees, refunds and disputes, and the timing differs by days. Add currency conversion and a timezone mismatch and the numbers will never tie without an explicit bridging table linking payouts to the transactions inside them.</p>



<h3 class="wp-block-heading">Can I use email as the primary key if my store requires account creation?</h3>



<p class="wp-block-paragraph">You can get away with it longer, but it&#8217;s still a bad primary key. People change email addresses, use one address for orders and another for support, and share addresses within a household. A surrogate key with an identifier table costs very little up front and means an email change is a new row rather than a broken join and a duplicate customer.</p>



<h3 class="wp-block-heading">How do I test the identity resolution before trusting it?</h3>



<p class="wp-block-paragraph">Run it in shadow mode. Compute person keys without activating them, then have support pick twenty customers they know are duplicates and twenty they know are distinct, and check what the resolver did. That exercise finds normalization bugs and over-eager merge rules faster than any aggregate metric, because the people who answer the tickets already know where the duplicates are.</p>



<h3 class="wp-block-heading">What&#8217;s the minimum viable version of this?</h3>



<p class="wp-block-paragraph">Raw landing with deduplication, an identifier table with deterministic matching, one fact table per source, and a single synced field into the helpdesk. That&#8217;s a couple of weeks of work and it removes most of the pain. Everything above it is refinement.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Conclusion: unify Shopify, Stripe, CRM and support data around a key you own</h2>



<p class="wp-block-paragraph">If you take one thing from this: the hard part of unifying Shopify, Stripe, CRM and support data is not moving the data. Connectors are commodity. The hard part is deciding what a customer <em>is</em>, writing that decision down as a key you own, and then refusing to let any downstream system quietly invent its own version of it.</p>



<p class="wp-block-paragraph">Do the identity work first, land everything raw so you can reprocess when you get it wrong, and keep personal data in one place so deletion stays a single operation. The dashboards will follow. Skip that order and you get exactly the screenshot this post opened with, eighteen months later and with a lot more rows behind it.</p>



<h2 class="wp-block-heading">Need help wiring this together?</h2>



<p class="wp-block-paragraph">I work on exactly this kind of plumbing: the unglamorous layer between an ecommerce platform, a payment processor and the tools your team actually opens all day. Things I can help with here:</p>



<ul class="wp-block-list">
<li>Designing the identity model (surrogate keys, identifier tables, deterministic matching rules) and auditing an existing one for over-merging.</li>

<li>Building idempotent, signature-verified webhook intake for Shopify and Stripe, with a dead-letter path and a reconciliation job behind it.</li>

<li>Setting up ELT into Snowflake, BigQuery, Redshift or Postgres, whether with Fivetran, self-hosted Airbyte, or Stripe Data Pipeline where it fits.</li>

<li>Modelling orders, payments, refunds, disputes and payouts so finance and marketing can each get a number they trust.</li>

<li>Reverse ETL back into the CRM and helpdesk without write loops, rate-limit incidents or a hundred junk fields.</li>

<li>Making deletion and data-subject requests actually work across the warehouse, the raw landing zone and every downstream copy.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;re stuck on a specific piece, send me the concrete thing: a webhook payload, a schema dump, the query that disagrees with your Stripe payout report. It&#8217;s much easier to say something useful about a real artifact than about a diagram.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/unify-shopify-stripe-crm-support-data/">One Customer, Four Systems: Unifying Shopify, Stripe, CRM and Support Data</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</title>
		<link>https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 17 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Cursor Pagination]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[GraphQL]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Orchestration]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[Step Functions]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=215</guid>

					<description><![CDATA[<p>A GraphQL source can hand you a 200 OK, a populated data block, and a quietly broken column in the same response. Here is how to build a GraphQL data ingestion pipeline on AWS that catches partial errors, respects cost-based rate limits, resumes cleanly from a cursor, and notices when the schema moves under you.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/">Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The row counts looked fine. That was the problem.</p>



<p class="wp-block-paragraph">A nightly sync from a partner&#8217;s GraphQL API had run for weeks without a single failed execution. Green across the board in the console. Then someone asked why one column had been empty since the previous Tuesday. Not sparse. Empty. Every row, every night, for six nights.</p>



<p class="wp-block-paragraph">The API had started returning a field-level error on that one field. Status code 200. A populated <code>data</code> block. An <code>errors</code> array sitting right next to it. The fetcher parsed <code>data</code>, wrote the records to S3, exited zero, and the orchestrator logged a success. Nothing in the stack was wrong, exactly. Everything in the stack was checking the wrong thing.</p>



<p class="wp-block-paragraph">This post covers how to build a GraphQL data ingestion pipeline on AWS that fails honestly. It is organised by failure family rather than by service, because the AWS wiring is the easy part. The hard part is that GraphQL breaks four assumptions that REST ingestion quietly relies on, and every one of them fails silently by default.</p>



<h2 class="wp-block-heading">Why GraphQL ingestion breaks the habits REST taught you</h2>



<p class="wp-block-paragraph">Most ETL code carries three inherited reflexes from years of REST work: the status code tells you whether it worked, the response shape is the server&#8217;s business, and rate limits are counted in requests. GraphQL invalidates all three.</p>



<ul class="wp-block-list">
<li><strong>The status code is not the contract.</strong> The GraphQL over HTTP specification says a server should return 200 even when field errors are raised during execution, because a partial response is still a successful execution.</li>

<li><strong>You own the response shape.</strong> The selection set is written by you, which means schema drift shows up as a query that stops being valid, not as a payload that changes underneath you.</li>

<li><strong>Cost is query-shaped, not endpoint-shaped.</strong> One endpoint can cost almost nothing or blow your entire budget depending on how deeply you nested it.</li>
</ul>



<p class="wp-block-paragraph">Each of those becomes a distinct failure family. Let&#8217;s take them in the order they bite.</p>



<h2 class="wp-block-heading">Failure one: the 200 OK that quietly ate a column</h2>



<p class="wp-block-paragraph">This is the one that costs the most and shows up the latest. GraphQL splits errors into two categories, and they behave completely differently on the wire.</p>



<ul class="wp-block-list">
<li><strong>Request errors</strong> happen before execution starts: syntax errors, validation failures, bad variable coercion. No <code>data</code> key is returned at all. Nothing executed.</li>

<li><strong>Field errors</strong> happen during execution: a resolver threw, a downstream service timed out, a permission check failed. Execution continues, the failed field goes null, and you get <code>data</code> and <code>errors</code> together.</li>
</ul>



<p class="wp-block-paragraph">Field errors are the dangerous ones for ingestion, because they look exactly like success to anything that only inspects the transport layer. Worse, GraphQL&#8217;s null propagation rules mean a single failing non-null field bubbles up to the nearest nullable ancestor, so one broken resolver can hollow out an entire nested object while the surrounding rows land normally.</p>



<p class="wp-block-paragraph">The fix is to make the response body the contract. Classify before you commit anything:</p>



<pre class="wp-block-code"><code>import json, urllib.request

class RequestError(Exception): pass
class FieldError(Exception): pass

def graphql(endpoint, token, query, variables):
    body = json.dumps({"query": query, "variables": variables}).encode()
    req = urllib.request.Request(
        endpoint,
        data=body,
        headers={
            "Content-Type": "application/json",
            "Accept": "application/graphql-response+json, application/json",
            "Authorization": f"Bearer {token}",
        },
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        payload = json.loads(resp.read())

    # The status code told us nothing. The body is the contract.
    if "data" not in payload:
        # Nothing executed. Query or auth problem. Do not retry blindly.
        raise RequestError(payload.get("errors"))

    if payload.get("errors"):
        # Partial response. data is populated but incomplete.
        raise FieldError(payload["errors"])

    return payload["data"]</code></pre>



<p class="wp-block-paragraph">Two things worth explaining. The <code>Accept</code> header asks for <code>application/graphql-response+json</code>, the media type the GraphQL over HTTP spec recommends, which lets a compliant server return a real 4xx for pre-execution failures instead of burying them in a 200. Servers that don&#8217;t support it fall back to <code>application/json</code>, so including both is free.</p>



<p class="wp-block-paragraph">The second is the split between the two exception types. They need different handling. A request error is a bug in your query or a credential problem, and retrying it just burns budget on the same failure. A field error is transient often enough to be worth retrying, but never worth committing.</p>



<h3 class="wp-block-heading">Deciding what &#8220;partial&#8221; means for your data</h3>



<p class="wp-block-paragraph">Failing the whole page on any error is the safe default and the one I reach for first. But it&#8217;s not always right. If you are pulling a hundred fields and one optional enrichment field is flaky, refusing every page forever is its own outage.</p>



<p class="wp-block-paragraph">The workable middle ground is to classify by path. Every error object carries a <code>path</code> array pointing at the exact field that failed. Maintain a small allowlist of paths that are permitted to be null, fail the page on anything outside it, and emit the error count as a CloudWatch metric so a rising trend is visible before someone in finance notices.</p>



<h2 class="wp-block-heading">Failure two: rate limits that never send a 429</h2>



<p class="wp-block-paragraph">Your retry logic almost certainly triggers on 429 and 5xx. Several major GraphQL APIs will throttle you without ever sending one.</p>



<p class="wp-block-paragraph">GitHub&#8217;s GraphQL API is explicit about this: exceeding the primary rate limit returns a 200 response with an error message in the body and <code>x-ratelimit-remaining</code> set to zero. Your Step Functions retrier sees a success. Your Lambda sees no data. The run completes, having ingested nothing, and reports green.</p>



<p class="wp-block-paragraph">The other half of the problem is that request counting is the wrong unit. Shopify&#8217;s GraphQL Admin API and several others price queries by <em>calculated cost</em>, using a leaky bucket. The server statically analyses your query before executing it, charges the bucket the worst-case <em>requested</em> cost, executes, then refunds the difference between requested and <em>actual</em> cost. That has a consequence people miss: a query asking for the maximum connection size gets charged as if every page were full, even when the last page returns three records.</p>



<p class="wp-block-paragraph">The good news is that these APIs tell you the state of the bucket on every response, under the <code>extensions</code> key:</p>



<pre class="wp-block-code"><code>{
  "extensions": {
    "cost": {
      "requestedQueryCost": 72,
      "actualQueryCost": 38,
      "throttleStatus": {
        "maximumAvailable": 1000.0,
        "currentlyAvailable": 962.0,
        "restoreRate": 100.0
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Read those numbers, never hardcode them. Bucket size and restore rate vary by plan tier, and a query that works fine against a large merchant&#8217;s store will fail instantly against a small one. The pattern that holds up is a pre-flight check: before dispatching, compare your estimated cost against <code>currentlyAvailable</code>, and if you&#8217;re short, sleep for the shortfall divided by <code>restoreRate</code>.</p>



<p class="wp-block-paragraph">On APIs that expose their own budget as a queryable field, ask for it in the same round trip rather than spending a second call to find out:</p>



<pre class="wp-block-code"><code>query BudgetCheck {
  rateLimit {
    limit
    cost
    remaining
    resetAt
  }
}</code></pre>



<p class="wp-block-paragraph">Two more things about concurrency. First, if you fan out across Lambda invocations, each worker&#8217;s local view of the bucket is wrong, because the bucket is shared. Multiple workers independently deciding they have headroom is the classic way to get throttled while every individual worker believes it&#8217;s behaving. Either centralise the token accounting or cap concurrency low enough that the arithmetic can&#8217;t go wrong.</p>



<p class="wp-block-paragraph">Second, if the source offers a bulk or async export operation, use it for backfills. Paginating fifty thousand records through the interactive endpoint burns the same budget your incremental syncs need, and it competes with itself.</p>



<h2 class="wp-block-heading">Failure three: cursors that don&#8217;t survive a restart</h2>



<p class="wp-block-paragraph">Most production GraphQL schemas follow the Relay cursor connections convention: <code>first</code> and <code>after</code> arguments, a <code>pageInfo</code> object with <code>hasNextPage</code> and <code>endCursor</code>, and edges carrying opaque per-item cursors.</p>



<pre class="wp-block-code"><code>query Orders($cursor: String) {
  orders(first: 100, after: $cursor) {
    pageInfo {
      hasNextPage
      endCursor
    }
    edges {
      cursor
      node {
        id
        updatedAt
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">The filter argument for incremental sync is not standardised. Some schemas take a query-string filter, some take a typed timestamp argument, some expose nothing at all. Run introspection against the schema and find out rather than guessing, because a wrong argument name is a request error and your run will fail before it starts.</p>



<p class="wp-block-paragraph">The failure here is subtle. Cursors are opaque and their validity is entirely up to the server. Some are stable positional pointers. Some encode a sort key and a snapshot identifier and expire. If a page-47 fetch fails and you retry an hour later with the stored cursor, you may get an error, or you may get results from a shifted window with a hole in the middle. The second outcome is worse, because it looks like it worked.</p>



<p class="wp-block-paragraph">Three habits make restarts safe:</p>



<ol class="wp-block-list">
<li><strong>Checkpoint after the write, not after the fetch.</strong> A cursor recorded before the page lands in S3 is a promise you can&#8217;t keep.</li>

<li><strong>Store a high watermark alongside the cursor.</strong> The cursor is your fast path. The watermark, taken from the newest record you actually committed, is your fallback when the cursor is rejected.</li>

<li><strong>Overlap the watermark deliberately.</strong> Restart a few minutes before the last committed timestamp and deduplicate on the primary key downstream. Re-reading a handful of rows is cheap. Missing rows that arrived during a clock skew is not.</li>
</ol>



<p class="wp-block-paragraph">A DynamoDB item per source stream is enough state for all of this:</p>



<pre class="wp-block-code"><code>{
  "source": "orders",
  "run_id": "01hq7k...",
  "cursor": "eyJsYXN0X2lkIjoxMjM0fQ==",
  "high_watermark": "&lt;updatedAt of newest committed record&gt;",
  "pages_committed": 47,
  "status": "IN_PROGRESS"
}</code></pre>



<p class="wp-block-paragraph">Overlap only works if downstream deduplication is real. Write raw pages with an ingest-time partition and a run identifier in the object key, then let the curated layer resolve duplicates by primary key and latest <code>updatedAt</code>. If you write straight into a merged table with no dedupe step, overlapping restarts will double-count and you&#8217;ll have traded silent loss for silent inflation.</p>



<h2 class="wp-block-heading">Failure four: the schema moved and nobody told you</h2>



<p class="wp-block-paragraph">GraphQL&#8217;s deprecation story is genuinely better than REST&#8217;s. Fields carry a <code>@deprecated</code> directive with a reason string, and introspection exposes it. The problem is that nothing forces you to look.</p>



<p class="wp-block-paragraph">The failure mode is a field that gets deprecated, stops being populated, then eventually gets removed. Stage one is invisible: your query still validates, the field returns null, and your warehouse fills with nulls. Stage two is loud but late: the query becomes invalid and the whole ingest breaks at 3am.</p>



<p class="wp-block-paragraph">Put an introspection diff in CI. Fetch the schema on a schedule, store it as an artifact, and diff against the last known good version. Fail the build on removals or type changes to fields your queries actually select, and warn on new deprecations. Tools in the Apollo and Hasura ecosystems ship schema-diff tooling for this, and rolling your own against the introspection result is not much work either. Either way, the point is that the check runs on a schedule rather than when the pager goes off.</p>



<p class="wp-block-paragraph">The complementary habit is to keep queries narrow. Every field you select is a field that can break you, a field that costs budget, and a field that widens your Glue table. Selecting a whole object because it was convenient during development is a liability you carry indefinitely.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Wiring the GraphQL data ingestion pipeline on AWS</h2>



<p class="wp-block-paragraph">With the failure families understood, the AWS assembly is mostly a set of trade-offs about where the fetch loop lives.</p>



<h3 class="wp-block-heading">Where the fetch loop runs</h3>



<p class="wp-block-paragraph">Step Functions can call an HTTPS endpoint directly with an HTTP Task, no Lambda involved. It uses an EventBridge connection to hold the credentials, which means your token lives in a Secrets Manager secret managed by EventBridge rather than in an environment variable:</p>



<pre class="wp-block-code"><code>"FetchPage": {
  "Type": "Task",
  "Resource": "arn:aws:states:::http:invoke",
  "Parameters": {
    "ApiEndpoint": "https://api.example.com/graphql",
    "Method": "POST",
    "InvocationConfig": {
      "ConnectionArn": "arn:aws:events:region:account-id:connection/name/id"
    },
    "Headers": { "Content-Type": "application/json" },
    "RequestBody": {
      "query.$": "$.query",
      "variables.$": "$.variables"
    }
  },
  "Retry": [{
    "ErrorEquals": [
      "States.Http.StatusCode.429",
      "States.Http.StatusCode.502",
      "States.Http.StatusCode.503",
      "States.Http.StatusCode.504"
    ],
    "IntervalSeconds": 1,
    "BackoffRate": 2,
    "MaxAttempts": 3,
    "JitterStrategy": "FULL"
  }],
  "Next": "InspectBody"
}</code></pre>



<p class="wp-block-paragraph">The execution role needs four permissions: <code>states:InvokeHTTPEndpoint</code>, <code>events:RetrieveConnectionCredentials</code>, and both <code>secretsmanager:GetSecretValue</code> and <code>secretsmanager:DescribeSecret</code> scoped to the EventBridge-managed secret.</p>



<p class="wp-block-paragraph">It&#8217;s a clean pattern, and I like it for small, high-value syncs. Two constraints decide whether it fits. HTTP Task requests time out after 60 seconds, which a deeply nested GraphQL query can exceed without being pathological. And Step Functions reads the response within its standard state payload ceiling of roughly 256 KB, so a page of a hundred rich records will not fit. Also note the retrier above: it fires on HTTP status codes, so it will not catch a throttle that arrives as a 200 with an error in the body. That check has to happen in the state after it.</p>



<p class="wp-block-paragraph">The alternative is a Lambda driver that paginates internally and writes pages straight to S3, returning only the cursor and a count to the state machine. This dodges both limits, gives you a real HTTP client with real backoff, and keeps large payloads out of the workflow&#8217;s state. It costs you per-page retry granularity, which you buy back with the checkpoint record.</p>



<p class="wp-block-paragraph">Distributed Map is worth mentioning for the fan-out case, such as one child execution per tenant or per shard. It gives you high concurrency and a per-item execution history, which is genuinely useful for debugging. Be aware of two things before committing: express child workflows cap at five minutes rather than Lambda&#8217;s fifteen, and practitioners have published benchmarks showing throughput well below what the headline concurrency numbers suggest for small, fast items. For a lot of ingestion work, SQS plus a DynamoDB job table remains the simpler answer.</p>



<p class="wp-block-paragraph">One case genuinely falls outside serverless: sources that require a long-lived session, a persistent websocket subscription, or a synchronisation window longer than fifteen minutes with no resumable cursor. A small always-on worker on a modest VPS from a provider like InterServer or Hetzner, feeding the same S3 landing zone, is less clever and more honest than contorting Lambda around the constraint.</p>



<h3 class="wp-block-heading">The landing zone</h3>



<p class="wp-block-paragraph">Land raw responses first, unmodified, including the <code>errors</code> and <code>extensions</code> keys. Storage is the cheapest thing in this architecture and the raw layer is what lets you reprocess after you discover a parsing bug six weeks later.</p>



<ul class="wp-block-list">
<li><strong>Partition by ingest time, not event time.</strong> Event-time partitioning requires reading the record before you know where to put it, which means you can&#8217;t write until you&#8217;ve parsed. Ingest-time keys let the raw write be dumb and fast.</li>

<li><strong>Put the run identifier in the key.</strong> When you need to unwind one bad run, being able to delete or ignore by prefix beats writing a surgical delete.</li>

<li><strong>Define Glue tables explicitly rather than crawling.</strong> You wrote the selection set, so you already know the schema. A crawler inferring types from JSON will surprise you the first time an optional field is absent from an entire page.</li>

<li><strong>Convert to columnar in the curated layer.</strong> Athena over raw JSON works and is fine for debugging. Parquet is what you want for anything anyone queries repeatedly.</li>
</ul>



<p class="wp-block-paragraph">If your source pushes rather than being polled, such as webhook deliveries carrying GraphQL payloads, Amazon Data Firehose (renamed from Kinesis Data Firehose, with APIs and IAM policies unchanged) will buffer, convert to Parquet and partition on the way to S3 without you running a consumer. For scheduled pull-based syncs it adds a hop you don&#8217;t need.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Runs succeed but a column is all null.</strong> Field errors. Pull a raw response from the landing zone and look at the <code>errors</code> array. The <code>path</code> tells you exactly which field.</li>

<li><strong>Runs succeed but ingest zero rows.</strong> Likely a throttle returned as a 200. Log the response body and the rate-limit headers on empty results, always.</li>

<li><strong>Intermittent failures partway through pagination.</strong> Cursor expiry. Fall back to the watermark and check whether the source documents a cursor lifetime.</li>

<li><strong>Query fails immediately with no <code>data</code> key.</strong> Request error. Validate against current introspection; something in the schema moved.</li>

<li><strong>Throttled despite staying under the documented limit.</strong> Either concurrent workers are sharing a bucket they each think they own, or you&#8217;re being charged requested cost on near-empty pages.</li>

<li><strong>HTTP Task fails on large pages.</strong> The response exceeds the state payload ceiling. Reduce page size or move the fetch into Lambda.</li>

<li><strong>Row counts drift upward over time.</strong> Overlap without deduplication. Check that the curated layer actually resolves on the primary key.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating a 200 as success and never reading the body.</li>

<li>Retrying request errors. They are deterministic; retrying just spends budget to fail identically.</li>

<li>Hardcoding bucket sizes and restore rates that differ per plan tier and change without notice.</li>

<li>Checkpointing the cursor before the data is durable.</li>

<li>Requesting the maximum page size on every call while being charged the worst-case cost for it.</li>

<li>Selecting entire objects because it was easier during development.</li>

<li>Alerting only on execution failure, so a job that succeeds while ingesting nothing never pages anyone.</li>

<li>Overlapping restart windows with no downstream deduplication.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Classify every response into clean, partial, or request-failed before anything downstream sees it.</li>

<li>Persist raw responses with <code>errors</code> and <code>extensions</code> intact, then parse from the raw layer.</li>

<li>Alarm on row-count deltas and partial-error rate, not just on execution status.</li>

<li>Read the throttle state the API hands you and back off before you hit the wall, not after.</li>

<li>Keep both a cursor and a watermark, and know which one is authoritative on restart.</li>

<li>Diff the schema on a schedule so deprecations reach you before they reach your data.</li>

<li>Hold credentials in Secrets Manager or an EventBridge connection, never in a Lambda environment variable.</li>

<li>Trace the pipeline end to end. CloudWatch covers the basics; Grafana or Datadog earn their keep once you&#8217;re correlating child executions across a fan-out.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can Step Functions call a GraphQL API without a Lambda function?</h3>



<p class="wp-block-paragraph">Yes. An HTTP Task using the <code>arn:aws:states:::http:invoke</code> resource posts your query directly, with an EventBridge connection handling authentication. It works well for small pages. The 60-second request timeout and the state payload size ceiling are what push most real ingestion work back into Lambda.</p>



<h3 class="wp-block-heading">How do I detect GraphQL errors when the status code is always 200?</h3>



<p class="wp-block-paragraph">Inspect the body. If there&#8217;s no <code>data</code> key, nothing executed and you have a request error. If <code>data</code> is present alongside a non-empty <code>errors</code> array, you have a partial response and the data is incomplete. Only a response with <code>data</code> and no errors is unambiguously clean.</p>



<h3 class="wp-block-heading">Does AWS AppSync fit into this?</h3>



<p class="wp-block-paragraph">Only if you&#8217;re on the serving side. AppSync is a managed GraphQL API for exposing your own data, so it&#8217;s what you&#8217;d reach for if you wanted clients to push events in via mutations. For pulling data out of somebody else&#8217;s GraphQL API, AppSync isn&#8217;t part of the picture.</p>



<h3 class="wp-block-heading">Should I store raw GraphQL responses or parsed records in S3?</h3>



<p class="wp-block-paragraph">Both, in separate layers. Raw responses are your audit trail and your reprocessing path, and they&#8217;re the only way to answer &#8220;was this field null in the source, or did my parser drop it?&#8221; weeks after the fact. Parsed and columnar output goes in a curated prefix that analysts query.</p>



<h3 class="wp-block-heading">How do I run an incremental sync from a GraphQL API?</h3>



<p class="wp-block-paragraph">Find the schema&#8217;s filter argument for a modification timestamp using introspection, pass your stored watermark into it, and page forward with cursors. Overlap the window slightly on each run and deduplicate on the primary key downstream. If the schema exposes no such filter, you&#8217;re limited to full pulls or webhook-driven change capture.</p>



<h3 class="wp-block-heading">Why am I throttled when I&#8217;m making very few requests?</h3>



<p class="wp-block-paragraph">Cost-based APIs charge for query complexity, not call count. A single deeply nested query pulling a connection inside a connection can consume a large fraction of the bucket by itself. Read the cost figures the API returns and flatten the query if the requested cost is consistently much higher than the actual cost.</p>



<h3 class="wp-block-heading">Is GraphQL a worse ingestion source than REST?</h3>



<p class="wp-block-paragraph">Different, not worse. You control the payload, which cuts transfer and lets you avoid the N+1 fan-out REST forces on nested data. Deprecation metadata is machine-readable, which REST rarely offers. The cost is that error signalling and rate limiting both moved into the body, so naive clients fail silently instead of loudly.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A GraphQL data ingestion pipeline on AWS fails differently from a REST one, and the difference is almost entirely about where the truth lives. In REST, the transport tells you what happened. In GraphQL, the transport tells you the request arrived, and the body tells you what happened. Every failure family in this post traces back to that: partial errors hidden behind a 200, throttles hidden behind a 200, cursors that fail by returning the wrong rows rather than an error.</p>



<p class="wp-block-paragraph">So build the pipeline so that a green run means something. Parse before you commit, alarm on row deltas rather than exit codes, and keep the raw responses so you can prove what the source actually said. Everything else is ordinary AWS plumbing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with a GraphQL ingestion pipeline?</h2>



<p class="wp-block-paragraph">This is the kind of work I take on regularly. Specifically:</p>



<ul class="wp-block-list">
<li>Auditing an existing GraphQL sync for silent partial-error loss and telling you which columns are already affected</li>

<li>Building the fetch, checkpoint and land loop in Step Functions and Lambda with cursors that survive restarts</li>

<li>Fitting a cost-aware throttle to whatever budget model your source actually uses, including multi-worker coordination</li>

<li>Designing the raw and curated S3 layout, Glue table definitions and Athena partitioning so queries stay cheap</li>

<li>Adding introspection diffing to CI so schema deprecations surface in a pull request, not in a dashboard</li>

<li>Wiring CloudWatch metrics and alarms on row-count deltas and partial-error rates, so a green run is actually green</li>
</ul>



<p class="wp-block-paragraph">Send me a run log, a state machine definition, or a raw response with an <code>errors</code> array in it, and I&#8217;ll tell you what it&#8217;s costing you.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/">Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Streaming Shopify Events into AWS Without Losing Orders</title>
		<link>https://john-nessime.com/blog/devops/streaming-shopify-events-into-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 16 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon SQS]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Dead Letter Queue]]></category>
		<category><![CDATA[DynamoDB]]></category>
		<category><![CDATA[Ecommerce Analytics]]></category>
		<category><![CDATA[Event-Driven Architecture]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Partner Event Source]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Reliability Engineering]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[Terraform]]></category>
		<category><![CDATA[Webhooks]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=221</guid>

					<description><![CDATA[<p>Wiring Shopify webhooks into Amazon EventBridge takes an afternoon. Keeping every order is the hard part. A walk through the five failure families that actually bite when streaming Shopify events into AWS: the partner source that silently drops everything, duplicate and out-of-order deliveries, rule patterns that match nothing, targets that fail without a dead-letter queue, and the 64 KB metering rule that quietly inflates the bill.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/streaming-shopify-events-into-aws/">Streaming Shopify Events into AWS Without Losing Orders</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The partner event source in the EventBridge console said <code>Pending</code>. It had said <code>Pending</code> for six days.</p>



<p class="wp-block-paragraph">Nobody noticed, because nothing errored. No 5xx in a log. No failed delivery in Shopify&#8217;s dashboard. No alarm. Shopify had been publishing order events the entire time, and AWS had been throwing every single one of them on the floor.</p>



<p class="wp-block-paragraph">That behaviour is documented, in one short note in the AWS docs: events published to a partner event source that has not been associated with an event bus are dropped immediately and are not persisted at rest. There is no retry for that. There is no buffer. The events are gone, and the only way to get the data back is to go ask the Shopify Admin API for it after the fact.</p>



<p class="wp-block-paragraph">That is the shape of most of the pain in this integration. Streaming Shopify events into AWS is easy to stand up and easy to get quietly wrong, and every one of the quiet failures looks identical from the outside: everything is green, and some of your data isn&#8217;t there.</p>



<p class="wp-block-paragraph">This post walks the five failure families that actually cost you records, plus the reconciliation layer that most teams only build after the first incident. It assumes you can read a rule pattern and an IAM policy. It does not assume you have shipped this before.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">What the pipe actually looks like</h2>



<p class="wp-block-paragraph">Four moving parts, and only two of them live in your account.</p>



<ol class="wp-block-list"><li>A Shopify app holds the webhook subscriptions. Each subscription has a topic and a delivery method. For this path the delivery method is EventBridge and the address is an ARN, not a URL.</li><li>Shopify creates a <strong>partner event source</strong> inside your AWS account, in the region you nominated.</li><li>You associate that source with a <strong>partner event bus</strong>. This is the step everyone forgets.</li><li>Rules on that bus match events and push them at targets: Lambda, SQS, Step Functions, Firehose, whatever fits.</li></ol>



<p class="wp-block-paragraph">The ARN trips people up more than anything else in the setup. Shopify wants the <em>event source</em> ARN, not the event bus ARN. They look similar and only one of them works:</p>



<pre class="wp-block-code"><code># Correct - the event source ARN. Note the empty account field.
arn:aws:events:eu-west-1::event-source/aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;

# Wrong - this is the bus, and Shopify will reject it
arn:aws:events:eu-west-1:123456789012:event-bus/aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;</code></pre>



<p class="wp-block-paragraph">Associating the source is a single call, and both the name and the source name are the same string:</p>



<pre class="wp-block-code"><code># Create the partner event bus that accepts the source
aws events create-event-bus 
  --name "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;" 
  --event-source-name "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;" 
  --region eu-west-1

# Confirm it flipped from PENDING to ACTIVE
aws events describe-event-source 
  --name "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;" 
  --region eu-west-1</code></pre>



<p class="wp-block-paragraph">The same architecture applies if you are not on Shopify. BigCommerce and commercetools both publish to EventBridge as partner sources, and the failure families below are identical because they come from EventBridge&#8217;s semantics, not the store&#8217;s.</p>



<h2 class="wp-block-heading">Failure family one: events that never existed</h2>



<p class="wp-block-paragraph">This is the one from the opening, and it is the most expensive because it is completely silent on both sides.</p>



<p class="wp-block-paragraph">Shopify considers the delivery successful. It handed the event to the partner source, which is its contract. AWS considers nothing to have happened, because an unassociated source has no bus to write to, and EventBridge does not persist events at rest before a bus exists. Your CloudWatch metrics show nothing, because metrics are emitted per bus and per rule, and you have neither.</p>



<p class="wp-block-paragraph">The same class of hole opens up in two other ways:</p>



<ul class="wp-block-list"><li><strong>Region mismatch.</strong> The source is created in the region you gave Shopify. Your bus, your rules, your targets and your dead-letter queues all have to be in that region. A rule in the right account but the wrong region matches nothing, forever, without complaint.</li><li><strong>Environment drift.</strong> A staging store pointed at a production source, or a source created against an account ID that belonged to an old sandbox. Nothing errors. Events just land somewhere you are not looking.</li></ul>



<p class="wp-block-paragraph">The fix is boring and it works: treat the source state as a monitored asset. A scheduled job that calls <code>describe-event-source</code> and alarms if <code>State</code> is anything other than <code>ACTIVE</code> costs you twenty minutes and covers the entire failure family. Put it next to your other synthetic checks, not inside the pipeline it is watching.</p>



<p class="wp-block-paragraph">The second half of that check is a heartbeat on volume. If a bus that normally sees a few thousand events a day sees zero for an hour, that is an incident even when every component reports healthy. Alarm on <code>MatchedEvents</code> hitting zero, not just on errors.</p>



<h2 class="wp-block-heading">Failure family two: events that arrive twice, or backwards</h2>



<p class="wp-block-paragraph">EventBridge is at-least-once. Shopify&#8217;s webhooks are at-least-once. Neither one promises ordering. Put those together and you get two distinct bugs that people usually try to fix with one patch.</p>



<p class="wp-block-paragraph">The duplicate is the obvious one. The same <code>orders/create</code> arrives twice, and if your handler posts to a fulfilment provider or sends a customer email, you have just done it twice. The dedupe key is sitting in the envelope: Shopify puts <code>X-Shopify-Webhook-Id</code> into <code>detail.metadata</code>, and it identifies the delivery. Write it into DynamoDB with a conditional put and a TTL of a few days, and drop the event if the write fails.</p>



<p class="wp-block-paragraph">The out-of-order case is the one that costs you money quietly. An <code>orders/updated</code> carrying a cancelled status arrives before the <code>orders/updated</code> carrying the address change, and your database ends up holding the older state because it was written last. Nothing failed. The row is just wrong, and it will stay wrong until someone complains.</p>



<p class="wp-block-paragraph">The envelope carries what you need for this too. <code>detail.metadata</code> includes <code>X-Shopify-Triggered-At</code>, and the resource in <code>detail.payload</code> carries its own <code>updated_at</code>. Compare before you write, and refuse to apply an update whose timestamp is older than the one already stored.</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow"><p>The dedupe key stops you from doing the work twice. The version check stops you from doing the work backwards. They solve different problems and you need both.</p></blockquote>



<p class="wp-block-paragraph">One thing you can skip on this path: HMAC verification. On the HTTPS delivery method you must verify the signature, because anyone can POST to your endpoint. On the EventBridge path, only the partner account behind the event source is permitted to publish to that bus, and the AWS docs are explicit that adding your own resource policy to a partner bus is rejected. The signature header still rides along in the metadata, but the trust boundary is enforced by AWS rather than by your code.</p>



<h2 class="wp-block-heading">Failure family three: rules that match nothing</h2>



<p class="wp-block-paragraph">Every Shopify event, regardless of topic, arrives with the same <code>detail-type</code>. That single fact invalidates the routing instinct most people bring from AWS service events.</p>



<pre class="wp-block-code"><code>{
  "version": "0",
  "id": "1b8e2e75-b771-e964-f0e6-fbca6a21dad8",
  "detail-type": "shopifyWebhook",
  "source": "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;",
  "account": "123456789012",
  "time": "2022-07-02T12:47:58Z",
  "region": "eu-west-1",
  "resources": [],
  "detail": {
    "payload": {
      "id": 1234567890,
      "title": "Columbia Las Hermosas"
    },
    "metadata": {
      "Content-Type": "application/json",
      "X-Shopify-Topic": "products/update",
      "X-Shopify-Shop-Domain": "example.myshopify.com",
      "X-Shopify-Hmac-SHA256": "...",
      "X-Shopify-Webhook-Id": "...",
      "X-Shopify-API-Version": "...",
      "X-Shopify-Triggered-At": "2022-07-02T12:47:57.989779121Z"
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Two things to take from that envelope. The resource body is nested under <code>detail.payload</code>, not at the top of <code>detail</code>, so a pattern copied from an HTTPS handler will match nothing. And the topic lives in <code>detail.metadata</code>, which is where all your routing has to happen.</p>



<pre class="wp-block-code"><code>// Exact topic match
{
  "detail-type": ["shopifyWebhook"],
  "detail": {
    "metadata": {
      "X-Shopify-Topic": ["orders/create"]
    }
  }
}

// Every orders topic, one rule
{
  "detail-type": ["shopifyWebhook"],
  "detail": {
    "metadata": {
      "X-Shopify-Topic": [{ "prefix": "orders/" }]
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Do not deploy a pattern you have not tested against a real envelope. <code>test-event-pattern</code> answers in a second and saves an afternoon:</p>



<pre class="wp-block-code"><code>aws events test-event-pattern 
  --event-pattern file://pattern.json 
  --event file://sample-event.json</code></pre>



<h3 class="wp-block-heading">One rule or thirty?</h3>



<p class="wp-block-paragraph">There is a real argument for a single catch-all rule that pushes everything into one queue and lets your consumer branch on the topic. It is less infrastructure, it deploys faster, and adding a topic does not require a Terraform run.</p>



<p class="wp-block-paragraph">What you give up is per-topic visibility. <code>MatchedEvents</code>, <code>FailedInvocations</code> and the dead-letter queue are all scoped to the rule. Collapse thirty topics into one rule and you can no longer tell that inventory events stopped three days ago, because the aggregate number still looks fine.</p>



<p class="wp-block-paragraph">The split I reach for first: a dedicated rule for each topic that touches money or fulfilment, and one catch-all for everything else. You get precise alarms where the cost of being wrong is high and low overhead everywhere else.</p>



<h2 class="wp-block-heading">Failure family four: events that arrive and die at the target</h2>



<p class="wp-block-paragraph">By default EventBridge keeps retrying a failed target invocation for up to a day, with exponential backoff and jitter. That is generous, and it is also the reason people assume they do not need a dead-letter queue. They do, for two reasons.</p>



<p class="wp-block-paragraph">First, a whole class of errors gets <em>no</em> retries at all. Missing permissions on the target, a target that no longer exists, an address that will not resolve. EventBridge does not retry those, because retrying cannot help. It sends them straight to the DLQ if one is configured, and drops them if one is not.</p>



<p class="wp-block-paragraph">Second, a day of retries is not much when the failure is a bad deploy discovered on a Friday evening.</p>



<pre class="wp-block-code"><code>aws events put-targets 
  --rule shopify-orders-create 
  --event-bus-name "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;" 
  --targets '[{
    "Id": "order-processor",
    "Arn": "arn:aws:lambda:eu-west-1:123456789012:function:order-processor",
    "RetryPolicy": {
      "MaximumRetryAttempts": 20,
      "MaximumEventAgeInSeconds": 3600
    },
    "DeadLetterConfig": {
      "Arn": "arn:aws:sqs:eu-west-1:123456789012:shopify-orders-dlq"
    }
  }]'</code></pre>



<p class="wp-block-paragraph">Lowering the retry window is deliberate here. Twenty-four hours of retries against a genuinely broken consumer buys you nothing and hides the problem; a shorter window pushes failures into the DLQ where they are visible and countable.</p>



<h3 class="wp-block-heading">The DLQ permission trap</h3>



<p class="wp-block-paragraph">This one catches almost everyone who manages infrastructure as code. Configure a DLQ through the console and AWS attaches the queue policy for you. Configure it through <code>PutTargets</code> — which is what Terraform, CloudFormation and the CLI all do — and you must attach it yourself. Miss it, and you have a dead-letter queue that cannot receive dead letters.</p>



<pre class="wp-block-code"><code>{
  "Sid": "Dead-letter queue permissions",
  "Effect": "Allow",
  "Principal": { "Service": "events.amazonaws.com" },
  "Action": "sqs:SendMessage",
  "Resource": "arn:aws:sqs:eu-west-1:123456789012:shopify-orders-dlq",
  "Condition": {
    "ArnEquals": {
      "aws:SourceArn": "arn:aws:events:eu-west-1:123456789012:rule/shopify-orders-create"
    }
  }
}</code></pre>



<p class="wp-block-paragraph">The metric that catches this is <code>InvocationsFailedToBeSentToDlq</code>. If it is ever non-zero, your safety net has a hole in it and events are being lost at the exact moment you were counting on it. Alarm on it at a threshold of one. It only reports when it is non-zero, so it costs nothing the rest of the time.</p>



<p class="wp-block-paragraph">Two more constraints worth knowing before you design around a DLQ: it must be a standard SQS queue, not FIFO, and it must live in the same region as the rule. Each message carries the error code, the exhausted retry condition, the retry count and both ARNs as message attributes, which is usually enough to triage without opening the payload.</p>



<h2 class="wp-block-heading">Failure family five: the bill</h2>



<p class="wp-block-paragraph">EventBridge does not meter one event as one event. It meters in 64 KB chunks, so an event larger than that bills as multiple events. Rates change and vary by region, so check the current pricing page rather than trusting any number you read in a blog post, but the mechanism is stable and it is what determines your bill.</p>



<p class="wp-block-paragraph">This matters more for commerce than for most event sources. A product update is small. An order with thirty line items, per-item discount allocations, tax lines, shipping lines, note attributes and a stack of metafields is not. Wholesale and subscription stores routinely produce order payloads that cross the chunk boundary, and the same order updated eight times through its lifecycle multiplies that.</p>



<p class="wp-block-paragraph">Three levers, roughly in order of how much they return:</p>



<ul class="wp-block-list"><li><strong>Trim at the subscription.</strong> Shopify&#8217;s webhook subscription API lets you restrict which fields are included in the payload and which metafield namespaces come along. Fields you never read cost you at ingestion, at archive and again at replay. This is the only lever that stops paying for the data before it enters AWS.</li><li><strong>Subscribe to fewer topics.</strong> Broad topics like <code>orders/updated</code> fire on changes you do not care about. If you only act on fulfilment state, subscribe to the fulfilment topics instead of filtering a firehose after you have paid for it.</li><li><strong>Archive selectively, and set retention.</strong> Archives bill for processing, for storage and again for replay. An archive with no retention period grows forever. Archive the topics you would genuinely replay and let the rest go.</li></ul>



<p class="wp-block-paragraph">One structural limit to design around: EventBridge caps the total size of a single event. A payload that exceeds it does not get truncated in a helpful way — the publish fails. Trimming at the subscription protects you here as well as on cost.</p>



<h2 class="wp-block-heading">The layer nobody builds until they need it</h2>



<p class="wp-block-paragraph">Archive and replay is genuinely useful, and it is also routinely misunderstood. Replay re-delivers events that <em>reached the bus</em>. It does nothing at all for the failure family at the top of this post, where the events never reached the bus in the first place. Replay fixes bugs in your consumer. It does not fix gaps in your ingestion.</p>



<p class="wp-block-paragraph">For that you need reconciliation: a scheduled job that queries the Shopify Admin API for resources changed since a stored watermark and compares them against what you hold. It is unglamorous, it is the thing that catches the outage you did not know about, and it is worth building before you need it rather than during the incident.</p>



<ul class="wp-block-list"><li>Run it hourly for orders and fulfilments, daily for products and customers. The cadence should track how expensive being wrong is, not how much data there is.</li><li>Store a watermark per topic and advance it only after a successful full pass. A partial pass that advances the watermark creates the exact gap you built the job to find.</li><li>Compare counts first, records second. A count mismatch is cheap to compute and tells you whether to bother with the expensive comparison.</li><li>Emit the drift as a metric, not just a log line. &#8220;Orders in Shopify but not in our store, last hour&#8221; is a graph worth putting on a dashboard, and it should normally read zero.</li></ul>



<p class="wp-block-paragraph">The reconciliation worker does not need to live in Lambda. It is a long, paginated, rate-limited crawl, which is an awkward fit for a function timeout and a comfortable fit for a small VPS you already run. If you have a box at InterServer or Hetzner sitting there for other jobs, a cron entry and a script is a perfectly respectable answer.</p>



<h2 class="wp-block-heading">Troubleshooting by symptom</h2>



<p class="wp-block-paragraph">Work these in order. Each one is cheap and rules out a whole branch.</p>



<h3 class="wp-block-heading">Nothing is arriving at all</h3>



<ol class="wp-block-list"><li>Run <code>describe-event-source</code>. If <code>State</code> is not <code>ACTIVE</code>, stop here. Everything published so far is gone and you need the reconciliation path.</li><li>Confirm the region of the bus matches the region in the source ARN.</li><li>List your webhook subscriptions through the Admin API and confirm the address is the event-source ARN, not the bus ARN.</li><li>Confirm the subscriptions belong to the app whose access token you are using. Registering with a token from a different app is a common and confusing dead end.</li><li>Check <code>MatchedEvents</code> on the bus with no rule dimension. Non-zero means events are landing and your rules are the problem, not the plumbing.</li></ol>



<h3 class="wp-block-heading">Some topics arrive, order or customer topics do not</h3>



<p class="wp-block-paragraph">This is almost always scopes rather than infrastructure. Order and customer topics sit behind protected customer data access, which is a separate approval in the app configuration on top of the read scopes. Without it, product events flow perfectly and order events silently do not — which looks exactly like a broken rule and is not.</p>



<h3 class="wp-block-heading">The rule matches but the target does nothing</h3>



<p class="wp-block-paragraph">Compare <code>MatchedEvents</code> against <code>SuccessfulInvocationAttempts</code> on the rule. A gap sends you to <code>FailedInvocations</code> and to the DLQ. Check the target&#8217;s resource policy, and check <code>InvocationsFailedToBeSentToDlq</code> before you trust that the DLQ is catching anything.</p>



<h3 class="wp-block-heading">Events arrive, but late</h3>



<p class="wp-block-paragraph">Look at <code>ThrottledRules</code> and at <code>IngestionToInvocationSuccessLatency</code>. Sustained throttling usually means an invocation quota rather than a rule problem, and it shows up first during flash sales, which is the worst possible time to discover it. Load-test the path before a peak event, not after.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list"><li>Creating the partner event source and never associating it with a bus. Silent, total, unrecoverable data loss for the whole window.</li><li>Registering the event bus ARN instead of the event source ARN, then debugging Shopify&#8217;s rejection for an hour.</li><li>Writing rule patterns against the resource shape from an HTTPS webhook, forgetting that the body sits under <code>detail.payload</code>.</li><li>Routing on <code>detail-type</code>. Every Shopify event carries the same one, so a pattern that matches on it alone matches everything.</li><li>Configuring a DLQ through Terraform without the queue policy, and only finding out when you needed it.</li><li>Assuming replay covers ingestion gaps. It replays what reached the bus and nothing else.</li><li>Deduplicating on the resource ID instead of the webhook ID, so legitimate subsequent updates get discarded as duplicates.</li><li>Skipping reconciliation because the pipeline &#8220;works&#8221;. It works right up until it doesn&#8217;t, and that is precisely when you need the other path.</li></ul>



<h2 class="wp-block-heading">Best practices for streaming Shopify events into AWS</h2>



<ul class="wp-block-list"><li>Alarm on the event source state and on <code>MatchedEvents</code> reaching zero. Absence of events is a signal, and it is the only signal you get for the worst failure.</li><li>Dedupe on <code>X-Shopify-Webhook-Id</code> and version-check on <code>X-Shopify-Triggered-At</code>. Two mechanisms, two problems.</li><li>Give every target a DLQ and a retry window you chose deliberately, rather than inheriting the default.</li><li>Keep dedicated rules for money and fulfilment topics so their metrics stay legible; batch the rest behind a catch-all.</li><li>Trim payloads at the Shopify subscription rather than in a Lambda. Filtering after ingestion means you already paid for the bytes.</li><li>Define the whole thing in Terraform or CloudFormation, including the queue policies. This stack has too many one-time console clicks to survive being hand-built twice.</li><li>Point your observability platform at the same bus. Datadog and New Relic are both EventBridge partners, so business events and infrastructure telemetry can share one pipeline instead of two.</li><li>Build reconciliation before your first peak trading period, not after your first missing-order ticket.</li></ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I still need to verify the HMAC signature on the EventBridge path?</h3>



<p class="wp-block-paragraph">No. Only the partner account behind the event source can publish to a partner event bus, and AWS actively rejects attempts to add your own resource policy granting anyone else access. The signature header is still present in the metadata, but the trust boundary is enforced by AWS rather than by your handler. On the HTTPS delivery method, verification remains mandatory.</p>



<h3 class="wp-block-heading">Should I use EventBridge or plain HTTPS webhooks?</h3>



<p class="wp-block-paragraph">HTTPS is simpler, works with any host, and is easier to debug because you can curl your own endpoint. It also puts you on the hook for absorbing burst traffic within a short response deadline, and for keeping the endpoint up well enough that Shopify does not remove the subscription after persistent failures. EventBridge moves that burst absorption to AWS and gives you native fan-out. If your consumers already live in AWS, the operational maths favours EventBridge. If they do not, a dedicated reliability layer such as Hookdeck in front of an HTTPS endpoint is a reasonable alternative and a much smaller change.</p>



<h3 class="wp-block-heading">Can I use one partner event source for multiple stores?</h3>



<p class="wp-block-paragraph">Events from every shop that installed your app flow through the source associated with that app, and the shop is identified by <code>X-Shopify-Shop-Domain</code> in the metadata. You can route per-shop with rule patterns matching that field. For genuine tenant isolation — separate accounts, separate blast radius — you want separate apps and separate sources, because a single bus is a single failure domain.</p>



<h3 class="wp-block-heading">Why do I get duplicate order events even though nothing failed?</h3>



<p class="wp-block-paragraph">Because at-least-once means exactly that. Duplicates are normal operation, not a fault to be investigated. Separately, an order genuinely does change several times shortly after creation — payment capture, risk assessment, post-purchase upsells — so several <code>orders/updated</code> events for one order are expected and are not duplicates at all. Deduplicate on the webhook ID to tell the two apart.</p>



<h3 class="wp-block-heading">What happens to events published while my consumer is broken?</h3>



<p class="wp-block-paragraph">They reach the bus, match your rules, and EventBridge retries the target within your configured window. Once that window is exhausted they go to the DLQ if you have one and are discarded if you do not. The events themselves are not lost at the bus level as long as the source is associated — this is the failure family you can actually engineer your way out of.</p>



<h3 class="wp-block-heading">Can I archive and replay Shopify events?</h3>



<p class="wp-block-paragraph">Yes, with an archive on the partner event bus and an event pattern controlling what gets archived. Budget for three separate charges — processing into the archive, storage while it sits there, and the replay itself — and always set an explicit retention period, because an archive without one grows indefinitely.</p>



<h3 class="wp-block-heading">Does this work the same way for BigCommerce or commercetools?</h3>



<p class="wp-block-paragraph">The AWS half is identical: partner source, association step, bus, rules, targets, and every failure family in this post. What differs is the envelope shape and how you register subscriptions on the vendor side. The association gap in particular bites the same way regardless of which platform is publishing.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Almost everything about streaming Shopify events into AWS degrades loudly. Targets throw errors, retries show up as metrics, dead letters pile up in a queue you can see. Those are the failures you will handle correctly, because they announce themselves.</p>



<p class="wp-block-paragraph">The one that will actually hurt you is the one that reports success on both sides while dropping every event on the floor. Association state and event volume are the two signals that catch it, and neither one appears on any dashboard by default. Add them on day one, before you write the first rule. Everything else in this post can be fixed after the fact; that one cannot.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a hand with your event pipeline?</h2>



<p class="wp-block-paragraph">Most of my work on this stack is either standing it up properly the first time or working out where records went after someone else stood it up. Things I can help with:</p>



<ul class="wp-block-list"><li>Building the Shopify-to-EventBridge path end to end in Terraform, including the queue policies and retry configuration that the console quietly does for you.</li><li>Auditing an existing pipeline for silent loss: source association, region drift, missing DLQ permissions, rules that have been matching nothing since the day they shipped.</li><li>Designing the idempotency and ordering layer — dedupe store, TTLs, version checks — so replays and duplicates stop corrupting downstream state.</li><li>Writing the reconciliation job against the Admin API, with watermarks, drift metrics and alarms that fire before a customer does.</li><li>Cutting EventBridge spend by trimming payloads at the subscription and rationalising archive retention, without losing anything you actually query.</li><li>Load-testing the whole path ahead of a peak trading period so throttling shows up in a test window rather than on the day.</li></ul>



<p class="wp-block-paragraph">If you have a rule pattern that isn&#8217;t matching, a DLQ that&#8217;s mysteriously empty, or a bill that grew faster than your order volume, send me the pattern, the metric graph or the line item and I&#8217;ll tell you what I&#8217;d look at first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/streaming-shopify-events-into-aws/">Streaming Shopify Events into AWS Without Losing Orders</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Shopify Sales Dashboard with AWS: Build One That Actually Reconciles</title>
		<link>https://john-nessime.com/blog/technical-guides/shopify-sales-dashboard-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 04 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Business Intelligence]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Ecommerce Analytics]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[Partition Projection]]></category>
		<category><![CDATA[QuickSight]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[SQL]]></category>
		<category><![CDATA[Webhooks]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=142</guid>

					<description><![CDATA[<p>Most Shopify dashboards built on AWS work perfectly for about three weeks, then quietly drift away from the numbers in the Shopify admin. Here is why that happens, and how to design the ingestion, storage and query layers so your totals still reconcile six months in.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/shopify-sales-dashboard-aws/">Shopify Sales Dashboard with AWS: Build One That Actually Reconciles</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone in the finance channel posts two screenshots side by side. On the left, the dashboard you built. On the right, the Shopify admin. The totals don&#8217;t match, and they&#8217;re not off by a rounding error either. They&#8217;re off by enough that nobody wants to use your dashboard for anything that matters.</p>



<p class="wp-block-paragraph">The frustrating part is that the pipeline is fine. Webhooks are arriving, Lambda is running clean, S3 has the files, Athena returns rows. Nothing is broken in the way monitoring understands &#8220;broken&#8221;. The pipeline is faithfully recording something that is no longer true.</p>



<p class="wp-block-paragraph">This post is about building a Shopify sales dashboard with AWS that survives that conversation. Not the wiring, which is well documented and mostly straightforward, but the design decisions that determine whether your numbers still hold up six months in. I&#8217;ll cover the three ingestion paths and when each one is the right call, why append-only pipelines drift, how to lay out S3 and Athena so recomputation is cheap, and what to do when the totals are already wrong.</p>



<h2 class="wp-block-heading">Why a Shopify sales dashboard with AWS drifts from the admin</h2>



<p class="wp-block-paragraph">Here&#8217;s the thing that catches almost everyone: <strong>a Shopify order is not an event, it&#8217;s a mutable record.</strong></p>



<p class="wp-block-paragraph">An event pipeline assumes facts are immutable once written. A payment happened. A shipment left. You append it, you never touch it again, and the sum of the log is the truth. That model is why streaming architectures are so clean, and it&#8217;s exactly wrong for order data.</p>



<p class="wp-block-paragraph">An order created on Monday can be edited on Tuesday, partially refunded on Friday, and fully refunded three weeks later. Every one of those changes belongs, financially, to Monday. If your pipeline appends the <code>orders/create</code> payload and never revisits it, Monday&#8217;s revenue is frozen at the moment of checkout and it will only ever be too high.</p>



<p class="wp-block-paragraph">This is the invisible failure. Nothing alerts. No queue backs up. Your dashboard is confidently wrong, and the gap widens roughly in proportion to your return rate. A store with a two percent return rate takes a long time to notice. A fashion store running thirty percent returns notices in about a month, usually via an angry accountant.</p>



<h3 class="wp-block-heading">The four adjustments that move historical numbers</h3>



<ul class="wp-block-list">
<li><strong>Refunds.</strong> Full or partial. A refund carries its own <code>created_at</code>, which is when the money moved back. The order it belongs to has a different, earlier date. You need both, and which one you attribute to depends on whether finance wants cash-basis or order-basis reporting. Ask before you build.</li>

<li><strong>Order edits.</strong> A merchant adds a line item or adjusts a quantity after the fact. The original payload is now stale. Shopify exposes both the original and the current totals precisely because of this.</li>

<li><strong>Cancellations.</strong> A cancelled order keeps existing in the API. If you filter only on payment status you will happily keep counting it.</li>

<li><strong>Test and draft orders.</strong> Test orders carry a flag marking them as such. Nobody remembers to filter these until a QA run during a quiet week produces a suspicious spike.</li>
</ul>



<p class="wp-block-paragraph">The design consequence is simple to state and annoying to implement: <strong>your pipeline must be able to recompute any past day.</strong> Every storage and partitioning decision below follows from that one requirement.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Getting data out of Shopify: three paths, three trade-offs</h2>



<p class="wp-block-paragraph">Before anything else: new Shopify apps are built on the GraphQL Admin API. The REST Admin API has been designated a legacy API and new public apps must use GraphQL. If you&#8217;re starting fresh, start there. If you inherited a REST integration, it probably still runs, but you&#8217;re on borrowed time and you should plan the migration rather than discover the deadline.</p>



<h3 class="wp-block-heading">Path 1: EventBridge partner event source</h3>



<p class="wp-block-paragraph">Shopify can deliver webhooks straight into an Amazon EventBridge partner event bus in your account. No public endpoint, no API Gateway, no HMAC verification code, because verification only applies to HTTPS deliveries. Shopify&#8217;s own docs confirm EventBridge and Pub/Sub deliveries skip it.</p>



<p class="wp-block-paragraph">You create the source in the Shopify app configuration using your AWS account ID, region and a source name, then associate it with an event bus in the EventBridge console and write rules to route it. The address you register with Shopify is the <em>partner event source</em> ARN, not the event bus ARN. That distinction accounts for a large share of the &#8220;I set it up and nothing arrives&#8221; threads on the Shopify forums.</p>



<p class="wp-block-paragraph">A rule matching everything from the Shopify partner source looks like this. Start broad, then narrow once you&#8217;ve seen the real shape of an event:</p>



<pre class="wp-block-code"><code>{
  "source": [ { "prefix": "aws.partner/shopify.com" } ]
}</code></pre>



<p class="wp-block-paragraph">Send that to an SQS queue with a dead-letter queue attached rather than straight to Lambda. Buffering gives you a replay buffer when a downstream deploy goes wrong, and the DLQ means a bad payload parks itself instead of poisoning the whole rule. This is the path I reach for first for anything already on AWS.</p>



<h3 class="wp-block-heading">Path 2: HTTPS webhooks into API Gateway and Lambda</h3>



<p class="wp-block-paragraph">The conventional route, and the right one if you need webhook delivery outside AWS too, or you want the payloads to pass through something you fully control. The cost is that you now own an internet-facing endpoint and the HMAC verification on it.</p>



<p class="wp-block-paragraph">Verify against the <strong>raw request body</strong>, before any JSON parsing. Re-serialising the payload changes byte-for-byte content and the signature will never match. Use a constant-time comparison so the check doesn&#8217;t leak timing information:</p>



<pre class="wp-block-code"><code>import base64, hashlib, hmac

def verify(raw_body: bytes, header_hmac: str, secret: str) -&gt; bool:
    digest = hmac.new(secret.encode(), raw_body, hashlib.sha256).digest()
    computed = base64.b64encode(digest).decode()
    return hmac.compare_digest(computed, header_hmac)</code></pre>



<p class="wp-block-paragraph">Shopify sends the signature in the <code>X-Shopify-Hmac-SHA256</code> header, base64-encoded, computed with your app&#8217;s client secret over the raw body. Store that secret in Secrets Manager or as an SSM SecureString parameter, not in a Lambda environment variable.</p>



<p class="wp-block-paragraph">Acknowledge fast. Shopify&#8217;s timeout is short and it retries with backoff over a finite window, so a handler that does real work inline will generate a wall of duplicate deliveries during a flash sale, exactly when you can least afford it. Return 2xx immediately, do the work asynchronously.</p>



<h3 class="wp-block-heading">Path 3: scheduled GraphQL bulk pull</h3>



<p class="wp-block-paragraph">Webhooks give you low latency. They do not give you completeness. Anything that fails past its retry window is gone, and Shopify will eventually remove a subscription that keeps failing. That&#8217;s a silent data loss mode with no local symptom at all.</p>



<p class="wp-block-paragraph">So run a scheduled reconciliation pull alongside the stream. Shopify&#8217;s GraphQL bulk operations are built for this: you submit a query, it runs asynchronously, and you fetch a JSONL result file when it finishes. That&#8217;s the right tool for backfills and nightly catch-up, rather than paginating thousands of pages against a points-based rate limiter and getting throttled halfway through.</p>



<p class="wp-block-paragraph">A nightly job that re-pulls the last seven to fourteen days and overwrites those partitions costs almost nothing and quietly fixes every category of drift described above. If you build one thing from this post, build that.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Decide what &#8220;revenue&#8221; means before you write a line of SQL</h2>



<p class="wp-block-paragraph">This is where most reconciliation arguments actually live, and it isn&#8217;t an engineering problem at all until you&#8217;ve had the conversation.</p>



<p class="wp-block-paragraph">Shopify&#8217;s own sales reporting builds total sales from gross sales, minus discounts, minus returns, plus taxes and shipping. Gift card sales sit outside that in a separate finance report. If your dashboard sums order totals and calls it revenue, you have built a different metric with the same name, and it will disagree with the admin forever no matter how good your pipeline is.</p>



<p class="wp-block-paragraph">Write the definition down. Put it in the dashboard as a tooltip. When someone challenges a number, you want the argument to be about the definition, not about whether your infrastructure works.</p>



<h3 class="wp-block-heading">The currency trap</h3>



<p class="wp-block-paragraph">If the store sells in more than one currency, the money fields split in two. Shopify exposes totals as a set containing both <code>shop_money</code> and <code>presentment_money</code>: the amount in the store&#8217;s base currency, and the amount the customer actually saw and paid.</p>



<p class="wp-block-paragraph">Sum the presentment amounts across a multi-currency store and you get a number with no meaning at all, euros and yen added together as if they were the same unit. For a single reporting figure you want the shop-currency side. Keep the presentment amount and its currency code in the table anyway, because the day someone asks &#8220;how much did we actually sell in Germany&#8221;, you&#8217;ll want it and it is painful to backfill.</p>



<p class="wp-block-paragraph">One caveat worth knowing: orders created through the API rather than through checkout can behave differently from native multi-currency checkout orders. If your store takes orders from an ERP or a marketplace integration, spot-check a few of those specifically.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Storage layout: partition by order date, never by arrival date</h2>



<p class="wp-block-paragraph">Two layers in S3. Keep them separate and keep them honest about what they are.</p>



<ol class="wp-block-list">
<li><strong>Raw.</strong> Every payload exactly as received, partitioned by ingestion date. Append-only, never edited. This is your audit trail and your rebuild source. Lifecycle it to a colder storage class after a few months, don&#8217;t delete it.</li>

<li><strong>Curated.</strong> One row per order representing current state, in Parquet, partitioned by <em>order date</em>. This is what the dashboard queries. It is derived, disposable and rewritable.</li>
</ol>



<p class="wp-block-paragraph">The partitioning choice on the curated layer is the load-bearing decision in the whole design. If you partition by arrival date, which is what Amazon Data Firehose does by default because it buckets on the moment it writes the file, then a refund that arrives three weeks late lands in today&#8217;s partition. Correcting Monday now means finding and rewriting fragments scattered across twenty other partitions. Partitioned by order date, correcting Monday means overwriting exactly one prefix.</p>



<p class="wp-block-paragraph">Firehose can do this with dynamic partitioning, which routes records by keys inside the payload rather than by write time. If you&#8217;re not using Firehose, extract the order date in your Lambda and write the prefix yourself.</p>



<h3 class="wp-block-heading">Use partition projection so Athena stops guessing</h3>



<p class="wp-block-paragraph">The default Glue Data Catalog approach means running a crawler or issuing <code>MSCK REPAIR TABLE</code> to register new partitions. Forget one and you get a query that silently returns nothing for recent days. Nobody notices until Monday.</p>



<p class="wp-block-paragraph">Partition projection removes the metastore lookup entirely. You tell Athena the shape of the partition keys and it calculates the prefixes at query time:</p>



<pre class="wp-block-code"><code>CREATE EXTERNAL TABLE shop_orders (
  order_id             bigint,
  order_number         string,
  created_at           timestamp,
  financial_status     string,
  cancelled_at         timestamp,
  is_test              boolean,
  total_shop           decimal(12,2),
  shop_currency        string,
  total_presentment    decimal(12,2),
  presentment_currency string
)
PARTITIONED BY (order_date string)
STORED AS PARQUET
LOCATION 's3://your-bucket/curated/orders/'
TBLPROPERTIES (
  'projection.enabled' = 'true',
  'projection.order_date.type' = 'date',
  'projection.order_date.format' = 'yyyy-MM-dd',
  'projection.order_date.range' = '2019-01-01,NOW',
  'projection.order_date.interval' = '1',
  'projection.order_date.interval.unit' = 'DAYS',
  'storage.location.template' =
    's3://your-bucket/curated/orders/order_date=${order_date}/'
);</code></pre>



<p class="wp-block-paragraph">Set the range start to your store&#8217;s actual first order month. Projection generates every prefix in the range, so a range starting a decade too early makes wide scans slower for no benefit.</p>



<p class="wp-block-paragraph">Parquet matters here for the same reason. Athena bills on bytes scanned, so a columnar format with good compression cuts the bill directly, and a dashboard that only ever selects six columns from a forty-column table never touches the rest.</p>



<h3 class="wp-block-heading">Net sales in one query</h3>



<p class="wp-block-paragraph">With refunds in their own table keyed by order and carrying their own date, attributing them back to the original order day is a left join and a subtraction:</p>



<pre class="wp-block-code"><code>SELECT
    o.order_date,
    SUM(o.total_shop)                                AS gross_shop,
    SUM(COALESCE(r.refunded_shop, 0))                AS refunded_shop,
    SUM(o.total_shop - COALESCE(r.refunded_shop, 0)) AS net_shop
FROM shop_orders o
LEFT JOIN (
    SELECT order_id, SUM(amount_shop) AS refunded_shop
    FROM shop_refunds
    GROUP BY order_id
) r ON r.order_id = o.order_id
WHERE o.order_date BETWEEN '2025-01-01' AND '2025-01-31'
  AND o.is_test = false
  AND o.cancelled_at IS NULL
GROUP BY o.order_date
ORDER BY o.order_date;</code></pre>



<p class="wp-block-paragraph">Note the two filters doing quiet work at the bottom. Those two lines are the difference between a number finance accepts and a number they don&#8217;t.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Choosing the dashboard layer</h2>



<p class="wp-block-paragraph">Once the data is correct, this part is genuinely a preference. All of these work.</p>



<ul class="wp-block-list">
<li><strong>Amazon QuickSight</strong>, now delivered as part of Amazon Quick Suite, is the least-friction option if you&#8217;re already in AWS. Its in-memory SPICE layer means viewers aren&#8217;t firing an Athena query per chart interaction, which controls both latency and scan cost. Per-viewer pricing tends to be the deciding factor either way, so model it for your actual audience size before committing.</li>

<li><strong>Grafana</strong> with the Athena data source is a good fit if you&#8217;re already running Grafana for infrastructure and want commercial and operational panels on one screen. Grafana Cloud removes the hosting question if you&#8217;d rather not run it.</li>

<li><strong>Power BI</strong> makes sense when the finance team already lives in Microsoft 365 and models in DAX. The cross-cloud hop is real but manageable.</li>

<li><strong>Metabase</strong> or a self-hosted alternative on a small VPS from a provider like InterServer or Hetzner is the pragmatic answer for a handful of internal viewers, where per-seat BI licensing costs more than the entire pipeline.</li>
</ul>



<p class="wp-block-paragraph">The honest trade-off: managed BI costs more per month and saves you from becoming the person who patches the reporting server. Self-hosting inverts that. Neither is wrong, but pick deliberately rather than by inertia.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting: symptom to cause</h2>



<h3 class="wp-block-heading">Totals are consistently higher than the Shopify admin</h3>



<p class="wp-block-paragraph">Almost always refunds, cancellations or test orders. Check in that order. If the gap grows with the age of the reporting window, it&#8217;s refunds. If it&#8217;s a fixed offset on specific days, look for test orders or a QA run.</p>



<h3 class="wp-block-heading">Totals are lower, and recent days are missing rows</h3>



<p class="wp-block-paragraph">Either partitions aren&#8217;t registered, which projection fixes permanently, or the webhook subscription has been dropped after repeated delivery failures. Check the subscription still exists before you go digging through Lambda logs. A nightly bulk pull would have masked this, which is another argument for having one.</p>



<h3 class="wp-block-heading">Orders appear twice</h3>



<p class="wp-block-paragraph">Shopify&#8217;s delivery model is at-least-once, not exactly-once, and you may also have more than one subscription on the same topic. Deduplicate on the delivery ID header before you touch anything else, and make the write itself idempotent so a duplicate is a no-op rather than a second row.</p>



<h3 class="wp-block-heading">Numbers are right on the daily view, wrong on the monthly</h3>



<p class="wp-block-paragraph">Timezone. Order timestamps carry an offset; your partition key is a date string. If you derive the date in UTC and the store reports in a local timezone, orders near midnight land on the wrong day. That averages out over a month, which is exactly why the discrepancy hides until month boundaries.</p>



<h3 class="wp-block-heading">Athena costs jumped without more data</h3>



<p class="wp-block-paragraph">Someone built a dashboard with a filter that doesn&#8217;t hit the partition column, so every panel refresh scans the full table. Look at bytes scanned per query and check whether the BI tool is caching results or re-querying on every interaction.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating orders as immutable events and never revisiting a past day.</li>

<li>Partitioning on arrival time because that&#8217;s the default, then discovering corrections are expensive.</li>

<li>Verifying the HMAC against a re-serialised body instead of the raw bytes.</li>

<li>Doing real work inside the webhook handler, generating duplicates under load.</li>

<li>Summing presentment amounts across currencies.</li>

<li>Registering the event bus ARN with Shopify instead of the partner event source ARN.</li>

<li>Relying on webhooks alone with no scheduled reconciliation.</li>

<li>Shipping a &#8220;revenue&#8221; number without ever defining what it includes.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Keep raw and curated layers separate. Raw is append-only; curated is rewritable.</li>

<li>Make every partition idempotently rebuildable from raw. Test that path deliberately, before you need it.</li>

<li>Buffer through SQS with a dead-letter queue. Free replay, free isolation of bad payloads.</li>

<li>Run a nightly bulk pull over a rolling window and overwrite those partitions.</li>

<li>Use partition projection. It removes an entire category of silent failure.</li>

<li>Alarm on the absence of events, not just on errors. A CloudWatch alarm on zero orders processed in an hour during business hours catches broken subscriptions the same day.</li>

<li>Store the API secret in Secrets Manager and scope the Lambda role to the exact prefixes it writes.</li>

<li>Publish a reconciliation panel comparing your total to the admin&#8217;s for the same window. Surfacing the gap builds more trust than hiding it.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need a data warehouse, or is S3 and Athena enough?</h3>



<p class="wp-block-paragraph">For a single store&#8217;s order data, S3 with Athena is almost certainly enough, and it&#8217;s cheaper because you pay per query rather than for a running cluster. Redshift starts to earn its place when you&#8217;re joining Shopify data against several other large sources, or when concurrent query load makes Athena&#8217;s queue times noticeable.</p>



<h3 class="wp-block-heading">How near-real-time can this be?</h3>



<p class="wp-block-paragraph">Events land within seconds. The practical floor is your buffering window, since writing one tiny file per order gives you a small-files problem that ruins query performance. A few minutes of buffering is the usual compromise. If you genuinely need sub-minute order counts, put a live counter in DynamoDB alongside the analytical pipeline rather than trying to make the data lake do both jobs.</p>



<h3 class="wp-block-heading">Which webhook topics should I subscribe to?</h3>



<p class="wp-block-paragraph">At minimum, order creation, order update, order cancellation and refund creation. Update and refund topics are the ones people skip, and they&#8217;re exactly the ones carrying the corrections. Subscribe to fewer topics than you think you need and add rather than subscribing to everything, since every extra topic is volume you pay to store and process.</p>



<h3 class="wp-block-heading">Can I skip AWS and use a connector tool?</h3>



<p class="wp-block-paragraph">Yes, and for many stores that&#8217;s the right answer. A managed connector into a hosted warehouse gets you a working dashboard in an afternoon. You&#8217;re paying a monthly fee to avoid owning any of this, and trading away control over the data model. Building it on AWS wins when you need Shopify data joined to systems the connector doesn&#8217;t cover, or when row-based connector pricing outgrows the infrastructure cost.</p>



<h3 class="wp-block-heading">How do I backfill historical orders?</h3>



<p class="wp-block-paragraph">Use a GraphQL bulk operation rather than paginating the API. Submit the query, poll for completion, then stream the JSONL result into your raw bucket and run the same transformation your live pipeline uses. If backfill and live processing use different code paths, they will diverge, and you&#8217;ll spend an afternoon working out which one is lying.</p>



<h3 class="wp-block-heading">What does a setup like this cost to run?</h3>



<p class="wp-block-paragraph">For a typical single store, the pipeline itself is small money: Lambda invocations, a few gigabytes in S3, and Athena billed on bytes scanned, which partitioning and Parquet keep low. The BI seats are usually the largest line item, which is why the dashboard layer decision deserves more thought than the ingestion one. Model it against current published rates rather than trusting any figure you read in a blog post, including this one.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">A Shopify sales dashboard with AWS doesn&#8217;t fail because the pipeline breaks. It fails because the pipeline keeps working perfectly on data that has since changed underneath it.</p>



<p class="wp-block-paragraph">Design for correction from the first commit. Partition by order date, keep the raw layer so you can always rebuild, run a scheduled pull to catch what the stream missed, and agree on what revenue means before anyone builds a chart. Do that and the Monday morning screenshot comparison becomes a non-event, which is the highest praise a reporting pipeline ever gets.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with your Shopify data pipeline on AWS?</h2>



<p class="wp-block-paragraph">I design and build ecommerce data pipelines and reporting stacks on AWS. Typical engagements look like:</p>



<ul class="wp-block-list">
<li>Working out why an existing Shopify dashboard disagrees with the admin, and fixing the root cause rather than patching the query</li>

<li>Building the ingestion layer end to end: EventBridge or API Gateway, Lambda, SQS with dead-letter handling, and a scheduled GraphQL bulk reconciliation job</li>

<li>Designing the S3 layout, Glue schema and Athena tables so past days can be recomputed cheaply and partitions never go missing</li>

<li>Migrating REST Admin API integrations to GraphQL before the deadline forces the issue</li>

<li>Building the dashboard itself in QuickSight, Grafana or Metabase, including the metric definitions finance will actually sign off on</li>

<li>Cutting Athena scan costs and BI licensing on a reporting stack that has grown more expensive than anyone planned</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;re in the middle of one of these, send me the actual thing: the Athena query, the S3 prefix layout, the two totals that don&#8217;t match. It&#8217;s a much faster conversation than describing it in the abstract.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/shopify-sales-dashboard-aws/">Shopify Sales Dashboard with AWS: Build One That Actually Reconciles</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
