<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>EventBridge | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/eventbridge/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/eventbridge/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sun, 20 Sep 2026 11:01:58 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>EventBridge | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/eventbridge/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Amazon Macie PII Detection: The Buckets It Never Opened</title>
		<link>https://john-nessime.com/blog/cloud-computing/amazon-macie-pii-detection/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 06 Sep 2026 06:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Amazon Data Firehose]]></category>
		<category><![CDATA[Amazon Macie]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Classification]]></category>
		<category><![CDATA[Data Governance]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Encryption]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[PII Redaction]]></category>
		<category><![CDATA[S3 Lifecycle Rules]]></category>
		<category><![CDATA[Security Hub]]></category>
		<category><![CDATA[Sensitive Data Discovery]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=405</guid>

					<description><![CDATA[<p>A Macie bucket labeled "Not sensitive" often just means Macie never read it. Extensionless objects, unsupported storage classes, unreachable KMS keys and quota truncation all produce silence that looks identical to a clean result. Here's how to measure coverage, fix the four gaps, tune identifiers, and keep the bill honest.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/amazon-macie-pii-detection/">Amazon Macie PII Detection: The Buckets It Never Opened</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The email came from legal, not from security. A customer had exercised a data access request, someone pulled the export by hand, and it contained email addresses and phone numbers sitting in a bucket Macie had labeled <em>Not sensitive</em>. Nobody had done anything wrong. The console was telling the truth as it understood it. It just didn&#8217;t understand very much about that bucket.</p>



<p class="wp-block-paragraph">That is the failure mode worth internalizing before anything else: Amazon Macie PII detection reports on what it managed to read, not on what is there. Objects it never opened produce no findings, and no findings looks identical to a clean result. The gap between &#8220;we scanned this&#8221; and &#8220;we found nothing&#8221; is where most Macie deployments quietly fail their first audit.</p>



<p class="wp-block-paragraph">This post covers where that gap comes from and how to close it: the eligibility chain every object goes through, the four distinct reasons an object gets skipped, how to tune identifiers so the findings are worth reading, what actually drives the bill, and how to route results somewhere a human will act on them.</p>



<h2 class="wp-block-heading">The eligibility chain behind Amazon Macie PII detection</h2>



<p class="wp-block-paragraph">Every object passes the same gates before a byte gets inspected. The order matters, because each gate fails differently and needs a different fix.</p>



<ol class="wp-block-list">
<li>Is it in a general purpose S3 bucket? Directory buckets are out of scope entirely.</li>



<li>Is the storage class supported? Standard, Standard-IA, One Zone-IA, Intelligent-Tiering, Glacier Instant Retrieval and Reduced Redundancy are in. Glacier Deep Archive and S3 Express One Zone are not.</li>



<li>Does the object key carry a recognized file extension? This is a string check on the name, not a content sniff.</li>



<li>Can Macie retrieve and decrypt it, given the bucket policy, object ACL and encryption key?</li>



<li>Does the content parse, and stay inside the per-file quotas?</li>



<li>Do the active data identifiers match anything in it?</li>
</ol>



<p class="wp-block-paragraph">Only the last gate produces a finding. Everything before it produces silence. Macie does record why each object was skipped, but that lives in coverage data and object samples, not on the findings page most people look at.</p>



<p class="wp-block-paragraph">One detail explains a lot of confusion: when you first enable automated sensitive data discovery, every bucket gets a sensitivity score of 50 and the label <em>Not yet analyzed</em>. A bucket whose permissions block Macie stays there permanently. It never turns red. It sits mid-list, looking unremarkable next to buckets that genuinely were analyzed.</p>



<h2 class="wp-block-heading">Gap one: objects Macie never opened</h2>



<p class="wp-block-paragraph">Macie calls these <em>unclassifiable</em>. They fail the storage class or extension check, so no retrieval is attempted. This is the largest source of false confidence I&#8217;ve seen, and the cheapest to fix.</p>



<h3 class="wp-block-heading">Missing file extensions</h3>



<p class="wp-block-paragraph">Classifiability comes from the file name extension. A file full of customer records named <code>part-00003-a4f9</code> is invisible. The same bytes named <code>part-00003-a4f9.json</code> get inspected and produce findings.</p>



<p class="wp-block-paragraph">This bites hardest on streaming ingestion. Amazon Data Firehose writes to S3 without appending an extension unless compression, format conversion, or the explicit file extension setting adds one. An uncompressed JSON stream lands as a tree of extensionless objects. Athena and Glue read them fine, because they infer format from the table definition. Macie skips every one.</p>



<p class="wp-block-paragraph">Fix it upstream: set the S3 file extension on the delivery configuration, or enable GZIP compression or Parquet conversion, both of which append a recognized extension. If you can&#8217;t change the producer, the fallback is a copy-and-rename step into a scan prefix, which costs you storage plus a pipeline to maintain. Fixing the producer is almost always cheaper.</p>



<h3 class="wp-block-heading">Unsupported storage classes</h3>



<p class="wp-block-paragraph">A lifecycle rule transitioning old exports to Glacier Deep Archive quietly removes them from scope. That may be the right storage decision, but it needs to be a conscious one, because archived customer data is still customer data when a regulator asks.</p>



<p class="wp-block-paragraph">Macie exposes these counts per bucket, split by cause. This is the first query I run against a new account:</p>



<pre class="wp-block-code"><code># Buckets where Macie can't classify objects, broken down by cause
aws macie2 describe-buckets 
  --query 'buckets[?unclassifiableObjectCount.total &gt; `0`].{
      bucket: bucketName,
      classifiable: classifiableObjectCount,
      skipped_total: unclassifiableObjectCount.total,
      skipped_extension: unclassifiableObjectCount.fileType,
      skipped_storage_class: unclassifiableObjectCount.storageClass
    }' 
  --output table</code></pre>



<p class="wp-block-paragraph">Read the ratio, not the raw number. Forty thousand classifiable objects and twelve skipped is fine. Three classifiable and four hundred thousand skipped is a reporting artifact pretending to be a scan result, and that bucket&#8217;s sensitivity score means nothing.</p>



<h2 class="wp-block-heading">Gap two: objects it opened and couldn&#8217;t read</h2>



<p class="wp-block-paragraph">These are classification errors rather than eligibility failures. Macie selected the object, tried to fetch it, and was refused. Three causes, and they need three different people to fix them.</p>



<ul class="wp-block-list">
<li><strong>Customer-provided keys (SSE-C).</strong> Macie cannot supply the key material, so it cannot retrieve the object. No permissions fix exists. Re-encrypt with S3 managed or KMS keys if you want coverage.</li>



<li><strong>KMS key policy.</strong> For customer managed keys, the key policy must allow the Macie service-linked role, <code>AWSServiceRoleForAmazonMacie</code>, to decrypt. Cross-account buckets need the key owner to grant it, not the bucket owner. AWS publishes a permission analyzer script in the <code>aws-samples/amazon-macie-scripts</code> repository that enumerates every key Macie needs and generates the CLI commands to fix them.</li>



<li><strong>Restrictive bucket policies.</strong> An explicit <code>Deny</code> conditioned on source IP or VPC endpoint blocks Macie along with everything else. The working pattern excludes the service-linked role ARN from the deny using the <code>aws:PrincipalArn</code> condition key.</li>
</ul>



<p class="wp-block-paragraph">To see which objects Macie actually touched in a bucket, pull the object samples. It&#8217;s the closest thing to a scan log you get:</p>



<pre class="wp-block-code"><code># Objects automated discovery selected, with per-object status
aws macie2 list-resource-profile-artifacts 
  --resource-arn arn:aws:s3:::your-bucket-name</code></pre>



<p class="wp-block-paragraph">Anything with a status of <code>SKIPPED</code> is a lead. Take the key, check its encryption settings in S3, and you usually have your answer inside a minute.</p>



<h2 class="wp-block-heading">Gap three: it read the file and nothing matched</h2>



<p class="wp-block-paragraph">Macie ships managed data identifiers covering common PII, financial data and credentials across many countries. Automated discovery uses a recommended subset by default rather than all of them. Sensible, but it is a default, and it will miss things that matter to you.</p>



<p class="wp-block-paragraph"><strong>Add the managed identifiers for your actual jurisdictions.</strong> If you hold records for customers in a country whose national ID identifier isn&#8217;t in the recommended set, you&#8217;re scanning for the wrong things with perfect efficiency. Pull the current list with <code>aws macie2 list-managed-data-identifiers</code> and compare it against where your customers live.</p>



<p class="wp-block-paragraph"><strong>Write custom identifiers for what only you know is sensitive.</strong> Internal account numbers, case references, employee IDs. A custom identifier is a regex plus optional keywords and a proximity rule, and the keywords are what save you. A bare <code>[0-9]{8}</code> matches timestamps, order totals and row counts, and you drown. The same regex with keywords and a match distance only fires when the number sits near a word that gives it meaning.</p>



<p class="wp-block-paragraph">Test the pattern before it goes near a job. This runs the criteria against sample text and returns match counts without creating anything:</p>



<pre class="wp-block-code"><code># Dry-run a custom identifier before creating it
aws macie2 test-custom-data-identifier 
  --regex 'ACC-[0-9]{4}-[0-9]{4}' 
  --keywords account customer acct 
  --maximum-match-distance 30 
  --sample-text 'customer account ACC-4821-9930 updated; total 1234-5678'</code></pre>



<p class="wp-block-paragraph">Run it against real samples, including the awkward ones, then against a file you know is clean and confirm zero. A pattern you only tested on positives is a pattern you haven&#8217;t tested.</p>



<p class="wp-block-paragraph">The mirror image is allow lists, which tell Macie to ignore specific text or patterns: your support inbox address, your published press contact, the seeded test records in everyone&#8217;s staging fixtures. Without them the same non-issues appear in every scan until the team stops reading findings. Allow lists accept predefined text or regex, and the per-account and per-job quotas are tight enough that you should curate rather than dump.</p>



<h2 class="wp-block-heading">Gap four: quota truncation inside large files</h2>



<p class="wp-block-paragraph">Subtle, because the object <em>is</em> analyzed. Just not completely, and a partial result reports like a full one.</p>



<ul class="wp-block-list">
<li>Per-file size quotas differ by format. Exceed the quota for a type and Macie analyzes none of that file, not part of it.</li>



<li>Archives have separate ceilings for nesting depth, extracted bytes and file count. If the metadata shows a breach up front, nothing is extracted. Cross a limit partway through and Macie stops, reporting only what it processed.</li>



<li>JSON and JSON Lines have a nested depth limit. Past it, the file is not analyzed at all.</li>



<li>Full names and mailing addresses cap out per file. After the cap Macie stops counting and stops reporting locations for that type, so the occurrence count is a floor, never a total.</li>
</ul>



<p class="wp-block-paragraph">Findings carry a status reason naming the limit that was hit, with values like <code>FILE_EXCEEDS_SIZE_LIMIT</code>, <code>ARCHIVE_NESTING_LEVEL_OVER_LIMIT</code> and <code>JSON_NESTING_LEVEL_OVER_LIMIT</code>. Filter on those before declaring a data lake of large compressed exports clean. Check current quota values in the Macie documentation rather than memorizing them, since they differ per format.</p>



<h2 class="wp-block-heading">Automated discovery or a discovery job?</h2>



<p class="wp-block-paragraph"><strong>Automated sensitive data discovery</strong> evaluates your inventory daily and samples representative objects across as many buckets as it can, grouping by metadata like prefix, extension and last-modified date. It is breadth-first and it does not re-analyze unchanged objects. Its job is to tell you which buckets deserve attention. It is not designed to prove a bucket is clean, and reading it that way is the root of most Macie misunderstandings.</p>



<p class="wp-block-paragraph"><strong>Sensitive data discovery jobs</strong> are the depth tool. You choose the buckets, scope by prefix, extension, size or object tag, choose the identifiers, and run once or on a schedule. This is what you point at the bucket automated discovery just flagged.</p>



<pre class="wp-block-code"><code># One-time job over a single prefix, recommended identifiers, 30% sampling
aws macie2 create-classification-job 
  --job-type ONE_TIME 
  --name "exports-prefix-review" 
  --managed-data-identifier-selector RECOMMENDED 
  --sampling-percentage 30 
  --s3-job-definition '{
    "bucketDefinitions": [
      {"accountId": "111122223333", "buckets": ["your-bucket-name"]}
    ],
    "scoping": {
      "includes": {
        "and": [
          {"simpleScopeTerm": {
             "comparator": "STARTS_WITH",
             "key": "OBJECT_KEY",
             "values": ["exports/"]
          }}
        ]
      }
    }
  }'</code></pre>



<p class="wp-block-paragraph">One detail about <code>--sampling-percentage</code> that trips people up: it selects a random percentage of eligible <em>objects</em> and then analyzes each selected object completely. It does not read a fraction of each file. For a bucket where every object comes from one pipeline, 20 or 30 percent tells you what you need. For heterogeneous uploads, sampling is a coin flip and you want the full pass.</p>



<h2 class="wp-block-heading">What actually drives the bill</h2>



<p class="wp-block-paragraph">Macie bills on three dimensions, and the one that surprises people is not the one they budget for.</p>



<ul class="wp-block-list">
<li><strong>Bucket evaluation.</strong> Per general purpose bucket monitored, prorated daily, up to the account ceiling. Predictable, driven by bucket count rather than data volume.</li>



<li><strong>Object monitoring.</strong> Per object tracked while automated discovery is enabled. This is inventory bookkeeping, not content reading, and it runs whether or not anything gets scanned that day. On an account with a log archive holding hundreds of millions of tiny objects, it can dwarf the other two.</li>



<li><strong>Data inspected.</strong> Per GB actually analyzed, covering both automated discovery and jobs.</li>
</ul>



<p class="wp-block-paragraph">So excluding buckets from automated discovery is a genuine cost lever, not just noise reduction. Log archives, CloudTrail destinations, build artifact stores and backup targets are usually safe exclusions. Exclude for the right reason though: a backup bucket holding database dumps of your customer table is exactly what you want scanned, however boring its name sounds.</p>



<p class="wp-block-paragraph">The three dimensions appear as distinct usage types in the Cost and Usage Report, so they split cleanly. A FinOps platform like Vantage or CloudZero, or just a Cost Explorer view grouped by usage type, tells you within a day whether object monitoring or inspection is driving the number. That changes the fix entirely: object monitoring is solved by scoping, inspection by sampling and scheduling.</p>



<h2 class="wp-block-heading">Getting findings somewhere a human sees them</h2>



<p class="wp-block-paragraph">Findings that only exist in the Macie console get read during onboarding week and never again. Two things are worth wiring up on day one.</p>



<p class="wp-block-paragraph"><strong>Publish findings outward.</strong> Macie emits findings to EventBridge and integrates with Security Hub. EventBridge earns its keep because you can filter on severity and finding type and route only what matters, whether that&#8217;s a Slack channel, a ticket, or a Lambda that tags the bucket. If you already centralize alerts in Grafana Cloud or similar, that path keeps sensitive data findings next to everything else on call sees.</p>



<p class="wp-block-paragraph"><strong>Configure the discovery results repository.</strong> Findings tell you what was found and where. The detailed discovery results, including analysis logs for objects where nothing was found, need an S3 bucket and a KMS key configured to persist. Nothing is retained long term until you set this up, and &#8220;we scanned it and found nothing&#8221; is exactly the record an auditor asks for.</p>



<p class="wp-block-paragraph">One operational note on the reveal feature, which retrieves sample occurrences so a human can confirm a finding is real. It needs a customer managed KMS key and it&#8217;s genuinely useful for triage. It also means someone is now looking at live customer PII, which belongs in your access model and audit trail. Same goes if an engineer downloads an object to investigate locally: that copy is unmanaged PII on a laptop and needs a secure deletion step, whether that&#8217;s a wipe tool from a vendor like O&amp;O Software or a documented and enforced process. Findings triage has a habit of creating the exposure it was meant to prevent.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting a bucket that reports nothing</h2>



<p class="wp-block-paragraph">When Amazon Macie PII detection reports nothing for a bucket you have doubts about, work down this in order. Each step rules out one gate, and stopping at the first hit saves you the rest.</p>



<ol class="wp-block-list">
<li><strong>Check the sensitivity label.</strong> <em>Not yet analyzed</em> at score 50 means nothing was analyzed. Do not read that as clean.</li>



<li><strong>Check the coverage page.</strong> Macie names the issue directly: access denied, invalid encryption, invalid KMS key, permission denied, or unclassifiable.</li>



<li><strong>Compare classifiable and unclassifiable counts</strong> with the <code>describe-buckets</code> query above.</li>



<li><strong>List a few object keys.</strong> Run <code>aws s3 ls</code> against a prefix and look for extensions. Extensionless keys explain the whole thing.</li>



<li><strong>Check the storage class.</strong> A lifecycle rule may have moved everything somewhere Macie doesn&#8217;t read.</li>



<li><strong>Check encryption.</strong> SSE-C is unfixable without re-encryption. A customer managed KMS key is fixable with a key policy change.</li>



<li><strong>Check object samples</strong> with <code>list-resource-profile-artifacts</code> to see exactly what was selected and skipped.</li>



<li><strong>Only now question the identifiers.</strong> Run a targeted job with a custom identifier for the data you expect and see whether it fires.</li>
</ol>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating a low sensitivity score as evidence of no PII. It reflects what was found <em>and</em> how much was analyzed, so low coverage produces a low score.</li>



<li>Enabling Macie in one region and assuming account-wide coverage. Settings and results are per region, every time.</li>



<li>Running default identifiers in a business operating outside the jurisdictions the recommended set covers.</li>



<li>Building custom identifiers from regex alone, then abandoning Macie over the false positive volume.</li>



<li>Skipping the discovery results repository, then having no evidence trail when someone asks what was scanned and when.</li>



<li>Excluding buckets by name pattern without checking contents. Backup and export buckets are frequently the highest-risk ones you own.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Measure coverage before findings. A coverage report is the first deliverable of a Macie rollout, not an afterthought.</li>



<li>Fix extensions at the producer rather than building a rename pipeline downstream.</li>



<li>Run the KMS permission analyzer script once per account, then again whenever a new customer managed key appears.</li>



<li>Use automated discovery for breadth and targeted jobs for depth, letting the first choose targets for the second.</li>



<li>Test every custom identifier against both a positive and a negative sample before it reaches a job.</li>



<li>Curate allow lists early. The cost of a noisy first month is a team that stops reading findings.</li>



<li>Re-check coverage after any change to bucket policies, KMS keys, lifecycle rules or ingestion pipelines. All four silently remove data from scope.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can Amazon Macie scan anything other than S3?</h3>



<p class="wp-block-paragraph">No. Macie analyzes objects in S3 general purpose buckets only. The documented workaround is bringing data to it: export RDS or Aurora snapshots to S3 in Parquet, or export a DynamoDB table to S3, then run a discovery job against the export. That works, but the export is now a second copy of your sensitive data with its own encryption and deletion requirements.</p>



<h3 class="wp-block-heading">Does Macie scan new objects as they land?</h3>



<p class="wp-block-paragraph">Not per object. Automated discovery evaluates inventory on a daily cycle and prioritizes objects that are new or recently changed, so fresh data moves up the queue rather than triggering an immediate scan. If you need scanning tied to arrival, drive a discovery job from S3 event notifications through EventBridge and accept that you own that orchestration.</p>



<h3 class="wp-block-heading">Why does a bucket I know has PII show a low sensitivity score?</h3>



<p class="wp-block-paragraph">Almost always coverage rather than detection. Check whether the objects are classifiable, whether Macie can decrypt them, and whether the label reads <em>Not yet analyzed</em>. If Macie genuinely read them and found nothing, the next suspect is the identifier set, and a targeted job with a custom identifier will tell you in one run.</p>



<h3 class="wp-block-heading">Do I need both automated discovery and discovery jobs?</h3>



<p class="wp-block-paragraph">For most teams, yes. Automated discovery gives you the map at predictable cost. Jobs give you proof for a specific bucket at a specific time, which is what compliance evidence actually requires. Running only jobs means you never discover the bucket nobody told you about.</p>



<h3 class="wp-block-heading">How do I cut false positives without missing real PII?</h3>



<p class="wp-block-paragraph">In order of preference: add keywords and a proximity rule so matches need context; use allow lists for specific known-benign values like your published contact details and test fixtures; and only then remove managed identifiers that are consistently wrong for your data. Removing identifiers is the bluntest option and the easiest to regret, so leave it last.</p>



<h3 class="wp-block-heading">Does Macie work across accounts in an organization?</h3>



<p class="wp-block-paragraph">Yes, through AWS Organizations with a delegated administrator. Design around this: the administrator&#8217;s automated discovery settings apply to member accounts, members can&#8217;t change them, and members see coverage and sensitivity data for their own buckets but not the sensitive data findings themselves. That shapes who can actually remediate what.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Amazon Macie PII detection is only as good as its coverage, and coverage fails silently by design. An object with the wrong extension, the wrong storage class, or an unreachable encryption key produces no finding, and no finding renders identically to a clean result. Before you hand a Macie report to anyone who will make a decision from it, put the coverage numbers next to it: how many objects were classifiable, how many were analyzed, how many were skipped and why.</p>



<p class="wp-block-paragraph">Do that once and the rest of the service becomes straightforward. Skip it and you&#8217;re maintaining an expensive dashboard reporting on whichever subset of your data happened to be readable.</p>



<h2 class="wp-block-heading">Need help getting Macie to actually see your data?</h2>



<p class="wp-block-paragraph">Most of the work in a Macie rollout isn&#8217;t turning it on. It&#8217;s the unglamorous part: proving what got scanned, fixing the reasons things didn&#8217;t, and making the output land somewhere a human acts on it. That&#8217;s the part I help with.</p>



<ul class="wp-block-list">
<li>Coverage audits across your S3 estate, with per-bucket classifiable, analyzed and skipped counts and the specific cause of each gap</li>



<li>Fixing those causes: KMS key policies, bucket policies blocking the service-linked role, lifecycle rules archiving data out of scope, pipelines writing extensionless objects</li>



<li>Custom data identifiers and allow lists tuned against your real data and tested on positive and negative samples first</li>



<li>Cost shaping: working out whether object monitoring or data inspection drives your bill, and scoping discovery so the number is defensible</li>



<li>Findings pipelines through EventBridge or Security Hub into the alerting and ticketing you already run, with severity filtering that keeps signal above noise</li>



<li>Multi-account setup under AWS Organizations, including the delegated administrator model and who can see and remediate what</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion, send me the output of the <code>describe-buckets</code> query above, or a screenshot of your coverage page, and I&#8217;ll tell you what&#8217;s actually being scanned.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/amazon-macie-pii-detection/">Amazon Macie PII Detection: The Buckets It Never Opened</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>MediaConvert Cost Per Minute: How the Bill Is Built and How Jobs Fail</title>
		<link>https://john-nessime.com/blog/technical-guides/mediaconvert-cost-per-minute/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 29 Aug 2026 09:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Adaptive Bitrate]]></category>
		<category><![CDATA[Amazon CloudFront]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Error Handling]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[MediaConvert]]></category>
		<category><![CDATA[Normalized Minutes]]></category>
		<category><![CDATA[QVBR]]></category>
		<category><![CDATA[Reserved Transcode Slots]]></category>
		<category><![CDATA[Video Transcoding]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=542</guid>

					<description><![CDATA[<p>One checkbox can reprice every output in a MediaConvert job. A working breakdown of how the per-minute charge is assembled from normalized minutes and per-output billing, why failed jobs are free but still expensive, and how to build retry and recovery that classifies before it retries.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/mediaconvert-cost-per-minute/">MediaConvert Cost Per Minute: How the Bill Is Built and How Jobs Fail</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A turnaround complaint lands: the promo team&#8217;s masters take too long to appear in the CMS. Someone opens the job template, finds a setting called accelerated transcoding, ticks it, and moves on. Turnaround improves. Nobody mentions it again.</p>



<p class="wp-block-paragraph">The invoice is where it surfaces, and not only on the big outputs. The 240p ladder rung went up. The audio-only rendition went up. Every output in every job that template touches went up, including the ones that never needed the feature.</p>



<p class="wp-block-paragraph">That is not a billing bug. It is the documented rule working as designed, and it is the most common reason a bill moves without anyone changing how much video goes through it. Understanding <strong>MediaConvert cost per minute</strong> starts with accepting that the unit you pay for is not a job, not a source file, and not a minute of compute.</p>



<p class="wp-block-paragraph">What follows is how that charge is assembled, what happens to it when a job fails, and how to build recovery around AWS Elemental MediaConvert. Rates are deliberately absent; AWS changes them and they vary by region. The mechanism is what catches people.</p>



<h2 class="wp-block-heading">How MediaConvert cost per minute is actually assembled</h2>



<p class="wp-block-paragraph">Three rules stack on top of each other. Miss one and your cost model is wrong by a multiple, not a rounding error.</p>



<h3 class="wp-block-heading">You pay per output, not per job</h3>



<p class="wp-block-paragraph">An eight-rung ABR ladder plus an audio-only rendition is nine billable items, each charged for its own duration. AWS bills the duration of the finished assets, not the transcoding time, in hundredth-of-a-minute increments. A ten-minute source producing nine outputs bills as ninety minutes of output.</p>



<p class="wp-block-paragraph">Ladder design is therefore a cost decision, not only a quality one. The bottom rungs are not cheap because they are small. They cost a full pass of the source duration at whatever multiplier their settings attract, whether anyone streams them or not.</p>



<h3 class="wp-block-heading">Normalized minutes, and why multipliers compound</h3>



<p class="wp-block-paragraph">There is no flat rate per output minute. MediaConvert converts each output into normalized minutes, a unit that scales with the work the encode demanded. Resolution, frame rate, codec and number of encoding passes each carry a multiplier, and those multipliers combine. Single-pass to multi-pass is one. 30fps to 60fps is another. Do both to a UHD rung and you have stacked three.</p>



<p class="wp-block-paragraph">Volume discounts then apply to your accumulated normalized minutes for the month, per region, with basic-tier and professional-tier usage accruing separately toward their own thresholds. Two consequences: splitting a workload across regions splits the accrual and can leave you paying a higher effective rate in both, and under consolidated billing in AWS Organizations usage accrues across accounts, so a per-account model understates the discount you actually get.</p>



<h3 class="wp-block-heading">The job-wide setting that reprices everything</h3>



<p class="wp-block-paragraph">Back to the opening. Each output is assessed for its billing tier separately, but only when the job has no job-wide professional-tier feature enabled. Turn one on, accelerated transcoding being the usual culprit, and every output in that job bills at pro-tier rates. An SD rung that would have billed at basic tier gets repriced because a different output in the same job needed to finish faster.</p>



<p class="wp-block-paragraph">If a subset of content genuinely needs acceleration, split it out. Long, visually complex assets get a template with the feature; routine web renditions get one without. Two templates is less tidy than one, and considerably cheaper than a single checkbox repricing your catalogue.</p>



<h2 class="wp-block-heading">Failed jobs are free, which changes how you should retry</h2>



<p class="wp-block-paragraph">MediaConvert charges only when a job reaches COMPLETED. Jobs ending in ERROR are not billed. That sounds like it removes the urgency from failure handling. It does the opposite.</p>



<p class="wp-block-paragraph">A job that fails on output seven of nine may still have written outputs one through six to S3. Those partial artefacts cost storage, and if a naive retry writes to the same prefix you get a mix of old and new segments that a player will serve quite happily until someone notices the ladder is inconsistent. Free at the transcode line, expensive everywhere downstream.</p>



<p class="wp-block-paragraph">&#8220;Not billed&#8221; also makes blind retries feel harmless, so people write them. A loop against a deterministic failure, a corrupt source or a settings mismatch, will never succeed. It burns concurrency paying jobs need and produces identical alerts until the team ignores the channel. The service already retries some conditions itself and exposes a retry count, so read that field before adding an attempt of your own.</p>



<h2 class="wp-block-heading">The queue model is a throughput control, not a cost control</h2>



<p class="wp-block-paragraph">On-demand queues bill per output minute. How many jobs run at once is governed by a service quota for concurrent jobs per account in the region, a quota per individual queue, and a concurrent-jobs setting on each queue. Creating a second queue does not conjure capacity out of the account; you raise it through Service Quotas. Jobs sitting in SUBMITTED rather than moving to PROGRESSING is the signal you have hit the ceiling.</p>



<p class="wp-block-paragraph">Reserved queues work on a different axis. You buy reserved transcode slots against a twelve-month commitment and pay for that capacity regardless of use. Each slot has its own dedicated compute, so one slot runs a given job just as fast as ten would. More slots means more jobs in parallel, not faster jobs. Three things about the commitment deserve a second read before anyone clicks agree:</p>



<ul class="wp-block-list">
<li>You cannot cancel it, and you cannot decrease the slot count.</li>

<li>Adding capacity does not extend the existing term. It starts a fresh twelve-month commitment for the larger count, from the day you add it.</li>

<li>When the plan expires, the queue persists and keeps accepting jobs. MediaConvert just does not run them.</li>
</ul>



<p class="wp-block-paragraph">That last one is the invisible failure in this whole topic. Your submission code gets a job ID back. The API call succeeded. Nothing errors. The jobs simply never progress, and unless you alarm on queue standby time or on the gap between jobs submitted and completed, the first person to notice is whoever asks why the library stopped updating. Set auto-renew deliberately, either way, and diary the term end date.</p>



<p class="wp-block-paragraph">Queue hopping is the pressure valve. Give a job a wait time and a destination queue, and if it is still waiting when the timer expires it moves. The usual shape is reserved to on-demand during a spike. Billing follows the queue the job actually runs in, so a hopped job becomes on-demand spend.</p>



<pre class="wp-block-code"><code>{
  "Settings": { "OutputGroups": [], "Inputs": [] },
  "HopDestinations": [
    {
      "WaitMinutes": 10,
      "Queue": "arn:aws:mediaconvert:us-west-2:111122223333:queues/ondemandqueue",
      "Priority": 25
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">WaitMinutes accepts 1 to 4320. Priority runs from -50 to 50 and defaults to 0, and the optional priority under HopDestinations lets a hopped job land higher or lower in the destination queue than it sat in the original. Hopping is a cost decision as much as a throughput one: it converts capacity you have already paid for into incremental on-demand spend.</p>



<h2 class="wp-block-heading">The failure families that actually bite</h2>



<p class="wp-block-paragraph">MediaConvert publishes numbered error codes with documented causes. Rather than reproduce the table, group them by what you have to do about them, because the correct response differs sharply between groups.</p>



<h3 class="wp-block-heading">Access failures</h3>



<p class="wp-block-paragraph">The service could not open an input or output location. Causes cluster around IAM permissions on the role the job runs under, cross-account bucket policies when the source lives elsewhere, keys that do not exist, and files whose headers cannot be read. Retry these once with a delay, because uploads still in flight are a real condition. If the second attempt fails identically, it is configuration, and no amount of retrying fixes configuration.</p>



<h3 class="wp-block-heading">Settings failures that only appear on some inputs</h3>



<p class="wp-block-paragraph">The nastiest to diagnose, because the template has been fine for the last thousand assets. An encoder initialisation failure on one rung, triggered by an unusual source aspect ratio or frame rate meeting a fixed constraint in the template, is the classic shape. Never retry these; the job fails identically forever. The fix is usually to loosen the constraint rather than pin it harder, because letting the service derive a value the source cannot satisfy removes the whole class of failure.</p>



<h3 class="wp-block-heading">Warnings, which are not errors and are easy to miss</h3>



<p class="wp-block-paragraph">MediaConvert also returns warning codes for problems that do not stop the job. It completes, you get billed, and something about the output is wrong. Video padding inserted to keep audio and video aligned, black frames carried through from the source, a QVBR score well below the rest of the catalogue: none raise an error, all are visible in metrics. Alert only on ERROR and this entire family is invisible to you.</p>



<h3 class="wp-block-heading">Pinned engine versions that quietly expire</h3>



<p class="wp-block-paragraph">MediaConvert groups its releases into job engine versions so you can hold production on one while validating another. Access is requested through AWS Support rather than open to every account, so check whether it applies to you first. The behaviour worth knowing is expiry: requesting an expired version does not fail. The job runs as though no version had been specified, and the field reporting the version used comes back empty. An invalid version, by contrast, returns a 400 and creates no job. The failure mode of pinning is silence.</p>



<h2 class="wp-block-heading">Building failure recovery that does not cost you twice</h2>



<p class="wp-block-paragraph">The pattern that holds up is event-driven, classified and idempotent. In order:</p>



<ol class="wp-block-list">
<li>Subscribe to job state changes in EventBridge rather than polling. The ERROR event carries the code and message in its detail block.</li>

<li>Route by error code, not by status: one target for retryable access conditions, another for deterministic settings failures that need a human, and a suppression path for anything already seen on that asset.</li>

<li>Write a retry&#8217;s outputs to a new prefix, or clear the old one first. Never let attempt two land beside attempt one&#8217;s partial results.</li>

<li>Send a client request token on job creation. Reusing the same token within a minute of a successful request returns the original job instead of creating a duplicate, which protects you when your orchestration retries a create call whose response it never saw.</li>

<li>Cap attempts and land the remainder in a dead letter queue with the original event attached. A failure nobody can see is a failure nobody fixes.</li>

<li>Carry your own identifiers through user metadata on the job, so the returning event joins back to the asset without a lookup table.</li>
</ol>



<pre class="wp-block-code"><code>{
  "source": ["aws.mediaconvert"],
  "detail-type": ["MediaConvert Job State Change"],
  "detail": {
    "status": ["ERROR"]
  }
}</code></pre>



<p class="wp-block-paragraph">The matching event delivers a detail object containing the queue ARN, job ID, status, an errorCode and an errorMessage. Route on errorCode. That one decision is the difference between a recovery system and a retry loop.</p>



<h2 class="wp-block-heading">Measuring what you spend before the invoice tells you</h2>



<p class="wp-block-paragraph">MediaConvert sends metrics to CloudWatch at the end of every job, several mapping directly onto the billing dimensions. The ones I reach for first:</p>



<ul class="wp-block-list">
<li><code>SDOutputDuration</code>, <code>HDOutputDuration</code>, <code>UHDOutputDuration</code>, <code>8KOutputDuration</code> and <code>AudioOutputDuration</code>, in milliseconds per queue. Summed over a month, that is billable output volume by resolution class, days ahead of Cost Explorer.</li>

<li><code>JobErrorsByCode</code>, counting jobs that failed with a particular code. Build the dashboard on this one; it shows which failure family is growing.</li>

<li><code>StandbyTime</code>, how long a job waits before processing starts. Alarm on it and you catch backlogs, and you catch an expired reserved plan.</li>

<li><code>JobsSubmittedCount</code> against <code>JobsCompletedCount</code>. A widening gap is the earliest honest signal something upstream is broken.</li>

<li><code>QVBRAvgQualityLowBitrate</code> and <code>QVBRMinQualityHighBitrate</code>, for quality regressions that never raise an error.</li>
</ul>



<p class="wp-block-paragraph">For attribution rather than volume, enable cost allocation tags and set the billing tags source on your jobs so per-output costs land against a tag you chose. Pushing that into a wider FinOps view through Vantage, CloudZero or a Grafana Cloud dashboard fed from CloudWatch gets you per-title cost without building the aggregation yourself. Either way, these metrics let you model MediaConvert cost per minute continuously instead of discovering it monthly.</p>



<p class="wp-block-paragraph">Transcoding is only part of the line. Storage for sources and renditions in S3, and egress through Amazon CloudFront or whichever CDN fronts delivery, are separate charges and often the larger ones. A cost model that stops at the encoder will be wrong.</p>



<h2 class="wp-block-heading">Troubleshooting a failed job</h2>



<ol class="wp-block-list">
<li>Get the error code first. Everything else follows from which family it belongs to.</li>

<li>Ask whether other assets in the same bucket succeeded in the same window. Yes points at the file, no points at permissions or configuration.</li>

<li>For a settings failure, bisect the job. Strip it to a single video output, confirm it runs, then add outputs and features back one at a time. Tedious, and faster than reasoning about it.</li>

<li>Check warnings on jobs that completed nearby. Warnings often precede errors from the same root cause.</li>

<li>If nothing ran at all, look at the queue before the job: plan status on a reserved queue, standby time on an on-demand one.</li>
</ol>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Modelling cost from source duration instead of total output duration across the ladder.</li>

<li>Enabling a job-wide professional-tier feature in a shared template and repricing every output in it.</li>

<li>Creating extra on-demand queues for throughput when the account quota is the real ceiling.</li>

<li>Buying reserved slots off a peak-week estimate and paying for idle capacity the other fifty-one.</li>

<li>Retrying every failure identically, including the deterministic ones that cannot succeed.</li>

<li>Alerting on ERROR only, and never looking at warnings or quality scores.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>Separate templates by cost profile rather than content type. Anything with a job-wide pro-tier feature gets its own.</li>

<li>Audit the ladder against playback data at least once. Rungs nobody streams are pure cost.</li>

<li>Keep a workload in one region so volume discounts accrue against a single pool.</li>

<li>Run a representative job on an on-demand queue with the reserved simulation setting enabled before committing to slots.</li>

<li>Use job priority instead of extra queues when the goal is for urgent work to jump ahead.</li>

<li>Derive client request tokens from the asset, not from a timestamp, so retries are genuinely idempotent.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I pay for MediaConvert jobs that fail?</h3>



<p class="wp-block-paragraph">No. Billing applies only to jobs reaching COMPLETED status, so a job ending in ERROR is not charged. You may still be paying for partial outputs that reached S3 before the failure, and the storage they occupy until something cleans them up.</p>



<h3 class="wp-block-heading">What is a normalized minute?</h3>



<p class="wp-block-paragraph">The billing unit MediaConvert uses so a minute of demanding output counts for more than a minute of simple output. Resolution, frame rate, codec and encoding passes each contribute a multiplier, and those combine into the normalized total your volume discount tier is measured against.</p>



<h3 class="wp-block-heading">Is a reserved queue cheaper than on demand?</h3>



<p class="wp-block-paragraph">Only if utilisation is genuinely steady. You pay for reserved transcode slots regardless of use, across a twelve-month commitment you cannot cancel or shrink. Bursty catalogues usually do better on demand, where volume discounts apply automatically as monthly usage climbs. Reserved pricing also excludes some features, so confirm everything your templates use is available there before committing.</p>



<h3 class="wp-block-heading">Why are my jobs stuck in SUBMITTED?</h3>



<p class="wp-block-paragraph">On an on-demand queue, it is already running its maximum number of concurrent jobs; raise the quota through Service Quotas rather than adding queues. On a reserved queue, check whether the pricing plan has expired, because an expired queue keeps accepting jobs and never processes them.</p>



<h3 class="wp-block-heading">Does adding queues give me more transcoding capacity?</h3>



<p class="wp-block-paragraph">No. Concurrency is bounded by account-level and queue-level quotas in the region. Extra queues isolate workloads from each other, for instance keeping one client&#8217;s jobs from starving another&#8217;s. Capacity increases come from Service Quotas.</p>



<h3 class="wp-block-heading">How do I stop one setting from repricing my whole job?</h3>



<p class="wp-block-paragraph">Split the job. Job-wide professional-tier features apply their tier to every output, so isolate the assets that need them into their own template and keep routine renditions in a template with no job-wide pro-tier features enabled.</p>



<h3 class="wp-block-heading">Should I run FFmpeg on my own servers instead?</h3>



<p class="wp-block-paragraph">Sometimes. Predictable, modest volume on dedicated hardware from a provider like Contabo or InterServer can undercut per-minute pricing, and you get exact control over encoder flags. What you take on is capacity planning, codec licensing questions, and building the retry and observability machinery the managed service hands you for free. That trade favours MediaConvert for spiky workloads and broadcast-grade features, and self-hosting for steady, simple ladders where an engineer already owns the pipeline.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">MediaConvert cost per minute is not a rate you look up. It is a product: output count, times duration, times a stack of feature multipliers, with a single job-wide setting able to reprice the lot. Get that model into whatever you plan from and the bill stops being a monthly surprise.</p>



<p class="wp-block-paragraph">On the failure side, the codes are documented and the events are structured, so there is no excuse for a retry loop that cannot tell a permissions problem from a settings problem. Classify before you retry. And put an alarm on queue standby time today, because the failure that costs you most is the one where nothing errors at all.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your transcoding pipeline?</h2>



<p class="wp-block-paragraph">I work with teams running video-on-demand pipelines on AWS, usually at the point where the bill or the failure rate has become someone&#8217;s problem. What I actually help with here:</p>



<ul class="wp-block-list">
<li>Auditing job templates and ABR ladders for outputs and multipliers costing more than they return.</li>

<li>Building a cost model from your CloudWatch output-duration metrics so you can forecast rather than react.</li>

<li>Designing EventBridge-driven failure handling that classifies by error code and retries only what can succeed.</li>

<li>Working the reserved versus on-demand decision against your real utilisation, including whether queue hopping earns its keep.</li>

<li>Debugging settings failures that only hit certain sources, and loosening the constraints causing them.</li>

<li>Wiring transcoding, S3 lifecycle and CDN delivery into one cost and reliability picture instead of three.</li>
</ul>



<p class="wp-block-paragraph">Send me a job template, an error code, or a month of output-duration metrics and I will tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/mediaconvert-cost-per-minute/">MediaConvert Cost Per Minute: How the Bill Is Built and How Jobs Fail</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Salesforce AWS Integration Architecture: The Four Paths and What Breaks Silently</title>
		<link>https://john-nessime.com/blog/devops/salesforce-aws-integration-architecture/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 19 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AppFlow]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Change Data Capture]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Event Relay]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[Named Credentials]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Platform Events]]></category>
		<category><![CDATA[PrivateLink]]></category>
		<category><![CDATA[Pub/Sub API]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[Serverless]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=209</guid>

					<description><![CDATA[<p>Most Salesforce AWS integrations don't fail loudly, they drift. This guide covers the four integration paths between Salesforce and AWS, when to use each, the silent failure mode hiding in every one of them, and the reconciliation job you have to build yourself.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-aws-integration-architecture/">Salesforce AWS Integration Architecture: The Four Paths and What Breaks Silently</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone in the ops channel asks why the opportunity count on the QuickSight dashboard doesn&#8217;t match the count in the Salesforce report. It&#8217;s off by a few hundred. Nobody can answer that, and worse, nobody can say when the two numbers stopped agreeing. Every pipeline is green. The scheduled flow reports successful runs. The EventBridge rule shows invocations. Everything is healthy except the data.</p>



<p class="wp-block-paragraph">That&#8217;s the shape of most Salesforce AWS integration problems. They&#8217;re rarely outages. They&#8217;re divergence, accumulating quietly while every status page stays green, and you find out about it when someone in finance runs a number past a customer.</p>



<p class="wp-block-paragraph">This guide covers the four integration paths between Salesforce and AWS, which one to reach for and when, the specific failure each one hides, and the reconciliation layer you&#8217;ll have to build yourself because neither vendor ships it.</p>



<h2 class="wp-block-heading">Why a Salesforce AWS integration fails silently</h2>



<p class="wp-block-paragraph">Start here, because it changes how you design everything downstream. There are three mechanisms that lose data without raising an error anywhere you&#8217;re looking.</p>



<p class="wp-block-paragraph"><strong>Events sent to an unassociated partner event bus are dropped on the floor.</strong> When you create an Event Relay, Salesforce creates a partner event source in your AWS account. Until you associate that source with an event bus, EventBridge discards anything the partner publishes to it. AWS is explicit that those events are not persisted at rest. There&#8217;s no queue holding them for you, no backlog to drain once you finish the setup. If the relay is running and the bus isn&#8217;t associated, you are streaming into a void.</p>



<p class="wp-block-paragraph"><strong>The Salesforce event bus retains high-volume events for 72 hours.</strong> Platform events and change data capture events sit on the event bus for three days, and each one carries a replay ID so a durable subscriber can resume from where it left off. Past that window, they&#8217;re purged. Salesforce doesn&#8217;t guarantee availability beyond it. So a relay that goes into an error state on a Friday afternoon is a data loss event by Monday morning, and the only symptom during those three days is an absence of messages, which looks identical to a quiet weekend.</p>



<p class="wp-block-paragraph"><strong>Deletes don&#8217;t show up in query-based syncs.</strong> A scheduled pull selects records that exist. Records that stopped existing between runs are simply not in the result set, so your S3 prefix or Redshift table keeps a row that Salesforce no longer has. Nothing errors. The row count on the AWS side just creeps upward relative to the CRM, forever, until someone notices.</p>



<p class="wp-block-paragraph">Every architecture decision below is really a decision about which of these three you&#8217;re accepting and how you&#8217;re going to detect it.</p>



<h2 class="wp-block-heading">A Salesforce AWS integration starts with a direction, not a service</h2>



<p class="wp-block-paragraph">The most common mistake I see is picking a service first, usually whichever one appeared in a conference demo, and then bending the requirement to fit it. Two questions decide the path, and they take about a minute to answer.</p>



<ol class="wp-block-list">
<li><strong>Which way does the data move?</strong> Salesforce to AWS, or AWS to Salesforce. If the answer is &#8220;both&#8221;, you have two integrations, not one, and they will have different failure modes and different owners.</li>

<li><strong>What shape is the workload?</strong> Batch, where you want a consistent set of records on a schedule, or event-driven, where you want a reaction within seconds of a change.</li>
</ol>



<p class="wp-block-paragraph">That gives you four cells, and each cell has a default answer.</p>



<h3 class="wp-block-heading">Salesforce to AWS, batch: Amazon AppFlow</h3>



<p class="wp-block-paragraph">Amazon AppFlow is the managed answer for pulling Salesforce objects into S3, Redshift, or the Glue Data Catalog on a schedule or on demand. It handles the OAuth connection, field mapping, and incremental pulls without you writing or hosting anything. For analytics loads it&#8217;s usually the right default, and it&#8217;s the path of least resistance if the destination is a data lake you&#8217;re already cataloguing with Glue and querying with Athena.</p>



<p class="wp-block-paragraph">The trade-off is that it&#8217;s a polling client, and it spends your Salesforce API allocation to do it. AWS documents this plainly with an example: an hourly flow pulling five pages of data makes 120 API calls a day, and all of them count against the org&#8217;s 24-hour API request limit. Multiply that by every object you&#8217;re syncing and every environment you&#8217;re syncing from, and AppFlow can become the largest single consumer of an allocation that Apex callouts, integration users, and third-party tools are also drawing on.</p>



<p class="wp-block-paragraph">The failure mode that catches people: if a field referenced in the flow&#8217;s mapping is deleted in Salesforce, the flow run fails. Someone cleans up an unused custom field on a Tuesday and your nightly load breaks on Wednesday, in a service that nobody on the Salesforce side knows exists. AppFlow will automatically pick up newly created Salesforce fields when the destination is S3, but the reverse case, a removed field, needs a human to edit the mapping. Put the flow names somewhere the Salesforce admins will see them before they run a field cleanup.</p>



<h3 class="wp-block-heading">Salesforce to AWS, event-driven: Event Relay and Amazon EventBridge</h3>



<p class="wp-block-paragraph">Event Relay subscribes to a channel on the Salesforce event bus and streams platform events and change data capture events into Amazon EventBridge as a partner event source. No middleware, no subscriber you have to keep alive, no polling and therefore no API allocation burn on the read path. From EventBridge you fan out to Lambda, Step Functions, SQS, Firehose, whatever the downstream is.</p>



<p class="wp-block-paragraph">Setup runs across both consoles, and the order matters. On the Salesforce side you create a named credential that points at your AWS account and region. It uses no authentication protocol, because trust is established by the partner event source relationship rather than by a credential in the request. The region goes in uppercase, which is the single most common reason a first attempt doesn&#8217;t work.</p>



<pre class="wp-block-code"><code>POST /services/data/vXX.X/tooling/sobjects/NamedCredential/

{
  "FullName": "AwsRelayCredential",
  "Metadata": {
    "label": "AwsRelayCredential",
    "endpoint": "arn:aws:US-EAST-1:111122223333",
    "principalType": "NamedUser",
    "protocol": "NoAuthentication",
    "generateAuthorizationHeader": true
  }
}</code></pre>



<p class="wp-block-paragraph">The endpoint is not a URL. It&#8217;s an ARN-shaped string carrying the AWS region and the twelve-digit account ID, and that&#8217;s the whole addressing scheme. Get the account wrong and the partner event source appears in a stranger&#8217;s console, not yours.</p>



<p class="wp-block-paragraph">You then create an event channel, add the platform events or change events you want as channel members, and create the relay configuration against that channel. A relay is created in a stopped state deliberately, so you get a chance to verify the AWS account and region before anything flows. That pause is the useful part of the design, not an inconvenience to click through.</p>



<p class="wp-block-paragraph">Now switch to AWS and associate the source. This is the step that, skipped, produces the silent drop described earlier.</p>



<pre class="wp-block-code"><code># Find the partner event source Salesforce created in your account
aws events list-event-sources --region us-east-1

# Create the matching bus. Both values must be the source name, character for character.
aws events create-event-bus 
  --name "aws.partner/salesforce.com/&lt;org-and-channel-suffix&gt;" 
  --event-source-name "aws.partner/salesforce.com/&lt;org-and-channel-suffix&gt;" 
  --region us-east-1

# Confirm the source moved from PENDING to ACTIVE
aws events describe-event-source 
  --name "aws.partner/salesforce.com/&lt;org-and-channel-suffix&gt;" 
  --region us-east-1</code></pre>



<p class="wp-block-paragraph"><code>describe-event-source</code> returning <code>ACTIVE</code> is your proof that a matching bus exists and is enabled. <code>PENDING</code> means either you haven&#8217;t created the bus or it&#8217;s deactivated, and every event published in that state is gone. This is the single check worth wiring into a synthetic monitor, because it&#8217;s cheap to run and it catches the most expensive mistake.</p>



<p class="wp-block-paragraph">Only once the source is active do you set the relay state to run. From the Salesforce side, the feedback object is where relay errors surface rather than in any log you&#8217;d normally tail:</p>



<pre class="wp-block-code"><code>SELECT Id, RemoteResource, Status, ErrorMessage, ErrorTime, ErrorIdentifier
FROM EventRelayFeedback
WHERE EventRelayConfigId = '7k2XXXXXXXXXXXXXXX'</code></pre>



<p class="wp-block-paragraph">If you&#8217;d rather not use Event Relay at all, the Pub/Sub API is the direct route: a gRPC interface to the same event bus, delivering Avro-encoded payloads, which you subscribe to from your own consumer. That&#8217;s more code and a process to keep running, on ECS or a VPS from a provider like InterServer or Hetzner if you&#8217;re keeping it outside AWS, but you get explicit control over replay ID checkpointing. Teams already running Kafka often go this way and land the stream through a managed connector from Confluent instead. Event Relay is the lower-operational-cost option; the Pub/Sub API is the one to reach for when you need the checkpoint under your own control.</p>



<h3 class="wp-block-heading">AWS to Salesforce, event-driven: EventBridge API Destinations</h3>



<p class="wp-block-paragraph">Going the other way, an API destination lets EventBridge call the Salesforce REST API directly, with no Lambda in the middle. You create a connection holding OAuth client credentials from a Salesforce connected app, point the destination at your org&#8217;s My Domain endpoint, and route matched events to it. Publishing back into an inbound platform event is the common pattern, so the write lands on the Salesforce event bus and Apex or Flow picks it up from there.</p>



<p class="wp-block-paragraph">Two constraints shape the design, and both are documented:</p>



<ul class="wp-block-list">
<li><strong>Five second client execution timeout.</strong> If Salesforce takes longer than that to respond, EventBridge times out the request and treats it as a retryable failure. Anything you make synchronous behind that endpoint, a trigger doing a callout, a validation chain, a busy org at peak, will eat into it.</li>

<li><strong>The default retry policy is generous, which is the problem.</strong> For event buses, EventBridge will retry a failed delivery for up to 24 hours and up to 185 times by default. If the endpoint is genuinely broken, that&#8217;s a day of retries against your org, consuming API calls, before the event is discarded with nowhere to go.</li>
</ul>



<p class="wp-block-paragraph">Tighten the policy and always attach a dead letter queue. Failed events land somewhere you can inspect and replay rather than evaporating at the end of the retry window:</p>



<pre class="wp-block-code"><code>{
  "Id": "salesforce-writeback",
  "Arn": "arn:aws:events:us-east-1:111122223333:api-destination/sf-platform-event/&lt;id&gt;",
  "RoleArn": "arn:aws:iam::111122223333:role/EventBridgeApiDestinationRole",
  "RetryPolicy": {
    "MaximumRetryAttempts": 20,
    "MaximumEventAgeInSeconds": 3600
  },
  "DeadLetterConfig": {
    "Arn": "arn:aws:sqs:us-east-1:111122223333:salesforce-writeback-dlq"
  }
}</code></pre>



<p class="wp-block-paragraph">Set the invocation rate limit on the destination to something your org can absorb. It exists specifically so you can align delivery with a third-party API&#8217;s limits, and Salesforce is very much a third-party API with limits.</p>



<p class="wp-block-paragraph">One thing worth knowing before you design around it: API destinations don&#8217;t support mutual TLS. If your security review requires mTLS on outbound calls, you&#8217;re putting a Lambda or a proxy in the path regardless.</p>



<h3 class="wp-block-heading">AWS to Salesforce, batch: Lambda and the Bulk API</h3>



<p class="wp-block-paragraph">For loading volume back into the org, nothing managed does this well and you write it yourself. A Lambda or a container job authenticates with the JWT bearer flow against a connected app, submits a Bulk API job, and polls for results. The REST API is fine for small writes; past a few thousand records the per-call overhead and the API allocation make the Bulk API the only sensible choice.</p>



<p class="wp-block-paragraph">This is the cell where you own everything: retries, idempotency, partial-failure handling, and backoff when the org throttles you. Use external ID fields and upsert semantics rather than insert, so a retried batch converges instead of duplicating. Store the job ID before you submit, not after, so a Lambda timeout mid-flight leaves you something to reconcile against. And keep the Salesforce credential in Secrets Manager with rotation wired up, because a hardcoded connected app secret in an environment variable is the finding that shows up in every audit.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The network layer: public internet, PrivateLink, or Private Connect</h2>



<p class="wp-block-paragraph">All four paths above work over the public internet with TLS, and for most workloads that&#8217;s genuinely fine. The question is whether your compliance posture or your risk appetite says otherwise.</p>



<p class="wp-block-paragraph">Salesforce Private Connect is the managed private path. Salesforce operates a transit VPC in each supported AWS region, and you create a PrivateLink connection between that transit VPC and yours. Traffic between the two never traverses the public internet. The terminology is worth getting right because it trips people up in design reviews:</p>



<ul class="wp-block-list">
<li><strong>Inbound</strong> means traffic flowing <em>into</em> Salesforce. AWS to Salesforce, or on-premises to Salesforce.</li>

<li><strong>Outbound</strong> means traffic flowing <em>out of</em> Salesforce. Salesforce calling a service in your VPC.</li>
</ul>



<p class="wp-block-paragraph">The direction is named from Salesforce&#8217;s point of view, not yours, and getting it backwards means provisioning the wrong connection.</p>



<p class="wp-block-paragraph">AppFlow can ride on this too. Choosing the PrivateLink option when you create the Salesforce connection has AppFlow route through Private Connect and manage the endpoint lifecycle for you, which is a meaningful simplification over building it by hand.</p>



<p class="wp-block-paragraph">Where I&#8217;d push back: Private Connect is a licensed add-on, and each VPC needs its own connection, so a multi-account AWS estate with separate VPCs per environment multiplies the cost. If the driver is a checkbox on a questionnaire rather than a real threat model, TLS with strict IP allowlisting on the Salesforce side and a well-scoped connected app gets you most of the way for nothing. If the driver is a regulator who wants to see that customer data never touches a public network path, Private Connect is the clean answer and the audit conversation ends quickly. Decide which one you&#8217;re in before you start pricing it.</p>



<h2 class="wp-block-heading">The reconciliation job nobody builds</h2>



<p class="wp-block-paragraph">This is the part that separates an integration that survives from one that quietly rots, and it&#8217;s about thirty lines of code plus a schedule.</p>



<p class="wp-block-paragraph">Run a count comparison on a window that both sides can express. Nightly is plenty:</p>



<pre class="wp-block-code"><code>-- Salesforce side, via the REST query endpoint
SELECT COUNT(Id) FROM Opportunity
WHERE LastModifiedDate = LAST_N_DAYS:1

-- AWS side, same window, in Athena
SELECT COUNT(DISTINCT id) FROM crm.opportunity
WHERE last_modified_date &gt;= date_add('day', -1, current_date)</code></pre>



<p class="wp-block-paragraph">If the two numbers differ by more than a small tolerance for clock skew at the window edges, alert. That single check catches a stopped relay, a failed flow, a mapping that silently dropped a field, and a partner event source sitting in <code>PENDING</code>. It catches all of them without you having to instrument each path separately, which is exactly why it&#8217;s worth building first.</p>



<p class="wp-block-paragraph">For the delete problem, Salesforce exposes a dedicated endpoint. Ask it for records removed in a window and apply those deletions on the AWS side:</p>



<pre class="wp-block-code"><code>GET /services/data/vXX.X/sobjects/Opportunity/deleted/
    ?start=&lt;ISO-8601 timestamp, URL encoded&gt;
    &amp;end=&lt;ISO-8601 timestamp, URL encoded&gt;</code></pre>



<p class="wp-block-paragraph">The window you can ask about is bounded by how long deleted records stay recoverable, so run it more often than that bound, not less. Change data capture delivers delete events too, which is the better answer when it&#8217;s available, but the endpoint is what you use to backfill after a gap, and after a 72-hour gap it&#8217;s the <em>only</em> thing that will tell you what you missed.</p>



<p class="wp-block-paragraph">Put both checks on one dashboard alongside your AppFlow run status and EventBridge failed-invocation metrics. CloudWatch will do it, and if you&#8217;re already running Grafana Cloud or Datadog for the rest of the estate, pull the Salesforce-side numbers in as a custom metric so one panel answers &#8220;are the two systems agreeing right now&#8221;.</p>



<h2 class="wp-block-heading">Troubleshooting a Salesforce AWS integration</h2>



<ul class="wp-block-list">
<li><strong>Relay is running, nothing arrives in EventBridge.</strong> Run <code>describe-event-source</code>. If the state is <code>PENDING</code>, the bus was never created or has been deactivated, and everything published so far is gone. Create the bus, then treat the gap as a backfill job.</li>

<li><strong>Relay won&#8217;t start, or flips to an error state.</strong> Query <code>EventRelayFeedback</code> for the config. The usual causes are a wrong AWS account ID in the named credential, a region in lowercase, or the channel having no members.</li>

<li><strong>Events arrive but no rule matches.</strong> Check the rule is attached to the partner event bus, not the default bus. This one costs people an afternoon. Match on the source prefix and the detail type, and test with an event pattern before wiring a real target.</li>

<li><strong>An AppFlow flow that ran for months suddenly fails.</strong> Look for a field in the mapping that no longer exists in the source object. Also check for a bad gateway error, which points at a connection configured with the wrong instance URL and is fixed by deleting the connection and recreating it rather than editing it.</li>

<li><strong>API destination returns 401 intermittently.</strong> The connection refreshes its OAuth token when it sees a 401 or 407. Persistent 401s usually mean the connected app has no usable scopes for the client credentials flow, or the run-as user was changed. Check the connected app&#8217;s scopes before you touch anything in AWS.</li>

<li><strong>Everything times out at almost exactly five seconds.</strong> That&#8217;s the API destination timeout, not Salesforce being slow. Move the work behind an inbound platform event so the HTTP call returns immediately and the processing happens asynchronously in the org.</li>

<li><strong>Unexplained API limit exhaustion.</strong> Sum the scheduled AppFlow runs against page counts before blaming a rogue integration user. A frequent flow across several objects adds up faster than most people expect.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Starting the relay before associating the partner event bus, and assuming the events queued somewhere.</li>

<li>Treating the 72-hour retention as a comfortable buffer. It&#8217;s three days including a weekend, and an unmonitored relay burns through it without a single alert.</li>

<li>Using change data capture as a replacement for an initial load. It streams changes from the moment you enable it; it doesn&#8217;t backfill history.</li>

<li>Building the whole integration bidirectionally through one mechanism, then discovering you&#8217;ve created an echo loop where a write from AWS fires a change event that triggers another write.</li>

<li>Sizing the AppFlow schedule for freshness without checking what it does to the org&#8217;s API allocation.</li>

<li>Leaving the default 24-hour EventBridge retry policy on an API destination with no dead letter queue.</li>

<li>Provisioning Private Connect for every VPC because it&#8217;s the secure-sounding option, without modelling the per-connection cost across environments.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Build the reconciliation check before you build the second integration path. It&#8217;s the only thing that tells you the whole system is working.</li>

<li>Monitor the partner event source state, not just rule invocations. Invocations go to zero on a quiet day and on a broken day, and the state field distinguishes them.</li>

<li>Give every integration its own connected app and its own integration user. Shared credentials make it impossible to attribute API consumption or revoke one path without breaking three.</li>

<li>Make writes idempotent with external IDs and upsert. Every path here retries, and some retry aggressively.</li>

<li>Version your event payloads. Platform events are immutable once published, and adding a required field later means a new event definition, not an edit.</li>

<li>Keep the AWS-side resource names discoverable from the Salesforce side. Put the flow name or event bus name in the named credential label so a Salesforce admin can see what&#8217;s attached before they change something.</li>

<li>Define the whole thing in Terraform or CloudFormation on the AWS side and in metadata on the Salesforce side, so a sandbox refresh doesn&#8217;t leave you clicking through two consoles from memory.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need MuleSoft to integrate Salesforce with AWS?</h3>



<p class="wp-block-paragraph">No. For the four patterns above, the native services cover it without an integration platform. MuleSoft, Boomi, and similar tools earn their place when you&#8217;re orchestrating across many systems, need transformation logic that lives outside both platforms, or have a governance requirement for a single integration layer. For a two-system integration, they&#8217;re usually more machinery than the problem needs.</p>



<h3 class="wp-block-heading">Does Event Relay use my Salesforce API allocation?</h3>



<p class="wp-block-paragraph">Event delivery counts against your platform event delivery allocation, which is a separate budget from the 24-hour API request limit that REST calls and AppFlow polling draw on. That separation is one of the strongest arguments for the event-driven path when API consumption is already tight in the org.</p>



<h3 class="wp-block-heading">Can I replay Salesforce events that AWS missed?</h3>



<p class="wp-block-paragraph">Within the 72-hour retention window, yes, using replay IDs from a durable subscriber. Beyond it, no. The events are purged and there&#8217;s no recovery path through the event bus. Backfilling from that point means querying the objects directly and using the deleted-records endpoint to catch removals, which is exactly why the reconciliation job matters.</p>



<h3 class="wp-block-heading">Should I use AppFlow or write my own Lambda extractor?</h3>



<p class="wp-block-paragraph">Start with AppFlow. It removes the auth handling, the pagination, and the scheduling, and for standard object pulls into S3 or Redshift it&#8217;s hard to justify writing that yourself. Move to a custom extractor when you need transformation logic AppFlow can&#8217;t express, when you&#8217;re hitting connector limitations on a specific object, or when you want tighter control over API consumption than a fixed schedule allows.</p>



<h3 class="wp-block-heading">How do I send data from AWS back into Salesforce without a Lambda?</h3>



<p class="wp-block-paragraph">EventBridge API destinations call the Salesforce REST API directly using OAuth client credentials from a connected app. Point them at an inbound platform event endpoint so the write returns fast and processing happens asynchronously in the org. You still need a Lambda if you require mTLS or payload transformation beyond what the input transformer handles.</p>



<h3 class="wp-block-heading">Is Private Connect required for a secure Salesforce AWS integration?</h3>



<p class="wp-block-paragraph">Not required. Everything runs over TLS by default. Private Connect removes the public internet from the path entirely, which matters when a regulator or a security review asks you to demonstrate it. It&#8217;s a licensed add-on with a per-connection cost, so model it against your actual VPC count before committing.</p>



<h3 class="wp-block-heading">Why does my EventBridge rule never fire even though events are arriving?</h3>



<p class="wp-block-paragraph">Almost always because the rule is on the default event bus rather than the partner event bus. Partner events land on their own bus and rules are bus-scoped, so a correct pattern on the wrong bus matches nothing and reports nothing.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A Salesforce AWS integration doesn&#8217;t usually break loudly. It drifts. A relay stops, a mapping goes stale, a delete never propagates, and the two systems keep answering questions confidently with different numbers. The services themselves are well built and the setup is genuinely not hard once you know the region goes in uppercase and the bus has to be associated first.</p>



<p class="wp-block-paragraph">So pick the path by direction and shape, wire the failure modes you&#8217;ve accepted into alerts, and build the nightly count comparison before you build anything clever. That one job is worth more than any amount of pipeline polish, because it&#8217;s the only thing in the architecture that can tell you the whole thing is still true.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your Salesforce and AWS setup?</h2>



<p class="wp-block-paragraph">I work with teams on the infrastructure side of CRM integrations, usually where the data is disagreeing and nobody can prove why. Things I can help with:</p>



<ul class="wp-block-list">
<li>Reviewing an existing Salesforce to AWS pipeline and finding where records are being lost between the two.</li>

<li>Standing up Event Relay into EventBridge end to end, including the partner bus association and the monitoring that proves it&#8217;s live.</li>

<li>Designing the AppFlow schedule and object set so it doesn&#8217;t eat the org&#8217;s API allocation.</li>

<li>Building the reconciliation and delete-propagation jobs, with alerting in CloudWatch, Grafana, or whatever you already run.</li>

<li>Writing the write-back path properly: Bulk API jobs, idempotent upserts, dead letter queues, and credential rotation through Secrets Manager.</li>

<li>Working out whether Private Connect is worth the licence in your specific environment, or whether TLS and tighter scoping gets you there.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;ve got a flow that&#8217;s failing, a relay stuck in an error state, or two record counts that won&#8217;t agree, send me the config or the error and I&#8217;ll tell you what I&#8217;d check first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-aws-integration-architecture/">Salesforce AWS Integration Architecture: The Four Paths and What Breaks Silently</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</title>
		<link>https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 17 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Cursor Pagination]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[GraphQL]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Orchestration]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[Step Functions]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=215</guid>

					<description><![CDATA[<p>A GraphQL source can hand you a 200 OK, a populated data block, and a quietly broken column in the same response. Here is how to build a GraphQL data ingestion pipeline on AWS that catches partial errors, respects cost-based rate limits, resumes cleanly from a cursor, and notices when the schema moves under you.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/">Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The row counts looked fine. That was the problem.</p>



<p class="wp-block-paragraph">A nightly sync from a partner&#8217;s GraphQL API had run for weeks without a single failed execution. Green across the board in the console. Then someone asked why one column had been empty since the previous Tuesday. Not sparse. Empty. Every row, every night, for six nights.</p>



<p class="wp-block-paragraph">The API had started returning a field-level error on that one field. Status code 200. A populated <code>data</code> block. An <code>errors</code> array sitting right next to it. The fetcher parsed <code>data</code>, wrote the records to S3, exited zero, and the orchestrator logged a success. Nothing in the stack was wrong, exactly. Everything in the stack was checking the wrong thing.</p>



<p class="wp-block-paragraph">This post covers how to build a GraphQL data ingestion pipeline on AWS that fails honestly. It is organised by failure family rather than by service, because the AWS wiring is the easy part. The hard part is that GraphQL breaks four assumptions that REST ingestion quietly relies on, and every one of them fails silently by default.</p>



<h2 class="wp-block-heading">Why GraphQL ingestion breaks the habits REST taught you</h2>



<p class="wp-block-paragraph">Most ETL code carries three inherited reflexes from years of REST work: the status code tells you whether it worked, the response shape is the server&#8217;s business, and rate limits are counted in requests. GraphQL invalidates all three.</p>



<ul class="wp-block-list">
<li><strong>The status code is not the contract.</strong> The GraphQL over HTTP specification says a server should return 200 even when field errors are raised during execution, because a partial response is still a successful execution.</li>

<li><strong>You own the response shape.</strong> The selection set is written by you, which means schema drift shows up as a query that stops being valid, not as a payload that changes underneath you.</li>

<li><strong>Cost is query-shaped, not endpoint-shaped.</strong> One endpoint can cost almost nothing or blow your entire budget depending on how deeply you nested it.</li>
</ul>



<p class="wp-block-paragraph">Each of those becomes a distinct failure family. Let&#8217;s take them in the order they bite.</p>



<h2 class="wp-block-heading">Failure one: the 200 OK that quietly ate a column</h2>



<p class="wp-block-paragraph">This is the one that costs the most and shows up the latest. GraphQL splits errors into two categories, and they behave completely differently on the wire.</p>



<ul class="wp-block-list">
<li><strong>Request errors</strong> happen before execution starts: syntax errors, validation failures, bad variable coercion. No <code>data</code> key is returned at all. Nothing executed.</li>

<li><strong>Field errors</strong> happen during execution: a resolver threw, a downstream service timed out, a permission check failed. Execution continues, the failed field goes null, and you get <code>data</code> and <code>errors</code> together.</li>
</ul>



<p class="wp-block-paragraph">Field errors are the dangerous ones for ingestion, because they look exactly like success to anything that only inspects the transport layer. Worse, GraphQL&#8217;s null propagation rules mean a single failing non-null field bubbles up to the nearest nullable ancestor, so one broken resolver can hollow out an entire nested object while the surrounding rows land normally.</p>



<p class="wp-block-paragraph">The fix is to make the response body the contract. Classify before you commit anything:</p>



<pre class="wp-block-code"><code>import json, urllib.request

class RequestError(Exception): pass
class FieldError(Exception): pass

def graphql(endpoint, token, query, variables):
    body = json.dumps({"query": query, "variables": variables}).encode()
    req = urllib.request.Request(
        endpoint,
        data=body,
        headers={
            "Content-Type": "application/json",
            "Accept": "application/graphql-response+json, application/json",
            "Authorization": f"Bearer {token}",
        },
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        payload = json.loads(resp.read())

    # The status code told us nothing. The body is the contract.
    if "data" not in payload:
        # Nothing executed. Query or auth problem. Do not retry blindly.
        raise RequestError(payload.get("errors"))

    if payload.get("errors"):
        # Partial response. data is populated but incomplete.
        raise FieldError(payload["errors"])

    return payload["data"]</code></pre>



<p class="wp-block-paragraph">Two things worth explaining. The <code>Accept</code> header asks for <code>application/graphql-response+json</code>, the media type the GraphQL over HTTP spec recommends, which lets a compliant server return a real 4xx for pre-execution failures instead of burying them in a 200. Servers that don&#8217;t support it fall back to <code>application/json</code>, so including both is free.</p>



<p class="wp-block-paragraph">The second is the split between the two exception types. They need different handling. A request error is a bug in your query or a credential problem, and retrying it just burns budget on the same failure. A field error is transient often enough to be worth retrying, but never worth committing.</p>



<h3 class="wp-block-heading">Deciding what &#8220;partial&#8221; means for your data</h3>



<p class="wp-block-paragraph">Failing the whole page on any error is the safe default and the one I reach for first. But it&#8217;s not always right. If you are pulling a hundred fields and one optional enrichment field is flaky, refusing every page forever is its own outage.</p>



<p class="wp-block-paragraph">The workable middle ground is to classify by path. Every error object carries a <code>path</code> array pointing at the exact field that failed. Maintain a small allowlist of paths that are permitted to be null, fail the page on anything outside it, and emit the error count as a CloudWatch metric so a rising trend is visible before someone in finance notices.</p>



<h2 class="wp-block-heading">Failure two: rate limits that never send a 429</h2>



<p class="wp-block-paragraph">Your retry logic almost certainly triggers on 429 and 5xx. Several major GraphQL APIs will throttle you without ever sending one.</p>



<p class="wp-block-paragraph">GitHub&#8217;s GraphQL API is explicit about this: exceeding the primary rate limit returns a 200 response with an error message in the body and <code>x-ratelimit-remaining</code> set to zero. Your Step Functions retrier sees a success. Your Lambda sees no data. The run completes, having ingested nothing, and reports green.</p>



<p class="wp-block-paragraph">The other half of the problem is that request counting is the wrong unit. Shopify&#8217;s GraphQL Admin API and several others price queries by <em>calculated cost</em>, using a leaky bucket. The server statically analyses your query before executing it, charges the bucket the worst-case <em>requested</em> cost, executes, then refunds the difference between requested and <em>actual</em> cost. That has a consequence people miss: a query asking for the maximum connection size gets charged as if every page were full, even when the last page returns three records.</p>



<p class="wp-block-paragraph">The good news is that these APIs tell you the state of the bucket on every response, under the <code>extensions</code> key:</p>



<pre class="wp-block-code"><code>{
  "extensions": {
    "cost": {
      "requestedQueryCost": 72,
      "actualQueryCost": 38,
      "throttleStatus": {
        "maximumAvailable": 1000.0,
        "currentlyAvailable": 962.0,
        "restoreRate": 100.0
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Read those numbers, never hardcode them. Bucket size and restore rate vary by plan tier, and a query that works fine against a large merchant&#8217;s store will fail instantly against a small one. The pattern that holds up is a pre-flight check: before dispatching, compare your estimated cost against <code>currentlyAvailable</code>, and if you&#8217;re short, sleep for the shortfall divided by <code>restoreRate</code>.</p>



<p class="wp-block-paragraph">On APIs that expose their own budget as a queryable field, ask for it in the same round trip rather than spending a second call to find out:</p>



<pre class="wp-block-code"><code>query BudgetCheck {
  rateLimit {
    limit
    cost
    remaining
    resetAt
  }
}</code></pre>



<p class="wp-block-paragraph">Two more things about concurrency. First, if you fan out across Lambda invocations, each worker&#8217;s local view of the bucket is wrong, because the bucket is shared. Multiple workers independently deciding they have headroom is the classic way to get throttled while every individual worker believes it&#8217;s behaving. Either centralise the token accounting or cap concurrency low enough that the arithmetic can&#8217;t go wrong.</p>



<p class="wp-block-paragraph">Second, if the source offers a bulk or async export operation, use it for backfills. Paginating fifty thousand records through the interactive endpoint burns the same budget your incremental syncs need, and it competes with itself.</p>



<h2 class="wp-block-heading">Failure three: cursors that don&#8217;t survive a restart</h2>



<p class="wp-block-paragraph">Most production GraphQL schemas follow the Relay cursor connections convention: <code>first</code> and <code>after</code> arguments, a <code>pageInfo</code> object with <code>hasNextPage</code> and <code>endCursor</code>, and edges carrying opaque per-item cursors.</p>



<pre class="wp-block-code"><code>query Orders($cursor: String) {
  orders(first: 100, after: $cursor) {
    pageInfo {
      hasNextPage
      endCursor
    }
    edges {
      cursor
      node {
        id
        updatedAt
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">The filter argument for incremental sync is not standardised. Some schemas take a query-string filter, some take a typed timestamp argument, some expose nothing at all. Run introspection against the schema and find out rather than guessing, because a wrong argument name is a request error and your run will fail before it starts.</p>



<p class="wp-block-paragraph">The failure here is subtle. Cursors are opaque and their validity is entirely up to the server. Some are stable positional pointers. Some encode a sort key and a snapshot identifier and expire. If a page-47 fetch fails and you retry an hour later with the stored cursor, you may get an error, or you may get results from a shifted window with a hole in the middle. The second outcome is worse, because it looks like it worked.</p>



<p class="wp-block-paragraph">Three habits make restarts safe:</p>



<ol class="wp-block-list">
<li><strong>Checkpoint after the write, not after the fetch.</strong> A cursor recorded before the page lands in S3 is a promise you can&#8217;t keep.</li>

<li><strong>Store a high watermark alongside the cursor.</strong> The cursor is your fast path. The watermark, taken from the newest record you actually committed, is your fallback when the cursor is rejected.</li>

<li><strong>Overlap the watermark deliberately.</strong> Restart a few minutes before the last committed timestamp and deduplicate on the primary key downstream. Re-reading a handful of rows is cheap. Missing rows that arrived during a clock skew is not.</li>
</ol>



<p class="wp-block-paragraph">A DynamoDB item per source stream is enough state for all of this:</p>



<pre class="wp-block-code"><code>{
  "source": "orders",
  "run_id": "01hq7k...",
  "cursor": "eyJsYXN0X2lkIjoxMjM0fQ==",
  "high_watermark": "&lt;updatedAt of newest committed record&gt;",
  "pages_committed": 47,
  "status": "IN_PROGRESS"
}</code></pre>



<p class="wp-block-paragraph">Overlap only works if downstream deduplication is real. Write raw pages with an ingest-time partition and a run identifier in the object key, then let the curated layer resolve duplicates by primary key and latest <code>updatedAt</code>. If you write straight into a merged table with no dedupe step, overlapping restarts will double-count and you&#8217;ll have traded silent loss for silent inflation.</p>



<h2 class="wp-block-heading">Failure four: the schema moved and nobody told you</h2>



<p class="wp-block-paragraph">GraphQL&#8217;s deprecation story is genuinely better than REST&#8217;s. Fields carry a <code>@deprecated</code> directive with a reason string, and introspection exposes it. The problem is that nothing forces you to look.</p>



<p class="wp-block-paragraph">The failure mode is a field that gets deprecated, stops being populated, then eventually gets removed. Stage one is invisible: your query still validates, the field returns null, and your warehouse fills with nulls. Stage two is loud but late: the query becomes invalid and the whole ingest breaks at 3am.</p>



<p class="wp-block-paragraph">Put an introspection diff in CI. Fetch the schema on a schedule, store it as an artifact, and diff against the last known good version. Fail the build on removals or type changes to fields your queries actually select, and warn on new deprecations. Tools in the Apollo and Hasura ecosystems ship schema-diff tooling for this, and rolling your own against the introspection result is not much work either. Either way, the point is that the check runs on a schedule rather than when the pager goes off.</p>



<p class="wp-block-paragraph">The complementary habit is to keep queries narrow. Every field you select is a field that can break you, a field that costs budget, and a field that widens your Glue table. Selecting a whole object because it was convenient during development is a liability you carry indefinitely.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Wiring the GraphQL data ingestion pipeline on AWS</h2>



<p class="wp-block-paragraph">With the failure families understood, the AWS assembly is mostly a set of trade-offs about where the fetch loop lives.</p>



<h3 class="wp-block-heading">Where the fetch loop runs</h3>



<p class="wp-block-paragraph">Step Functions can call an HTTPS endpoint directly with an HTTP Task, no Lambda involved. It uses an EventBridge connection to hold the credentials, which means your token lives in a Secrets Manager secret managed by EventBridge rather than in an environment variable:</p>



<pre class="wp-block-code"><code>"FetchPage": {
  "Type": "Task",
  "Resource": "arn:aws:states:::http:invoke",
  "Parameters": {
    "ApiEndpoint": "https://api.example.com/graphql",
    "Method": "POST",
    "InvocationConfig": {
      "ConnectionArn": "arn:aws:events:region:account-id:connection/name/id"
    },
    "Headers": { "Content-Type": "application/json" },
    "RequestBody": {
      "query.$": "$.query",
      "variables.$": "$.variables"
    }
  },
  "Retry": [{
    "ErrorEquals": [
      "States.Http.StatusCode.429",
      "States.Http.StatusCode.502",
      "States.Http.StatusCode.503",
      "States.Http.StatusCode.504"
    ],
    "IntervalSeconds": 1,
    "BackoffRate": 2,
    "MaxAttempts": 3,
    "JitterStrategy": "FULL"
  }],
  "Next": "InspectBody"
}</code></pre>



<p class="wp-block-paragraph">The execution role needs four permissions: <code>states:InvokeHTTPEndpoint</code>, <code>events:RetrieveConnectionCredentials</code>, and both <code>secretsmanager:GetSecretValue</code> and <code>secretsmanager:DescribeSecret</code> scoped to the EventBridge-managed secret.</p>



<p class="wp-block-paragraph">It&#8217;s a clean pattern, and I like it for small, high-value syncs. Two constraints decide whether it fits. HTTP Task requests time out after 60 seconds, which a deeply nested GraphQL query can exceed without being pathological. And Step Functions reads the response within its standard state payload ceiling of roughly 256 KB, so a page of a hundred rich records will not fit. Also note the retrier above: it fires on HTTP status codes, so it will not catch a throttle that arrives as a 200 with an error in the body. That check has to happen in the state after it.</p>



<p class="wp-block-paragraph">The alternative is a Lambda driver that paginates internally and writes pages straight to S3, returning only the cursor and a count to the state machine. This dodges both limits, gives you a real HTTP client with real backoff, and keeps large payloads out of the workflow&#8217;s state. It costs you per-page retry granularity, which you buy back with the checkpoint record.</p>



<p class="wp-block-paragraph">Distributed Map is worth mentioning for the fan-out case, such as one child execution per tenant or per shard. It gives you high concurrency and a per-item execution history, which is genuinely useful for debugging. Be aware of two things before committing: express child workflows cap at five minutes rather than Lambda&#8217;s fifteen, and practitioners have published benchmarks showing throughput well below what the headline concurrency numbers suggest for small, fast items. For a lot of ingestion work, SQS plus a DynamoDB job table remains the simpler answer.</p>



<p class="wp-block-paragraph">One case genuinely falls outside serverless: sources that require a long-lived session, a persistent websocket subscription, or a synchronisation window longer than fifteen minutes with no resumable cursor. A small always-on worker on a modest VPS from a provider like InterServer or Hetzner, feeding the same S3 landing zone, is less clever and more honest than contorting Lambda around the constraint.</p>



<h3 class="wp-block-heading">The landing zone</h3>



<p class="wp-block-paragraph">Land raw responses first, unmodified, including the <code>errors</code> and <code>extensions</code> keys. Storage is the cheapest thing in this architecture and the raw layer is what lets you reprocess after you discover a parsing bug six weeks later.</p>



<ul class="wp-block-list">
<li><strong>Partition by ingest time, not event time.</strong> Event-time partitioning requires reading the record before you know where to put it, which means you can&#8217;t write until you&#8217;ve parsed. Ingest-time keys let the raw write be dumb and fast.</li>

<li><strong>Put the run identifier in the key.</strong> When you need to unwind one bad run, being able to delete or ignore by prefix beats writing a surgical delete.</li>

<li><strong>Define Glue tables explicitly rather than crawling.</strong> You wrote the selection set, so you already know the schema. A crawler inferring types from JSON will surprise you the first time an optional field is absent from an entire page.</li>

<li><strong>Convert to columnar in the curated layer.</strong> Athena over raw JSON works and is fine for debugging. Parquet is what you want for anything anyone queries repeatedly.</li>
</ul>



<p class="wp-block-paragraph">If your source pushes rather than being polled, such as webhook deliveries carrying GraphQL payloads, Amazon Data Firehose (renamed from Kinesis Data Firehose, with APIs and IAM policies unchanged) will buffer, convert to Parquet and partition on the way to S3 without you running a consumer. For scheduled pull-based syncs it adds a hop you don&#8217;t need.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Runs succeed but a column is all null.</strong> Field errors. Pull a raw response from the landing zone and look at the <code>errors</code> array. The <code>path</code> tells you exactly which field.</li>

<li><strong>Runs succeed but ingest zero rows.</strong> Likely a throttle returned as a 200. Log the response body and the rate-limit headers on empty results, always.</li>

<li><strong>Intermittent failures partway through pagination.</strong> Cursor expiry. Fall back to the watermark and check whether the source documents a cursor lifetime.</li>

<li><strong>Query fails immediately with no <code>data</code> key.</strong> Request error. Validate against current introspection; something in the schema moved.</li>

<li><strong>Throttled despite staying under the documented limit.</strong> Either concurrent workers are sharing a bucket they each think they own, or you&#8217;re being charged requested cost on near-empty pages.</li>

<li><strong>HTTP Task fails on large pages.</strong> The response exceeds the state payload ceiling. Reduce page size or move the fetch into Lambda.</li>

<li><strong>Row counts drift upward over time.</strong> Overlap without deduplication. Check that the curated layer actually resolves on the primary key.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating a 200 as success and never reading the body.</li>

<li>Retrying request errors. They are deterministic; retrying just spends budget to fail identically.</li>

<li>Hardcoding bucket sizes and restore rates that differ per plan tier and change without notice.</li>

<li>Checkpointing the cursor before the data is durable.</li>

<li>Requesting the maximum page size on every call while being charged the worst-case cost for it.</li>

<li>Selecting entire objects because it was easier during development.</li>

<li>Alerting only on execution failure, so a job that succeeds while ingesting nothing never pages anyone.</li>

<li>Overlapping restart windows with no downstream deduplication.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Classify every response into clean, partial, or request-failed before anything downstream sees it.</li>

<li>Persist raw responses with <code>errors</code> and <code>extensions</code> intact, then parse from the raw layer.</li>

<li>Alarm on row-count deltas and partial-error rate, not just on execution status.</li>

<li>Read the throttle state the API hands you and back off before you hit the wall, not after.</li>

<li>Keep both a cursor and a watermark, and know which one is authoritative on restart.</li>

<li>Diff the schema on a schedule so deprecations reach you before they reach your data.</li>

<li>Hold credentials in Secrets Manager or an EventBridge connection, never in a Lambda environment variable.</li>

<li>Trace the pipeline end to end. CloudWatch covers the basics; Grafana or Datadog earn their keep once you&#8217;re correlating child executions across a fan-out.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can Step Functions call a GraphQL API without a Lambda function?</h3>



<p class="wp-block-paragraph">Yes. An HTTP Task using the <code>arn:aws:states:::http:invoke</code> resource posts your query directly, with an EventBridge connection handling authentication. It works well for small pages. The 60-second request timeout and the state payload size ceiling are what push most real ingestion work back into Lambda.</p>



<h3 class="wp-block-heading">How do I detect GraphQL errors when the status code is always 200?</h3>



<p class="wp-block-paragraph">Inspect the body. If there&#8217;s no <code>data</code> key, nothing executed and you have a request error. If <code>data</code> is present alongside a non-empty <code>errors</code> array, you have a partial response and the data is incomplete. Only a response with <code>data</code> and no errors is unambiguously clean.</p>



<h3 class="wp-block-heading">Does AWS AppSync fit into this?</h3>



<p class="wp-block-paragraph">Only if you&#8217;re on the serving side. AppSync is a managed GraphQL API for exposing your own data, so it&#8217;s what you&#8217;d reach for if you wanted clients to push events in via mutations. For pulling data out of somebody else&#8217;s GraphQL API, AppSync isn&#8217;t part of the picture.</p>



<h3 class="wp-block-heading">Should I store raw GraphQL responses or parsed records in S3?</h3>



<p class="wp-block-paragraph">Both, in separate layers. Raw responses are your audit trail and your reprocessing path, and they&#8217;re the only way to answer &#8220;was this field null in the source, or did my parser drop it?&#8221; weeks after the fact. Parsed and columnar output goes in a curated prefix that analysts query.</p>



<h3 class="wp-block-heading">How do I run an incremental sync from a GraphQL API?</h3>



<p class="wp-block-paragraph">Find the schema&#8217;s filter argument for a modification timestamp using introspection, pass your stored watermark into it, and page forward with cursors. Overlap the window slightly on each run and deduplicate on the primary key downstream. If the schema exposes no such filter, you&#8217;re limited to full pulls or webhook-driven change capture.</p>



<h3 class="wp-block-heading">Why am I throttled when I&#8217;m making very few requests?</h3>



<p class="wp-block-paragraph">Cost-based APIs charge for query complexity, not call count. A single deeply nested query pulling a connection inside a connection can consume a large fraction of the bucket by itself. Read the cost figures the API returns and flatten the query if the requested cost is consistently much higher than the actual cost.</p>



<h3 class="wp-block-heading">Is GraphQL a worse ingestion source than REST?</h3>



<p class="wp-block-paragraph">Different, not worse. You control the payload, which cuts transfer and lets you avoid the N+1 fan-out REST forces on nested data. Deprecation metadata is machine-readable, which REST rarely offers. The cost is that error signalling and rate limiting both moved into the body, so naive clients fail silently instead of loudly.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A GraphQL data ingestion pipeline on AWS fails differently from a REST one, and the difference is almost entirely about where the truth lives. In REST, the transport tells you what happened. In GraphQL, the transport tells you the request arrived, and the body tells you what happened. Every failure family in this post traces back to that: partial errors hidden behind a 200, throttles hidden behind a 200, cursors that fail by returning the wrong rows rather than an error.</p>



<p class="wp-block-paragraph">So build the pipeline so that a green run means something. Parse before you commit, alarm on row deltas rather than exit codes, and keep the raw responses so you can prove what the source actually said. Everything else is ordinary AWS plumbing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with a GraphQL ingestion pipeline?</h2>



<p class="wp-block-paragraph">This is the kind of work I take on regularly. Specifically:</p>



<ul class="wp-block-list">
<li>Auditing an existing GraphQL sync for silent partial-error loss and telling you which columns are already affected</li>

<li>Building the fetch, checkpoint and land loop in Step Functions and Lambda with cursors that survive restarts</li>

<li>Fitting a cost-aware throttle to whatever budget model your source actually uses, including multi-worker coordination</li>

<li>Designing the raw and curated S3 layout, Glue table definitions and Athena partitioning so queries stay cheap</li>

<li>Adding introspection diffing to CI so schema deprecations surface in a pull request, not in a dashboard</li>

<li>Wiring CloudWatch metrics and alarms on row-count deltas and partial-error rates, so a green run is actually green</li>
</ul>



<p class="wp-block-paragraph">Send me a run log, a state machine definition, or a raw response with an <code>errors</code> array in it, and I&#8217;ll tell you what it&#8217;s costing you.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/">Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Streaming Shopify Events into AWS Without Losing Orders</title>
		<link>https://john-nessime.com/blog/devops/streaming-shopify-events-into-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 16 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon SQS]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Dead Letter Queue]]></category>
		<category><![CDATA[DynamoDB]]></category>
		<category><![CDATA[Ecommerce Analytics]]></category>
		<category><![CDATA[Event-Driven Architecture]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Partner Event Source]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Reliability Engineering]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[Terraform]]></category>
		<category><![CDATA[Webhooks]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=221</guid>

					<description><![CDATA[<p>Wiring Shopify webhooks into Amazon EventBridge takes an afternoon. Keeping every order is the hard part. A walk through the five failure families that actually bite when streaming Shopify events into AWS: the partner source that silently drops everything, duplicate and out-of-order deliveries, rule patterns that match nothing, targets that fail without a dead-letter queue, and the 64 KB metering rule that quietly inflates the bill.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/streaming-shopify-events-into-aws/">Streaming Shopify Events into AWS Without Losing Orders</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The partner event source in the EventBridge console said <code>Pending</code>. It had said <code>Pending</code> for six days.</p>



<p class="wp-block-paragraph">Nobody noticed, because nothing errored. No 5xx in a log. No failed delivery in Shopify&#8217;s dashboard. No alarm. Shopify had been publishing order events the entire time, and AWS had been throwing every single one of them on the floor.</p>



<p class="wp-block-paragraph">That behaviour is documented, in one short note in the AWS docs: events published to a partner event source that has not been associated with an event bus are dropped immediately and are not persisted at rest. There is no retry for that. There is no buffer. The events are gone, and the only way to get the data back is to go ask the Shopify Admin API for it after the fact.</p>



<p class="wp-block-paragraph">That is the shape of most of the pain in this integration. Streaming Shopify events into AWS is easy to stand up and easy to get quietly wrong, and every one of the quiet failures looks identical from the outside: everything is green, and some of your data isn&#8217;t there.</p>



<p class="wp-block-paragraph">This post walks the five failure families that actually cost you records, plus the reconciliation layer that most teams only build after the first incident. It assumes you can read a rule pattern and an IAM policy. It does not assume you have shipped this before.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">What the pipe actually looks like</h2>



<p class="wp-block-paragraph">Four moving parts, and only two of them live in your account.</p>



<ol class="wp-block-list"><li>A Shopify app holds the webhook subscriptions. Each subscription has a topic and a delivery method. For this path the delivery method is EventBridge and the address is an ARN, not a URL.</li><li>Shopify creates a <strong>partner event source</strong> inside your AWS account, in the region you nominated.</li><li>You associate that source with a <strong>partner event bus</strong>. This is the step everyone forgets.</li><li>Rules on that bus match events and push them at targets: Lambda, SQS, Step Functions, Firehose, whatever fits.</li></ol>



<p class="wp-block-paragraph">The ARN trips people up more than anything else in the setup. Shopify wants the <em>event source</em> ARN, not the event bus ARN. They look similar and only one of them works:</p>



<pre class="wp-block-code"><code># Correct - the event source ARN. Note the empty account field.
arn:aws:events:eu-west-1::event-source/aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;

# Wrong - this is the bus, and Shopify will reject it
arn:aws:events:eu-west-1:123456789012:event-bus/aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;</code></pre>



<p class="wp-block-paragraph">Associating the source is a single call, and both the name and the source name are the same string:</p>



<pre class="wp-block-code"><code># Create the partner event bus that accepts the source
aws events create-event-bus 
  --name "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;" 
  --event-source-name "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;" 
  --region eu-west-1

# Confirm it flipped from PENDING to ACTIVE
aws events describe-event-source 
  --name "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;" 
  --region eu-west-1</code></pre>



<p class="wp-block-paragraph">The same architecture applies if you are not on Shopify. BigCommerce and commercetools both publish to EventBridge as partner sources, and the failure families below are identical because they come from EventBridge&#8217;s semantics, not the store&#8217;s.</p>



<h2 class="wp-block-heading">Failure family one: events that never existed</h2>



<p class="wp-block-paragraph">This is the one from the opening, and it is the most expensive because it is completely silent on both sides.</p>



<p class="wp-block-paragraph">Shopify considers the delivery successful. It handed the event to the partner source, which is its contract. AWS considers nothing to have happened, because an unassociated source has no bus to write to, and EventBridge does not persist events at rest before a bus exists. Your CloudWatch metrics show nothing, because metrics are emitted per bus and per rule, and you have neither.</p>



<p class="wp-block-paragraph">The same class of hole opens up in two other ways:</p>



<ul class="wp-block-list"><li><strong>Region mismatch.</strong> The source is created in the region you gave Shopify. Your bus, your rules, your targets and your dead-letter queues all have to be in that region. A rule in the right account but the wrong region matches nothing, forever, without complaint.</li><li><strong>Environment drift.</strong> A staging store pointed at a production source, or a source created against an account ID that belonged to an old sandbox. Nothing errors. Events just land somewhere you are not looking.</li></ul>



<p class="wp-block-paragraph">The fix is boring and it works: treat the source state as a monitored asset. A scheduled job that calls <code>describe-event-source</code> and alarms if <code>State</code> is anything other than <code>ACTIVE</code> costs you twenty minutes and covers the entire failure family. Put it next to your other synthetic checks, not inside the pipeline it is watching.</p>



<p class="wp-block-paragraph">The second half of that check is a heartbeat on volume. If a bus that normally sees a few thousand events a day sees zero for an hour, that is an incident even when every component reports healthy. Alarm on <code>MatchedEvents</code> hitting zero, not just on errors.</p>



<h2 class="wp-block-heading">Failure family two: events that arrive twice, or backwards</h2>



<p class="wp-block-paragraph">EventBridge is at-least-once. Shopify&#8217;s webhooks are at-least-once. Neither one promises ordering. Put those together and you get two distinct bugs that people usually try to fix with one patch.</p>



<p class="wp-block-paragraph">The duplicate is the obvious one. The same <code>orders/create</code> arrives twice, and if your handler posts to a fulfilment provider or sends a customer email, you have just done it twice. The dedupe key is sitting in the envelope: Shopify puts <code>X-Shopify-Webhook-Id</code> into <code>detail.metadata</code>, and it identifies the delivery. Write it into DynamoDB with a conditional put and a TTL of a few days, and drop the event if the write fails.</p>



<p class="wp-block-paragraph">The out-of-order case is the one that costs you money quietly. An <code>orders/updated</code> carrying a cancelled status arrives before the <code>orders/updated</code> carrying the address change, and your database ends up holding the older state because it was written last. Nothing failed. The row is just wrong, and it will stay wrong until someone complains.</p>



<p class="wp-block-paragraph">The envelope carries what you need for this too. <code>detail.metadata</code> includes <code>X-Shopify-Triggered-At</code>, and the resource in <code>detail.payload</code> carries its own <code>updated_at</code>. Compare before you write, and refuse to apply an update whose timestamp is older than the one already stored.</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow"><p>The dedupe key stops you from doing the work twice. The version check stops you from doing the work backwards. They solve different problems and you need both.</p></blockquote>



<p class="wp-block-paragraph">One thing you can skip on this path: HMAC verification. On the HTTPS delivery method you must verify the signature, because anyone can POST to your endpoint. On the EventBridge path, only the partner account behind the event source is permitted to publish to that bus, and the AWS docs are explicit that adding your own resource policy to a partner bus is rejected. The signature header still rides along in the metadata, but the trust boundary is enforced by AWS rather than by your code.</p>



<h2 class="wp-block-heading">Failure family three: rules that match nothing</h2>



<p class="wp-block-paragraph">Every Shopify event, regardless of topic, arrives with the same <code>detail-type</code>. That single fact invalidates the routing instinct most people bring from AWS service events.</p>



<pre class="wp-block-code"><code>{
  "version": "0",
  "id": "1b8e2e75-b771-e964-f0e6-fbca6a21dad8",
  "detail-type": "shopifyWebhook",
  "source": "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;",
  "account": "123456789012",
  "time": "2022-07-02T12:47:58Z",
  "region": "eu-west-1",
  "resources": [],
  "detail": {
    "payload": {
      "id": 1234567890,
      "title": "Columbia Las Hermosas"
    },
    "metadata": {
      "Content-Type": "application/json",
      "X-Shopify-Topic": "products/update",
      "X-Shopify-Shop-Domain": "example.myshopify.com",
      "X-Shopify-Hmac-SHA256": "...",
      "X-Shopify-Webhook-Id": "...",
      "X-Shopify-API-Version": "...",
      "X-Shopify-Triggered-At": "2022-07-02T12:47:57.989779121Z"
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Two things to take from that envelope. The resource body is nested under <code>detail.payload</code>, not at the top of <code>detail</code>, so a pattern copied from an HTTPS handler will match nothing. And the topic lives in <code>detail.metadata</code>, which is where all your routing has to happen.</p>



<pre class="wp-block-code"><code>// Exact topic match
{
  "detail-type": ["shopifyWebhook"],
  "detail": {
    "metadata": {
      "X-Shopify-Topic": ["orders/create"]
    }
  }
}

// Every orders topic, one rule
{
  "detail-type": ["shopifyWebhook"],
  "detail": {
    "metadata": {
      "X-Shopify-Topic": [{ "prefix": "orders/" }]
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Do not deploy a pattern you have not tested against a real envelope. <code>test-event-pattern</code> answers in a second and saves an afternoon:</p>



<pre class="wp-block-code"><code>aws events test-event-pattern 
  --event-pattern file://pattern.json 
  --event file://sample-event.json</code></pre>



<h3 class="wp-block-heading">One rule or thirty?</h3>



<p class="wp-block-paragraph">There is a real argument for a single catch-all rule that pushes everything into one queue and lets your consumer branch on the topic. It is less infrastructure, it deploys faster, and adding a topic does not require a Terraform run.</p>



<p class="wp-block-paragraph">What you give up is per-topic visibility. <code>MatchedEvents</code>, <code>FailedInvocations</code> and the dead-letter queue are all scoped to the rule. Collapse thirty topics into one rule and you can no longer tell that inventory events stopped three days ago, because the aggregate number still looks fine.</p>



<p class="wp-block-paragraph">The split I reach for first: a dedicated rule for each topic that touches money or fulfilment, and one catch-all for everything else. You get precise alarms where the cost of being wrong is high and low overhead everywhere else.</p>



<h2 class="wp-block-heading">Failure family four: events that arrive and die at the target</h2>



<p class="wp-block-paragraph">By default EventBridge keeps retrying a failed target invocation for up to a day, with exponential backoff and jitter. That is generous, and it is also the reason people assume they do not need a dead-letter queue. They do, for two reasons.</p>



<p class="wp-block-paragraph">First, a whole class of errors gets <em>no</em> retries at all. Missing permissions on the target, a target that no longer exists, an address that will not resolve. EventBridge does not retry those, because retrying cannot help. It sends them straight to the DLQ if one is configured, and drops them if one is not.</p>



<p class="wp-block-paragraph">Second, a day of retries is not much when the failure is a bad deploy discovered on a Friday evening.</p>



<pre class="wp-block-code"><code>aws events put-targets 
  --rule shopify-orders-create 
  --event-bus-name "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;" 
  --targets '[{
    "Id": "order-processor",
    "Arn": "arn:aws:lambda:eu-west-1:123456789012:function:order-processor",
    "RetryPolicy": {
      "MaximumRetryAttempts": 20,
      "MaximumEventAgeInSeconds": 3600
    },
    "DeadLetterConfig": {
      "Arn": "arn:aws:sqs:eu-west-1:123456789012:shopify-orders-dlq"
    }
  }]'</code></pre>



<p class="wp-block-paragraph">Lowering the retry window is deliberate here. Twenty-four hours of retries against a genuinely broken consumer buys you nothing and hides the problem; a shorter window pushes failures into the DLQ where they are visible and countable.</p>



<h3 class="wp-block-heading">The DLQ permission trap</h3>



<p class="wp-block-paragraph">This one catches almost everyone who manages infrastructure as code. Configure a DLQ through the console and AWS attaches the queue policy for you. Configure it through <code>PutTargets</code> — which is what Terraform, CloudFormation and the CLI all do — and you must attach it yourself. Miss it, and you have a dead-letter queue that cannot receive dead letters.</p>



<pre class="wp-block-code"><code>{
  "Sid": "Dead-letter queue permissions",
  "Effect": "Allow",
  "Principal": { "Service": "events.amazonaws.com" },
  "Action": "sqs:SendMessage",
  "Resource": "arn:aws:sqs:eu-west-1:123456789012:shopify-orders-dlq",
  "Condition": {
    "ArnEquals": {
      "aws:SourceArn": "arn:aws:events:eu-west-1:123456789012:rule/shopify-orders-create"
    }
  }
}</code></pre>



<p class="wp-block-paragraph">The metric that catches this is <code>InvocationsFailedToBeSentToDlq</code>. If it is ever non-zero, your safety net has a hole in it and events are being lost at the exact moment you were counting on it. Alarm on it at a threshold of one. It only reports when it is non-zero, so it costs nothing the rest of the time.</p>



<p class="wp-block-paragraph">Two more constraints worth knowing before you design around a DLQ: it must be a standard SQS queue, not FIFO, and it must live in the same region as the rule. Each message carries the error code, the exhausted retry condition, the retry count and both ARNs as message attributes, which is usually enough to triage without opening the payload.</p>



<h2 class="wp-block-heading">Failure family five: the bill</h2>



<p class="wp-block-paragraph">EventBridge does not meter one event as one event. It meters in 64 KB chunks, so an event larger than that bills as multiple events. Rates change and vary by region, so check the current pricing page rather than trusting any number you read in a blog post, but the mechanism is stable and it is what determines your bill.</p>



<p class="wp-block-paragraph">This matters more for commerce than for most event sources. A product update is small. An order with thirty line items, per-item discount allocations, tax lines, shipping lines, note attributes and a stack of metafields is not. Wholesale and subscription stores routinely produce order payloads that cross the chunk boundary, and the same order updated eight times through its lifecycle multiplies that.</p>



<p class="wp-block-paragraph">Three levers, roughly in order of how much they return:</p>



<ul class="wp-block-list"><li><strong>Trim at the subscription.</strong> Shopify&#8217;s webhook subscription API lets you restrict which fields are included in the payload and which metafield namespaces come along. Fields you never read cost you at ingestion, at archive and again at replay. This is the only lever that stops paying for the data before it enters AWS.</li><li><strong>Subscribe to fewer topics.</strong> Broad topics like <code>orders/updated</code> fire on changes you do not care about. If you only act on fulfilment state, subscribe to the fulfilment topics instead of filtering a firehose after you have paid for it.</li><li><strong>Archive selectively, and set retention.</strong> Archives bill for processing, for storage and again for replay. An archive with no retention period grows forever. Archive the topics you would genuinely replay and let the rest go.</li></ul>



<p class="wp-block-paragraph">One structural limit to design around: EventBridge caps the total size of a single event. A payload that exceeds it does not get truncated in a helpful way — the publish fails. Trimming at the subscription protects you here as well as on cost.</p>



<h2 class="wp-block-heading">The layer nobody builds until they need it</h2>



<p class="wp-block-paragraph">Archive and replay is genuinely useful, and it is also routinely misunderstood. Replay re-delivers events that <em>reached the bus</em>. It does nothing at all for the failure family at the top of this post, where the events never reached the bus in the first place. Replay fixes bugs in your consumer. It does not fix gaps in your ingestion.</p>



<p class="wp-block-paragraph">For that you need reconciliation: a scheduled job that queries the Shopify Admin API for resources changed since a stored watermark and compares them against what you hold. It is unglamorous, it is the thing that catches the outage you did not know about, and it is worth building before you need it rather than during the incident.</p>



<ul class="wp-block-list"><li>Run it hourly for orders and fulfilments, daily for products and customers. The cadence should track how expensive being wrong is, not how much data there is.</li><li>Store a watermark per topic and advance it only after a successful full pass. A partial pass that advances the watermark creates the exact gap you built the job to find.</li><li>Compare counts first, records second. A count mismatch is cheap to compute and tells you whether to bother with the expensive comparison.</li><li>Emit the drift as a metric, not just a log line. &#8220;Orders in Shopify but not in our store, last hour&#8221; is a graph worth putting on a dashboard, and it should normally read zero.</li></ul>



<p class="wp-block-paragraph">The reconciliation worker does not need to live in Lambda. It is a long, paginated, rate-limited crawl, which is an awkward fit for a function timeout and a comfortable fit for a small VPS you already run. If you have a box at InterServer or Hetzner sitting there for other jobs, a cron entry and a script is a perfectly respectable answer.</p>



<h2 class="wp-block-heading">Troubleshooting by symptom</h2>



<p class="wp-block-paragraph">Work these in order. Each one is cheap and rules out a whole branch.</p>



<h3 class="wp-block-heading">Nothing is arriving at all</h3>



<ol class="wp-block-list"><li>Run <code>describe-event-source</code>. If <code>State</code> is not <code>ACTIVE</code>, stop here. Everything published so far is gone and you need the reconciliation path.</li><li>Confirm the region of the bus matches the region in the source ARN.</li><li>List your webhook subscriptions through the Admin API and confirm the address is the event-source ARN, not the bus ARN.</li><li>Confirm the subscriptions belong to the app whose access token you are using. Registering with a token from a different app is a common and confusing dead end.</li><li>Check <code>MatchedEvents</code> on the bus with no rule dimension. Non-zero means events are landing and your rules are the problem, not the plumbing.</li></ol>



<h3 class="wp-block-heading">Some topics arrive, order or customer topics do not</h3>



<p class="wp-block-paragraph">This is almost always scopes rather than infrastructure. Order and customer topics sit behind protected customer data access, which is a separate approval in the app configuration on top of the read scopes. Without it, product events flow perfectly and order events silently do not — which looks exactly like a broken rule and is not.</p>



<h3 class="wp-block-heading">The rule matches but the target does nothing</h3>



<p class="wp-block-paragraph">Compare <code>MatchedEvents</code> against <code>SuccessfulInvocationAttempts</code> on the rule. A gap sends you to <code>FailedInvocations</code> and to the DLQ. Check the target&#8217;s resource policy, and check <code>InvocationsFailedToBeSentToDlq</code> before you trust that the DLQ is catching anything.</p>



<h3 class="wp-block-heading">Events arrive, but late</h3>



<p class="wp-block-paragraph">Look at <code>ThrottledRules</code> and at <code>IngestionToInvocationSuccessLatency</code>. Sustained throttling usually means an invocation quota rather than a rule problem, and it shows up first during flash sales, which is the worst possible time to discover it. Load-test the path before a peak event, not after.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list"><li>Creating the partner event source and never associating it with a bus. Silent, total, unrecoverable data loss for the whole window.</li><li>Registering the event bus ARN instead of the event source ARN, then debugging Shopify&#8217;s rejection for an hour.</li><li>Writing rule patterns against the resource shape from an HTTPS webhook, forgetting that the body sits under <code>detail.payload</code>.</li><li>Routing on <code>detail-type</code>. Every Shopify event carries the same one, so a pattern that matches on it alone matches everything.</li><li>Configuring a DLQ through Terraform without the queue policy, and only finding out when you needed it.</li><li>Assuming replay covers ingestion gaps. It replays what reached the bus and nothing else.</li><li>Deduplicating on the resource ID instead of the webhook ID, so legitimate subsequent updates get discarded as duplicates.</li><li>Skipping reconciliation because the pipeline &#8220;works&#8221;. It works right up until it doesn&#8217;t, and that is precisely when you need the other path.</li></ul>



<h2 class="wp-block-heading">Best practices for streaming Shopify events into AWS</h2>



<ul class="wp-block-list"><li>Alarm on the event source state and on <code>MatchedEvents</code> reaching zero. Absence of events is a signal, and it is the only signal you get for the worst failure.</li><li>Dedupe on <code>X-Shopify-Webhook-Id</code> and version-check on <code>X-Shopify-Triggered-At</code>. Two mechanisms, two problems.</li><li>Give every target a DLQ and a retry window you chose deliberately, rather than inheriting the default.</li><li>Keep dedicated rules for money and fulfilment topics so their metrics stay legible; batch the rest behind a catch-all.</li><li>Trim payloads at the Shopify subscription rather than in a Lambda. Filtering after ingestion means you already paid for the bytes.</li><li>Define the whole thing in Terraform or CloudFormation, including the queue policies. This stack has too many one-time console clicks to survive being hand-built twice.</li><li>Point your observability platform at the same bus. Datadog and New Relic are both EventBridge partners, so business events and infrastructure telemetry can share one pipeline instead of two.</li><li>Build reconciliation before your first peak trading period, not after your first missing-order ticket.</li></ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I still need to verify the HMAC signature on the EventBridge path?</h3>



<p class="wp-block-paragraph">No. Only the partner account behind the event source can publish to a partner event bus, and AWS actively rejects attempts to add your own resource policy granting anyone else access. The signature header is still present in the metadata, but the trust boundary is enforced by AWS rather than by your handler. On the HTTPS delivery method, verification remains mandatory.</p>



<h3 class="wp-block-heading">Should I use EventBridge or plain HTTPS webhooks?</h3>



<p class="wp-block-paragraph">HTTPS is simpler, works with any host, and is easier to debug because you can curl your own endpoint. It also puts you on the hook for absorbing burst traffic within a short response deadline, and for keeping the endpoint up well enough that Shopify does not remove the subscription after persistent failures. EventBridge moves that burst absorption to AWS and gives you native fan-out. If your consumers already live in AWS, the operational maths favours EventBridge. If they do not, a dedicated reliability layer such as Hookdeck in front of an HTTPS endpoint is a reasonable alternative and a much smaller change.</p>



<h3 class="wp-block-heading">Can I use one partner event source for multiple stores?</h3>



<p class="wp-block-paragraph">Events from every shop that installed your app flow through the source associated with that app, and the shop is identified by <code>X-Shopify-Shop-Domain</code> in the metadata. You can route per-shop with rule patterns matching that field. For genuine tenant isolation — separate accounts, separate blast radius — you want separate apps and separate sources, because a single bus is a single failure domain.</p>



<h3 class="wp-block-heading">Why do I get duplicate order events even though nothing failed?</h3>



<p class="wp-block-paragraph">Because at-least-once means exactly that. Duplicates are normal operation, not a fault to be investigated. Separately, an order genuinely does change several times shortly after creation — payment capture, risk assessment, post-purchase upsells — so several <code>orders/updated</code> events for one order are expected and are not duplicates at all. Deduplicate on the webhook ID to tell the two apart.</p>



<h3 class="wp-block-heading">What happens to events published while my consumer is broken?</h3>



<p class="wp-block-paragraph">They reach the bus, match your rules, and EventBridge retries the target within your configured window. Once that window is exhausted they go to the DLQ if you have one and are discarded if you do not. The events themselves are not lost at the bus level as long as the source is associated — this is the failure family you can actually engineer your way out of.</p>



<h3 class="wp-block-heading">Can I archive and replay Shopify events?</h3>



<p class="wp-block-paragraph">Yes, with an archive on the partner event bus and an event pattern controlling what gets archived. Budget for three separate charges — processing into the archive, storage while it sits there, and the replay itself — and always set an explicit retention period, because an archive without one grows indefinitely.</p>



<h3 class="wp-block-heading">Does this work the same way for BigCommerce or commercetools?</h3>



<p class="wp-block-paragraph">The AWS half is identical: partner source, association step, bus, rules, targets, and every failure family in this post. What differs is the envelope shape and how you register subscriptions on the vendor side. The association gap in particular bites the same way regardless of which platform is publishing.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Almost everything about streaming Shopify events into AWS degrades loudly. Targets throw errors, retries show up as metrics, dead letters pile up in a queue you can see. Those are the failures you will handle correctly, because they announce themselves.</p>



<p class="wp-block-paragraph">The one that will actually hurt you is the one that reports success on both sides while dropping every event on the floor. Association state and event volume are the two signals that catch it, and neither one appears on any dashboard by default. Add them on day one, before you write the first rule. Everything else in this post can be fixed after the fact; that one cannot.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a hand with your event pipeline?</h2>



<p class="wp-block-paragraph">Most of my work on this stack is either standing it up properly the first time or working out where records went after someone else stood it up. Things I can help with:</p>



<ul class="wp-block-list"><li>Building the Shopify-to-EventBridge path end to end in Terraform, including the queue policies and retry configuration that the console quietly does for you.</li><li>Auditing an existing pipeline for silent loss: source association, region drift, missing DLQ permissions, rules that have been matching nothing since the day they shipped.</li><li>Designing the idempotency and ordering layer — dedupe store, TTLs, version checks — so replays and duplicates stop corrupting downstream state.</li><li>Writing the reconciliation job against the Admin API, with watermarks, drift metrics and alarms that fire before a customer does.</li><li>Cutting EventBridge spend by trimming payloads at the subscription and rationalising archive retention, without losing anything you actually query.</li><li>Load-testing the whole path ahead of a peak trading period so throttling shows up in a test window rather than on the day.</li></ul>



<p class="wp-block-paragraph">If you have a rule pattern that isn&#8217;t matching, a DLQ that&#8217;s mysteriously empty, or a bill that grew faster than your order volume, send me the pattern, the metric graph or the line item and I&#8217;ll tell you what I&#8217;d look at first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/streaming-shopify-events-into-aws/">Streaming Shopify Events into AWS Without Losing Orders</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Agentforce and AWS: Where the Trust Layer Stops and Your Logs Begin</title>
		<link>https://john-nessime.com/blog/devops/agentforce-aws-architecture-security/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 06 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Agentforce]]></category>
		<category><![CDATA[AI Agents]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[API Gateway]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Einstein Trust Layer]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[MCP]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[PrivateLink]]></category>
		<category><![CDATA[Salesforce]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=155</guid>

					<description><![CDATA[<p>Agentforce and AWS wire together in four standard patterns, and every one of them has a point where Salesforce's guarantees stop and yours start. This traces a single request across each boundary it crosses, covers the Trust Layer default most write-ups get wrong (LLM data masking is disabled for agents), and sets out what changes the moment a callout lands in your own account: retention, audit trail, and user identity that does not travel.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/agentforce-aws-architecture-security/">Agentforce and AWS: Where the Trust Layer Stops and Your Logs Begin</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The architecture review goes fine right up until someone from security asks the only question that matters: when the agent answers a customer&#8217;s question, where exactly does that customer&#8217;s data go, and who keeps a copy?</p>



<p class="wp-block-paragraph">&#8220;Salesforce handles it&#8221; is not an answer. Neither is &#8220;it&#8217;s in our VPC,&#8221; because it usually isn&#8217;t, and the parts that are in your VPC are the parts nobody has threat-modelled.</p>



<p class="wp-block-paragraph">This post traces a single Agentforce request end to end and stops at each boundary it crosses. Agentforce and AWS get wired together in three or four standard patterns, and every one of them has a specific point where Salesforce&#8217;s guarantees stop and yours start. I will cover where that line sits, what changes about the security model when you cross it, and one default that surprises almost everyone.</p>



<h2 class="wp-block-heading">The boundaries a single request crosses</h2>



<p class="wp-block-paragraph">One user turn touches more systems than the diagrams suggest. Roughly:</p>



<ol class="wp-block-list">
<li><strong>Retrieval.</strong> The agent grounds itself in CRM records, Data Cloud objects and knowledge content.</li>

<li><strong>Reasoning.</strong> The request gets classified to a topic, then an action gets chosen. Both steps involve a model call.</li>

<li><strong>Generation.</strong> A prompt is assembled and sent through the LLM gateway to a model provider.</li>

<li><strong>Action.</strong> If the agent decides to do something, it calls out. That callout is where AWS usually enters the picture.</li>
</ol>



<p class="wp-block-paragraph">Steps one to three happen inside Salesforce&#8217;s trust boundary, and the Einstein Trust Layer applies. Step four does not, and it doesn&#8217;t. That sentence is the whole post, but the detail matters.</p>



<h2 class="wp-block-heading">Retrieval: the part that&#8217;s better than you&#8217;d expect</h2>



<p class="wp-block-paragraph">Grounding is where Agentforce gets its context, and the architecturally interesting decision is that vector search lives inside Data Cloud rather than in a separate vector database. Structured records and unstructured content are retrieved from the same layer, which removes a whole class of sync problem you would otherwise be building.</p>



<p class="wp-block-paragraph">The security-relevant property is that retrieval is permission-aware. The agent retrieves as the running user, so sharing rules and field-level security apply. If a rep cannot see an opportunity, the agent acting on their behalf should not be able to summarise it either.</p>



<p class="wp-block-paragraph">Two caveats worth putting in your notes. First, this only holds while retrieval goes through Salesforce; the moment you ground from an external source, you are enforcing access control yourself. Second, if your CRM sharing model is loose, the agent inherits that looseness and makes it much easier to exploit, because a model will cheerfully summarise a thousand records a human would never have opened one by one. Agentforce does not create the over-permissioning problem, it just removes the friction that was hiding it.</p>



<p class="wp-block-paragraph">Where AWS shows up here: if your analytical data lives in Redshift, an Iceberg lake or Databricks, zero-copy federation lets Data Cloud register those tables and query them in place rather than replicating them. Good for cost and duplication, and it moves an access-control decision into the federation configuration, which is a place people forget to audit.</p>



<h2 class="wp-block-heading">The Trust Layer, and the default nobody expects</h2>



<p class="wp-block-paragraph">The Einstein Trust Layer sits between every Salesforce-originated prompt and the model. Its components are well documented: secure grounding, data masking, system policies against prompt injection, the LLM gateway, zero data retention agreements with model providers, toxicity checks on the way back, and an audit trail of the whole journey.</p>



<p class="wp-block-paragraph">Here is the part that gets misreported constantly, including in a lot of otherwise decent write-ups: <strong>LLM data masking is disabled for agents.</strong> Salesforce says so plainly in its own documentation on the Trust Layer architecture and repeats it in the Trailhead module on masking. Masking remains available and configurable for embedded generative features like service replies and work summaries. For Agentforce agents, it is off.</p>



<p class="wp-block-paragraph">The reason is not hard to work out once you think about what an agent does. Masking swaps real values for placeholders. An agent that has to actually perform an action needs the real record ID, the real amount, the real email address, because those become tool arguments. You cannot pass <code>&lt;Person_0&gt;</code> to a Flow and expect it to update a contact. Masking and acting are in genuine tension, and Salesforce resolved it in favour of acting.</p>



<p class="wp-block-paragraph">What this changes in practice:</p>



<ul class="wp-block-list">
<li>Real PII reaches the model provider. The zero data retention commitment is what protects it, not de-identification. Those are different controls with different failure modes: masking is technical and verifiable, a retention agreement is contractual.</li>

<li>If your compliance position was &#8220;no personal data leaves our boundary in identifiable form,&#8221; Agentforce does not satisfy it the way you may have assumed, and this is worth raising before your security review rather than during it.</li>

<li>Controlling exposure moves upstream. What the model sees is now decided by what you ground on and what your sharing model permits, not by a masking filter downstream.</li>
</ul>



<p class="wp-block-paragraph">None of this makes the Trust Layer weak. Zero retention, the gateway, system policies and the audit trail all still apply to agent traffic, and the Trust Layer typically runs several times in a single user turn, once for topic classification and again for generation. It is a real control plane. Just know which control is doing the work.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Connecting Agentforce and AWS: four patterns</h2>



<p class="wp-block-paragraph">There are four ways these two systems talk, and they have different latency, failure and security characteristics.</p>



<h3 class="wp-block-heading">Synchronous outbound: agent action calls AWS</h3>



<p class="wp-block-paragraph">A custom agent action, implemented via External Services with an OpenAPI schema, an Apex invocable method, or a Flow, calls an endpoint you own. Behind it sits API Gateway and Lambda, and often a Bedrock agent or knowledge base.</p>



<p class="wp-block-paragraph">Authentication uses named credentials, which keep the secret out of your Apex and centralise the auth config. Use them; hardcoding a key into an Apex class is the version of this that ends up in a git history.</p>



<p class="wp-block-paragraph">The trade-off is that you have put a network call and a second model invocation inside a conversational turn. Users notice. Budget for it, and set the timeout deliberately rather than inheriting a default.</p>



<h3 class="wp-block-heading">Asynchronous outbound: Event Relay to EventBridge</h3>



<p class="wp-block-paragraph">Salesforce Event Relay delivers platform events from the Salesforce event bus to Amazon EventBridge without code, and EventBridge routes onward to Lambda, S3, Step Functions or a Bedrock agent.</p>



<p class="wp-block-paragraph">This is the right pattern for anything that does not need to complete inside the conversation: enrichment, downstream processing, long-running work. It also fails better, because a retry is a queue concern rather than a user watching a spinner.</p>



<h3 class="wp-block-heading">Inbound: AWS calls Agentforce</h3>



<p class="wp-block-paragraph">The Agent API lets something on the AWS side invoke an Agentforce agent, typically through EventBridge API destinations. That gives you bidirectional multi-agent flows: a Bedrock agent that detects a problem can open a case through Agentforce rather than writing to the Salesforce API directly and bypassing the agent&#8217;s own logic.</p>



<p class="wp-block-paragraph">Worth being deliberate here. An inbound path means an AWS workload can trigger agent reasoning, which means anything that can inject content into that workload can influence a prompt. Treat the payload as untrusted input, because it is.</p>



<h3 class="wp-block-heading">MCP: tools rather than integrations</h3>



<p class="wp-block-paragraph">Agentforce has a native MCP client, so an agent can connect to any MCP-compliant server without a bespoke integration, alongside a server registry intended to let admins govern which agents connect to which tools under central policy.</p>



<p class="wp-block-paragraph">This is the pattern that will cause the most trouble over the next couple of years, and not because the protocol is bad. It is because MCP makes adding a tool trivial, and every tool is a new path by which text from somewhere else reaches your agent&#8217;s context and by which your agent&#8217;s decisions reach someone else&#8217;s system. The registry exists precisely so that &#8220;which tools can this agent reach&#8221; is an administered answer rather than an emergent one. Use it as a control, not a catalogue.</p>



<h3 class="wp-block-heading">Keeping the traffic private</h3>



<p class="wp-block-paragraph">By default these callouts traverse the public internet with TLS. Salesforce Private Connect uses AWS PrivateLink to give you a private path to your own AWS endpoints instead. Whether that is required is a policy question rather than a technical one, but it is much easier to set up at design time than to retrofit after an auditor asks.</p>



<h2 class="wp-block-heading">What changes the moment you cross into AWS</h2>



<p class="wp-block-paragraph">This is the seam, and it is where most real incidents will come from. Four things stop being true simultaneously.</p>



<ul class="wp-block-list">
<li><strong>Zero retention becomes your policy, not Salesforce&#8217;s.</strong> If your Lambda logs its input at INFO, you now have customer PII in CloudWatch Logs with whatever retention that log group happens to have, which by default is forever. Your own debug logging is the single most likely leak path in this entire architecture, and it will not show up in any Salesforce audit report.</li>

<li><strong>The audit trail splits.</strong> Salesforce logs the prompt journey. AWS logs the invocation. Nothing correlates them unless you pass a request identifier through the callout and log it on both sides. Do that from day one; reconstructing an incident across two disconnected audit systems is genuinely unpleasant.</li>

<li><strong>User identity does not travel.</strong> This is the important one. Inside Salesforce the agent acts as the running user and sharing rules apply. Your Lambda runs under an IAM execution role with fixed permissions and no idea who asked. If that role can read a whole DynamoDB table, then every user of the agent can effectively read the whole table through it, regardless of their CRM permissions. That is a classic confused deputy, and it is easy to build without noticing. If the action needs to be scoped per user, pass the identity explicitly and enforce it in your own code.</li>

<li><strong>Model governance forks.</strong> A Bedrock model you invoke from your own Lambda is outside the Trust Layer. Its guardrails, logging and retention are configured in your account and are yours to get right. If you want prompt filtering or PII detection on that leg, you are building or buying it.</li>
</ul>



<p class="wp-block-paragraph">Note the difference between two things people conflate. Salesforce running on Hyperforce, which is built on public cloud infrastructure including AWS, means Salesforce&#8217;s own workloads run there under Salesforce&#8217;s controls. It does not mean your Salesforce data sits in your AWS account. When an agent action calls your Lambda, that is the first moment data lands in infrastructure you control and are responsible for.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Assuming masking protects agent traffic to the model. It does not; check the current documentation yourself rather than taking a blog&#8217;s word for it, including this one.</li>

<li>Logging full request payloads in Lambda during development and never turning it off.</li>

<li>Giving the Lambda execution role broad data access because &#8220;the agent is trusted,&#8221; when the agent is a proxy for every user who can talk to it.</li>

<li>Putting a slow AWS call inside a synchronous agent action when the work did not need to be synchronous.</li>

<li>Not passing a correlation identifier across the boundary, so Salesforce and CloudWatch tell two unlinked stories.</li>

<li>Treating content returned from an external tool or MCP server as trusted context rather than as untrusted input that will be read by a model.</li>

<li>Deploying agents against a permissive sharing model and discovering the blast radius later.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does Agentforce data get used to train models?</h3>



<p class="wp-block-paragraph">Not on the Salesforce leg. Salesforce&#8217;s zero data retention arrangements with model providers mean prompts and responses are not stored or used for training. That commitment covers traffic going through the LLM gateway. It says nothing about a model you invoke yourself from your own AWS account, where the retention and logging configuration is entirely yours.</p>



<h3 class="wp-block-heading">Can I use my own Bedrock model inside Agentforce?</h3>



<p class="wp-block-paragraph">There are two distinct things here. Bedrock-hosted models can be brought into Salesforce through the Einstein and Bedrock integration, in which case they sit behind the Trust Layer. Separately, you can call a Bedrock model or agent from a custom agent action, in which case you are outside the Trust Layer and own the controls. Be clear which one you are building, because they have completely different compliance stories.</p>



<h3 class="wp-block-heading">Sync or async for AWS integrations?</h3>



<p class="wp-block-paragraph">Default to async through Event Relay and EventBridge unless the agent genuinely needs the result to continue the conversation. Synchronous callouts put a network round trip and often a second inference call inside a user&#8217;s turn, and they fail in front of the user rather than into a retry queue.</p>



<h3 class="wp-block-heading">How do I stop the agent seeing data a user shouldn&#8217;t?</h3>



<p class="wp-block-paragraph">Inside Salesforce, fix the sharing model, because retrieval respects it and inherits its flaws. Outside Salesforce, sharing rules do not apply at all: pass the user identity through the callout and enforce authorisation in your own code, or scope the integration to data that is safe for every user of that agent.</p>



<h3 class="wp-block-heading">Do I need PrivateLink for this?</h3>



<p class="wp-block-paragraph">Technically no; callouts work over TLS on the public internet. Practically it depends on your regulatory position and whether your AWS endpoints should be publicly reachable at all. If they should not be, Private Connect over PrivateLink is the supported path and is far cheaper to design in than to add later.</p>



<h3 class="wp-block-heading">What should I log, and where?</h3>



<p class="wp-block-paragraph">On the AWS side, log the correlation identifier, the action name, the outcome and the timing. Do not log the payload. If you need payload-level debugging, gate it behind a flag that is off in production and set an explicit short retention on the log group. Ship the operational metrics somewhere you already watch, whether that is CloudWatch alarms, Grafana or Datadog, because an agent action that silently starts failing looks like nothing at all from the Salesforce side.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">Wiring up Agentforce and AWS is not hard. The patterns are documented, the connectors exist, and a working proof of concept is a day&#8217;s work.</p>



<p class="wp-block-paragraph">What is hard is knowing precisely where Salesforce&#8217;s guarantees end. They end at the callout. Everything before it is covered by the Trust Layer, with the significant exception that masking is off for agents, so what reaches the model is real data protected by a retention agreement rather than by de-identification. Everything after it is yours: your retention, your audit trail, your authorisation, your logs.</p>



<p class="wp-block-paragraph">Draw that line on your architecture diagram before the security review, not during it.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help designing or reviewing this?</h2>



<p class="wp-block-paragraph">I work on the AWS side of Salesforce integrations, which is usually the side nobody owns. Things I can help with:</p>



<ul class="wp-block-list">
<li>Threat-modelling the seam between Agentforce and your AWS account, and writing down what each side actually guarantees.</li>

<li>Building agent actions on API Gateway and Lambda with named credentials, sane timeouts and no payload logging.</li>

<li>Event-driven integration via Event Relay and EventBridge, including retry, dead-letter and replay design.</li>

<li>Scoping IAM execution roles so an agent action cannot become a confused deputy for every user who can reach it.</li>

<li>Correlated observability across Salesforce and CloudWatch so an incident can be reconstructed in one timeline.</li>

<li>Private Connect and PrivateLink setup where callouts should not touch the public internet.</li>
</ul>



<p class="wp-block-paragraph">If you have an integration already built, send me the agent action definition and the Lambda&#8217;s IAM policy. Those two together usually tell the whole story.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/agentforce-aws-architecture-security/">Agentforce and AWS: Where the Trust Layer Stops and Your Logs Begin</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Shopify Sales Dashboard with AWS: Build One That Actually Reconciles</title>
		<link>https://john-nessime.com/blog/technical-guides/shopify-sales-dashboard-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 04 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Business Intelligence]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Ecommerce Analytics]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[Partition Projection]]></category>
		<category><![CDATA[QuickSight]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[SQL]]></category>
		<category><![CDATA[Webhooks]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=142</guid>

					<description><![CDATA[<p>Most Shopify dashboards built on AWS work perfectly for about three weeks, then quietly drift away from the numbers in the Shopify admin. Here is why that happens, and how to design the ingestion, storage and query layers so your totals still reconcile six months in.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/shopify-sales-dashboard-aws/">Shopify Sales Dashboard with AWS: Build One That Actually Reconciles</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone in the finance channel posts two screenshots side by side. On the left, the dashboard you built. On the right, the Shopify admin. The totals don&#8217;t match, and they&#8217;re not off by a rounding error either. They&#8217;re off by enough that nobody wants to use your dashboard for anything that matters.</p>



<p class="wp-block-paragraph">The frustrating part is that the pipeline is fine. Webhooks are arriving, Lambda is running clean, S3 has the files, Athena returns rows. Nothing is broken in the way monitoring understands &#8220;broken&#8221;. The pipeline is faithfully recording something that is no longer true.</p>



<p class="wp-block-paragraph">This post is about building a Shopify sales dashboard with AWS that survives that conversation. Not the wiring, which is well documented and mostly straightforward, but the design decisions that determine whether your numbers still hold up six months in. I&#8217;ll cover the three ingestion paths and when each one is the right call, why append-only pipelines drift, how to lay out S3 and Athena so recomputation is cheap, and what to do when the totals are already wrong.</p>



<h2 class="wp-block-heading">Why a Shopify sales dashboard with AWS drifts from the admin</h2>



<p class="wp-block-paragraph">Here&#8217;s the thing that catches almost everyone: <strong>a Shopify order is not an event, it&#8217;s a mutable record.</strong></p>



<p class="wp-block-paragraph">An event pipeline assumes facts are immutable once written. A payment happened. A shipment left. You append it, you never touch it again, and the sum of the log is the truth. That model is why streaming architectures are so clean, and it&#8217;s exactly wrong for order data.</p>



<p class="wp-block-paragraph">An order created on Monday can be edited on Tuesday, partially refunded on Friday, and fully refunded three weeks later. Every one of those changes belongs, financially, to Monday. If your pipeline appends the <code>orders/create</code> payload and never revisits it, Monday&#8217;s revenue is frozen at the moment of checkout and it will only ever be too high.</p>



<p class="wp-block-paragraph">This is the invisible failure. Nothing alerts. No queue backs up. Your dashboard is confidently wrong, and the gap widens roughly in proportion to your return rate. A store with a two percent return rate takes a long time to notice. A fashion store running thirty percent returns notices in about a month, usually via an angry accountant.</p>



<h3 class="wp-block-heading">The four adjustments that move historical numbers</h3>



<ul class="wp-block-list">
<li><strong>Refunds.</strong> Full or partial. A refund carries its own <code>created_at</code>, which is when the money moved back. The order it belongs to has a different, earlier date. You need both, and which one you attribute to depends on whether finance wants cash-basis or order-basis reporting. Ask before you build.</li>

<li><strong>Order edits.</strong> A merchant adds a line item or adjusts a quantity after the fact. The original payload is now stale. Shopify exposes both the original and the current totals precisely because of this.</li>

<li><strong>Cancellations.</strong> A cancelled order keeps existing in the API. If you filter only on payment status you will happily keep counting it.</li>

<li><strong>Test and draft orders.</strong> Test orders carry a flag marking them as such. Nobody remembers to filter these until a QA run during a quiet week produces a suspicious spike.</li>
</ul>



<p class="wp-block-paragraph">The design consequence is simple to state and annoying to implement: <strong>your pipeline must be able to recompute any past day.</strong> Every storage and partitioning decision below follows from that one requirement.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Getting data out of Shopify: three paths, three trade-offs</h2>



<p class="wp-block-paragraph">Before anything else: new Shopify apps are built on the GraphQL Admin API. The REST Admin API has been designated a legacy API and new public apps must use GraphQL. If you&#8217;re starting fresh, start there. If you inherited a REST integration, it probably still runs, but you&#8217;re on borrowed time and you should plan the migration rather than discover the deadline.</p>



<h3 class="wp-block-heading">Path 1: EventBridge partner event source</h3>



<p class="wp-block-paragraph">Shopify can deliver webhooks straight into an Amazon EventBridge partner event bus in your account. No public endpoint, no API Gateway, no HMAC verification code, because verification only applies to HTTPS deliveries. Shopify&#8217;s own docs confirm EventBridge and Pub/Sub deliveries skip it.</p>



<p class="wp-block-paragraph">You create the source in the Shopify app configuration using your AWS account ID, region and a source name, then associate it with an event bus in the EventBridge console and write rules to route it. The address you register with Shopify is the <em>partner event source</em> ARN, not the event bus ARN. That distinction accounts for a large share of the &#8220;I set it up and nothing arrives&#8221; threads on the Shopify forums.</p>



<p class="wp-block-paragraph">A rule matching everything from the Shopify partner source looks like this. Start broad, then narrow once you&#8217;ve seen the real shape of an event:</p>



<pre class="wp-block-code"><code>{
  "source": [ { "prefix": "aws.partner/shopify.com" } ]
}</code></pre>



<p class="wp-block-paragraph">Send that to an SQS queue with a dead-letter queue attached rather than straight to Lambda. Buffering gives you a replay buffer when a downstream deploy goes wrong, and the DLQ means a bad payload parks itself instead of poisoning the whole rule. This is the path I reach for first for anything already on AWS.</p>



<h3 class="wp-block-heading">Path 2: HTTPS webhooks into API Gateway and Lambda</h3>



<p class="wp-block-paragraph">The conventional route, and the right one if you need webhook delivery outside AWS too, or you want the payloads to pass through something you fully control. The cost is that you now own an internet-facing endpoint and the HMAC verification on it.</p>



<p class="wp-block-paragraph">Verify against the <strong>raw request body</strong>, before any JSON parsing. Re-serialising the payload changes byte-for-byte content and the signature will never match. Use a constant-time comparison so the check doesn&#8217;t leak timing information:</p>



<pre class="wp-block-code"><code>import base64, hashlib, hmac

def verify(raw_body: bytes, header_hmac: str, secret: str) -&gt; bool:
    digest = hmac.new(secret.encode(), raw_body, hashlib.sha256).digest()
    computed = base64.b64encode(digest).decode()
    return hmac.compare_digest(computed, header_hmac)</code></pre>



<p class="wp-block-paragraph">Shopify sends the signature in the <code>X-Shopify-Hmac-SHA256</code> header, base64-encoded, computed with your app&#8217;s client secret over the raw body. Store that secret in Secrets Manager or as an SSM SecureString parameter, not in a Lambda environment variable.</p>



<p class="wp-block-paragraph">Acknowledge fast. Shopify&#8217;s timeout is short and it retries with backoff over a finite window, so a handler that does real work inline will generate a wall of duplicate deliveries during a flash sale, exactly when you can least afford it. Return 2xx immediately, do the work asynchronously.</p>



<h3 class="wp-block-heading">Path 3: scheduled GraphQL bulk pull</h3>



<p class="wp-block-paragraph">Webhooks give you low latency. They do not give you completeness. Anything that fails past its retry window is gone, and Shopify will eventually remove a subscription that keeps failing. That&#8217;s a silent data loss mode with no local symptom at all.</p>



<p class="wp-block-paragraph">So run a scheduled reconciliation pull alongside the stream. Shopify&#8217;s GraphQL bulk operations are built for this: you submit a query, it runs asynchronously, and you fetch a JSONL result file when it finishes. That&#8217;s the right tool for backfills and nightly catch-up, rather than paginating thousands of pages against a points-based rate limiter and getting throttled halfway through.</p>



<p class="wp-block-paragraph">A nightly job that re-pulls the last seven to fourteen days and overwrites those partitions costs almost nothing and quietly fixes every category of drift described above. If you build one thing from this post, build that.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Decide what &#8220;revenue&#8221; means before you write a line of SQL</h2>



<p class="wp-block-paragraph">This is where most reconciliation arguments actually live, and it isn&#8217;t an engineering problem at all until you&#8217;ve had the conversation.</p>



<p class="wp-block-paragraph">Shopify&#8217;s own sales reporting builds total sales from gross sales, minus discounts, minus returns, plus taxes and shipping. Gift card sales sit outside that in a separate finance report. If your dashboard sums order totals and calls it revenue, you have built a different metric with the same name, and it will disagree with the admin forever no matter how good your pipeline is.</p>



<p class="wp-block-paragraph">Write the definition down. Put it in the dashboard as a tooltip. When someone challenges a number, you want the argument to be about the definition, not about whether your infrastructure works.</p>



<h3 class="wp-block-heading">The currency trap</h3>



<p class="wp-block-paragraph">If the store sells in more than one currency, the money fields split in two. Shopify exposes totals as a set containing both <code>shop_money</code> and <code>presentment_money</code>: the amount in the store&#8217;s base currency, and the amount the customer actually saw and paid.</p>



<p class="wp-block-paragraph">Sum the presentment amounts across a multi-currency store and you get a number with no meaning at all, euros and yen added together as if they were the same unit. For a single reporting figure you want the shop-currency side. Keep the presentment amount and its currency code in the table anyway, because the day someone asks &#8220;how much did we actually sell in Germany&#8221;, you&#8217;ll want it and it is painful to backfill.</p>



<p class="wp-block-paragraph">One caveat worth knowing: orders created through the API rather than through checkout can behave differently from native multi-currency checkout orders. If your store takes orders from an ERP or a marketplace integration, spot-check a few of those specifically.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Storage layout: partition by order date, never by arrival date</h2>



<p class="wp-block-paragraph">Two layers in S3. Keep them separate and keep them honest about what they are.</p>



<ol class="wp-block-list">
<li><strong>Raw.</strong> Every payload exactly as received, partitioned by ingestion date. Append-only, never edited. This is your audit trail and your rebuild source. Lifecycle it to a colder storage class after a few months, don&#8217;t delete it.</li>

<li><strong>Curated.</strong> One row per order representing current state, in Parquet, partitioned by <em>order date</em>. This is what the dashboard queries. It is derived, disposable and rewritable.</li>
</ol>



<p class="wp-block-paragraph">The partitioning choice on the curated layer is the load-bearing decision in the whole design. If you partition by arrival date, which is what Amazon Data Firehose does by default because it buckets on the moment it writes the file, then a refund that arrives three weeks late lands in today&#8217;s partition. Correcting Monday now means finding and rewriting fragments scattered across twenty other partitions. Partitioned by order date, correcting Monday means overwriting exactly one prefix.</p>



<p class="wp-block-paragraph">Firehose can do this with dynamic partitioning, which routes records by keys inside the payload rather than by write time. If you&#8217;re not using Firehose, extract the order date in your Lambda and write the prefix yourself.</p>



<h3 class="wp-block-heading">Use partition projection so Athena stops guessing</h3>



<p class="wp-block-paragraph">The default Glue Data Catalog approach means running a crawler or issuing <code>MSCK REPAIR TABLE</code> to register new partitions. Forget one and you get a query that silently returns nothing for recent days. Nobody notices until Monday.</p>



<p class="wp-block-paragraph">Partition projection removes the metastore lookup entirely. You tell Athena the shape of the partition keys and it calculates the prefixes at query time:</p>



<pre class="wp-block-code"><code>CREATE EXTERNAL TABLE shop_orders (
  order_id             bigint,
  order_number         string,
  created_at           timestamp,
  financial_status     string,
  cancelled_at         timestamp,
  is_test              boolean,
  total_shop           decimal(12,2),
  shop_currency        string,
  total_presentment    decimal(12,2),
  presentment_currency string
)
PARTITIONED BY (order_date string)
STORED AS PARQUET
LOCATION 's3://your-bucket/curated/orders/'
TBLPROPERTIES (
  'projection.enabled' = 'true',
  'projection.order_date.type' = 'date',
  'projection.order_date.format' = 'yyyy-MM-dd',
  'projection.order_date.range' = '2019-01-01,NOW',
  'projection.order_date.interval' = '1',
  'projection.order_date.interval.unit' = 'DAYS',
  'storage.location.template' =
    's3://your-bucket/curated/orders/order_date=${order_date}/'
);</code></pre>



<p class="wp-block-paragraph">Set the range start to your store&#8217;s actual first order month. Projection generates every prefix in the range, so a range starting a decade too early makes wide scans slower for no benefit.</p>



<p class="wp-block-paragraph">Parquet matters here for the same reason. Athena bills on bytes scanned, so a columnar format with good compression cuts the bill directly, and a dashboard that only ever selects six columns from a forty-column table never touches the rest.</p>



<h3 class="wp-block-heading">Net sales in one query</h3>



<p class="wp-block-paragraph">With refunds in their own table keyed by order and carrying their own date, attributing them back to the original order day is a left join and a subtraction:</p>



<pre class="wp-block-code"><code>SELECT
    o.order_date,
    SUM(o.total_shop)                                AS gross_shop,
    SUM(COALESCE(r.refunded_shop, 0))                AS refunded_shop,
    SUM(o.total_shop - COALESCE(r.refunded_shop, 0)) AS net_shop
FROM shop_orders o
LEFT JOIN (
    SELECT order_id, SUM(amount_shop) AS refunded_shop
    FROM shop_refunds
    GROUP BY order_id
) r ON r.order_id = o.order_id
WHERE o.order_date BETWEEN '2025-01-01' AND '2025-01-31'
  AND o.is_test = false
  AND o.cancelled_at IS NULL
GROUP BY o.order_date
ORDER BY o.order_date;</code></pre>



<p class="wp-block-paragraph">Note the two filters doing quiet work at the bottom. Those two lines are the difference between a number finance accepts and a number they don&#8217;t.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Choosing the dashboard layer</h2>



<p class="wp-block-paragraph">Once the data is correct, this part is genuinely a preference. All of these work.</p>



<ul class="wp-block-list">
<li><strong>Amazon QuickSight</strong>, now delivered as part of Amazon Quick Suite, is the least-friction option if you&#8217;re already in AWS. Its in-memory SPICE layer means viewers aren&#8217;t firing an Athena query per chart interaction, which controls both latency and scan cost. Per-viewer pricing tends to be the deciding factor either way, so model it for your actual audience size before committing.</li>

<li><strong>Grafana</strong> with the Athena data source is a good fit if you&#8217;re already running Grafana for infrastructure and want commercial and operational panels on one screen. Grafana Cloud removes the hosting question if you&#8217;d rather not run it.</li>

<li><strong>Power BI</strong> makes sense when the finance team already lives in Microsoft 365 and models in DAX. The cross-cloud hop is real but manageable.</li>

<li><strong>Metabase</strong> or a self-hosted alternative on a small VPS from a provider like InterServer or Hetzner is the pragmatic answer for a handful of internal viewers, where per-seat BI licensing costs more than the entire pipeline.</li>
</ul>



<p class="wp-block-paragraph">The honest trade-off: managed BI costs more per month and saves you from becoming the person who patches the reporting server. Self-hosting inverts that. Neither is wrong, but pick deliberately rather than by inertia.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting: symptom to cause</h2>



<h3 class="wp-block-heading">Totals are consistently higher than the Shopify admin</h3>



<p class="wp-block-paragraph">Almost always refunds, cancellations or test orders. Check in that order. If the gap grows with the age of the reporting window, it&#8217;s refunds. If it&#8217;s a fixed offset on specific days, look for test orders or a QA run.</p>



<h3 class="wp-block-heading">Totals are lower, and recent days are missing rows</h3>



<p class="wp-block-paragraph">Either partitions aren&#8217;t registered, which projection fixes permanently, or the webhook subscription has been dropped after repeated delivery failures. Check the subscription still exists before you go digging through Lambda logs. A nightly bulk pull would have masked this, which is another argument for having one.</p>



<h3 class="wp-block-heading">Orders appear twice</h3>



<p class="wp-block-paragraph">Shopify&#8217;s delivery model is at-least-once, not exactly-once, and you may also have more than one subscription on the same topic. Deduplicate on the delivery ID header before you touch anything else, and make the write itself idempotent so a duplicate is a no-op rather than a second row.</p>



<h3 class="wp-block-heading">Numbers are right on the daily view, wrong on the monthly</h3>



<p class="wp-block-paragraph">Timezone. Order timestamps carry an offset; your partition key is a date string. If you derive the date in UTC and the store reports in a local timezone, orders near midnight land on the wrong day. That averages out over a month, which is exactly why the discrepancy hides until month boundaries.</p>



<h3 class="wp-block-heading">Athena costs jumped without more data</h3>



<p class="wp-block-paragraph">Someone built a dashboard with a filter that doesn&#8217;t hit the partition column, so every panel refresh scans the full table. Look at bytes scanned per query and check whether the BI tool is caching results or re-querying on every interaction.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating orders as immutable events and never revisiting a past day.</li>

<li>Partitioning on arrival time because that&#8217;s the default, then discovering corrections are expensive.</li>

<li>Verifying the HMAC against a re-serialised body instead of the raw bytes.</li>

<li>Doing real work inside the webhook handler, generating duplicates under load.</li>

<li>Summing presentment amounts across currencies.</li>

<li>Registering the event bus ARN with Shopify instead of the partner event source ARN.</li>

<li>Relying on webhooks alone with no scheduled reconciliation.</li>

<li>Shipping a &#8220;revenue&#8221; number without ever defining what it includes.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Keep raw and curated layers separate. Raw is append-only; curated is rewritable.</li>

<li>Make every partition idempotently rebuildable from raw. Test that path deliberately, before you need it.</li>

<li>Buffer through SQS with a dead-letter queue. Free replay, free isolation of bad payloads.</li>

<li>Run a nightly bulk pull over a rolling window and overwrite those partitions.</li>

<li>Use partition projection. It removes an entire category of silent failure.</li>

<li>Alarm on the absence of events, not just on errors. A CloudWatch alarm on zero orders processed in an hour during business hours catches broken subscriptions the same day.</li>

<li>Store the API secret in Secrets Manager and scope the Lambda role to the exact prefixes it writes.</li>

<li>Publish a reconciliation panel comparing your total to the admin&#8217;s for the same window. Surfacing the gap builds more trust than hiding it.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need a data warehouse, or is S3 and Athena enough?</h3>



<p class="wp-block-paragraph">For a single store&#8217;s order data, S3 with Athena is almost certainly enough, and it&#8217;s cheaper because you pay per query rather than for a running cluster. Redshift starts to earn its place when you&#8217;re joining Shopify data against several other large sources, or when concurrent query load makes Athena&#8217;s queue times noticeable.</p>



<h3 class="wp-block-heading">How near-real-time can this be?</h3>



<p class="wp-block-paragraph">Events land within seconds. The practical floor is your buffering window, since writing one tiny file per order gives you a small-files problem that ruins query performance. A few minutes of buffering is the usual compromise. If you genuinely need sub-minute order counts, put a live counter in DynamoDB alongside the analytical pipeline rather than trying to make the data lake do both jobs.</p>



<h3 class="wp-block-heading">Which webhook topics should I subscribe to?</h3>



<p class="wp-block-paragraph">At minimum, order creation, order update, order cancellation and refund creation. Update and refund topics are the ones people skip, and they&#8217;re exactly the ones carrying the corrections. Subscribe to fewer topics than you think you need and add rather than subscribing to everything, since every extra topic is volume you pay to store and process.</p>



<h3 class="wp-block-heading">Can I skip AWS and use a connector tool?</h3>



<p class="wp-block-paragraph">Yes, and for many stores that&#8217;s the right answer. A managed connector into a hosted warehouse gets you a working dashboard in an afternoon. You&#8217;re paying a monthly fee to avoid owning any of this, and trading away control over the data model. Building it on AWS wins when you need Shopify data joined to systems the connector doesn&#8217;t cover, or when row-based connector pricing outgrows the infrastructure cost.</p>



<h3 class="wp-block-heading">How do I backfill historical orders?</h3>



<p class="wp-block-paragraph">Use a GraphQL bulk operation rather than paginating the API. Submit the query, poll for completion, then stream the JSONL result into your raw bucket and run the same transformation your live pipeline uses. If backfill and live processing use different code paths, they will diverge, and you&#8217;ll spend an afternoon working out which one is lying.</p>



<h3 class="wp-block-heading">What does a setup like this cost to run?</h3>



<p class="wp-block-paragraph">For a typical single store, the pipeline itself is small money: Lambda invocations, a few gigabytes in S3, and Athena billed on bytes scanned, which partitioning and Parquet keep low. The BI seats are usually the largest line item, which is why the dashboard layer decision deserves more thought than the ingestion one. Model it against current published rates rather than trusting any figure you read in a blog post, including this one.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">A Shopify sales dashboard with AWS doesn&#8217;t fail because the pipeline breaks. It fails because the pipeline keeps working perfectly on data that has since changed underneath it.</p>



<p class="wp-block-paragraph">Design for correction from the first commit. Partition by order date, keep the raw layer so you can always rebuild, run a scheduled pull to catch what the stream missed, and agree on what revenue means before anyone builds a chart. Do that and the Monday morning screenshot comparison becomes a non-event, which is the highest praise a reporting pipeline ever gets.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with your Shopify data pipeline on AWS?</h2>



<p class="wp-block-paragraph">I design and build ecommerce data pipelines and reporting stacks on AWS. Typical engagements look like:</p>



<ul class="wp-block-list">
<li>Working out why an existing Shopify dashboard disagrees with the admin, and fixing the root cause rather than patching the query</li>

<li>Building the ingestion layer end to end: EventBridge or API Gateway, Lambda, SQS with dead-letter handling, and a scheduled GraphQL bulk reconciliation job</li>

<li>Designing the S3 layout, Glue schema and Athena tables so past days can be recomputed cheaply and partitions never go missing</li>

<li>Migrating REST Admin API integrations to GraphQL before the deadline forces the issue</li>

<li>Building the dashboard itself in QuickSight, Grafana or Metabase, including the metric definitions finance will actually sign off on</li>

<li>Cutting Athena scan costs and BI licensing on a reporting stack that has grown more expensive than anyone planned</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;re in the middle of one of these, send me the actual thing: the Athena query, the S3 prefix layout, the two totals that don&#8217;t match. It&#8217;s a much faster conversation than describing it in the abstract.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/shopify-sales-dashboard-aws/">Shopify Sales Dashboard with AWS: Build One That Actually Reconciles</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
