<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Webhooks | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/webhooks/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/webhooks/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Tue, 06 Oct 2026 06:08:51 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Webhooks | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/webhooks/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Monitoring Zoho Integrations: Catching the Failures That Never Throw an Error</title>
		<link>https://john-nessime.com/blog/devops/monitoring-zoho-integrations/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 06 Oct 2026 06:08:51 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Workflow Automation]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Cardinality]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Deadman Alerting]]></category>
		<category><![CDATA[Embedded Metric Format]]></category>
		<category><![CDATA[Grafana]]></category>
		<category><![CDATA[Grafana Alloy]]></category>
		<category><![CDATA[LogQL]]></category>
		<category><![CDATA[Loki]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[Webhooks]]></category>
		<category><![CDATA[Zoho]]></category>
		<category><![CDATA[Zoho CRM]]></category>
		<category><![CDATA[Zoho Flow]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=395</guid>

					<description><![CDATA[<p>Zoho integrations rarely fail loudly. Channels expire after an hour, credits drain against a rolling window, and bulk writes return HTTP 200 while rejecting records. Here is how to instrument for absence using CloudWatch Embedded Metric Format, Loki's absent_over_time, and Grafana across both.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/monitoring-zoho-integrations/">Monitoring Zoho Integrations: Catching the Failures That Never Throw an Error</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message came in on a Tuesday: &#8220;Have we stopped getting leads from the website form?&#8221; Not an alert. Not a page. A salesperson noticing the pipeline looked thin.</p>



<p class="wp-block-paragraph">The sync worker was healthy. Zero errors in the last twenty-four hours. CPU flat, memory flat, no restarts, no failed jobs. Every dashboard was green because every dashboard was measuring the wrong thing. The integration had stopped receiving anything from Zoho four days earlier, and an integration that receives nothing produces no errors at all.</p>



<p class="wp-block-paragraph">That is the whole problem with monitoring Zoho integrations. The failures that hurt do not throw exceptions. They show up as silence, as slowly degrading throughput, or as HTTP 200 responses that quietly did not do what you asked. This post covers how to instrument for those three failure families using CloudWatch, Grafana and Loki, and why the obvious metrics miss all three.</p>



<h2 class="wp-block-heading">The three ways Zoho integrations actually break</h2>



<p class="wp-block-paragraph">Before touching a dashboard, it helps to be honest about what you are defending against. In my experience these fall into three groups, and they need genuinely different instrumentation:</p>



<ol class="wp-block-list">
<li><strong>Silent stop.</strong> Zoho stops sending you events. Nothing fails, because nothing happens.</li>

<li><strong>Silent throttle.</strong> You run out of API credits or hit a concurrency ceiling. Throughput degrades before anything hard-fails.</li>

<li><strong>Silent drift.</strong> Calls succeed, records do not land, and the two systems diverge without a single error log.</li>
</ol>



<p class="wp-block-paragraph">An error-rate panel catches none of these. That is not a criticism of error-rate panels, it is just what they are for.</p>



<h2 class="wp-block-heading">Failure family one: the integration that just stops</h2>



<p class="wp-block-paragraph">If you push data out of Zoho CRM using the Notifications API, you subscribe a <code>notify_url</code> to module events and Zoho posts to it when records change. It works well. It also has a detail that catches people out repeatedly.</p>



<p class="wp-block-paragraph">Notification channels expire. Zoho&#8217;s documentation is explicit: the expiry can be set to a maximum of one week from the time the channel is enabled, and if you do not specify one, or you ask for longer than a week, the default expiry is <strong>one hour</strong>. So the failure mode is not &#8220;someone forgot to renew after a year.&#8221; It is &#8220;someone deployed a subscription script without an expiry field and the integration worked beautifully for sixty minutes.&#8221;</p>



<p class="wp-block-paragraph">What makes this genuinely hard rather than just annoying is that there is no endpoint to list your currently active channels. You cannot ask Zoho what it thinks it is subscribed to. You can only ask whether events are still arriving. That single fact should shape your entire monitoring design: you are not monitoring for errors, you are monitoring for <em>absence</em>.</p>



<h3 class="wp-block-heading">Emit a heartbeat, then alarm on its absence</h3>



<p class="wp-block-paragraph">The receiving end of the webhook should emit a metric on every delivery, including deliveries you decide to discard. Embedded Metric Format is the cheapest way to do this from Lambda or a container, because you write structured JSON to stdout and CloudWatch extracts the metrics asynchronously. No <code>PutMetricData</code> call, no synchronous latency in your request path.</p>



<pre class="wp-block-code"><code>{
  "_aws": {
    "Timestamp": 1710000000000,
    "CloudWatchMetrics": [
      {
        "Namespace": "ZohoIntegration",
        "Dimensions": [["Module", "Channel"]],
        "Metrics": [
          { "Name": "NotificationsReceived", "Unit": "Count" },
          { "Name": "ChannelSecondsRemaining", "Unit": "Seconds" }
        ]
      }
    ]
  },
  "Module": "Leads",
  "Channel": "leads-prod",
  "NotificationsReceived": 1,
  "ChannelSecondsRemaining": 486000,
  "record_id": "5725767000002166520",
  "operation": "insert"
}</code></pre>



<p class="wp-block-paragraph">Note what is a dimension and what is not. <code>Module</code> and <code>Channel</code> are bounded sets, so they are safe dimensions. <code>record_id</code> sits in the log body only. EMF creates a separate custom metric for every unique dimension combination, so promoting a record ID to a dimension is how you turn a monitoring project into a billing incident. AWS says this plainly in the EMF documentation and it is worth reading twice before you ship.</p>



<p class="wp-block-paragraph">Now the alarm. A normal CloudWatch alarm goes to <code>INSUFFICIENT_DATA</code> when the metric stops reporting, and <code>INSUFFICIENT_DATA</code> does not notify. That is precisely backwards for this failure. You want missing data treated as a breach:</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-alarm 
  --alarm-name zoho-leads-notifications-silent 
  --namespace ZohoIntegration 
  --metric-name NotificationsReceived 
  --dimensions Name=Module,Value=Leads Name=Channel,Value=leads-prod 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 2 
  --threshold 1 
  --comparison-operator LessThanThreshold 
  --treat-missing-data breaching 
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:ops-alerts</code></pre>



<p class="wp-block-paragraph">The <code>--treat-missing-data breaching</code> flag is the entire point of that command. Without it, an integration that goes completely dead produces an alarm that sits quietly in an unknown state forever.</p>



<p class="wp-block-paragraph">On the Loki side the equivalent is <code>absent_over_time</code>, which returns 1 when a stream has produced no entries in the window. Run it from the Loki ruler so it fires even when nobody has a browser open:</p>



<pre class="wp-block-code"><code>groups:
  - name: zoho-integration
    interval: 5m
    rules:
      - alert: ZohoNotificationsSilent
        expr: absent_over_time({job="zoho-webhook", module="leads"}[2h])
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: "No Zoho notifications for Leads in 2h"
          description: "Channel may have expired. Re-subscribe and check channel_expiry."</code></pre>



<p class="wp-block-paragraph">One caveat worth knowing: <code>absent_over_time</code> only works for label combinations you name explicitly. It cannot tell you that a stream you never thought about has gone quiet. That is a real limitation, and it means your channel inventory has to live in your own configuration, not be discovered from Loki.</p>



<p class="wp-block-paragraph">Because you cannot list channels through the API, treat renewal as a scheduled job with its own deadman alert, and emit <code>ChannelSecondsRemaining</code> at each renewal. Alarm when it drops below roughly a day. That gives you warning before the silence starts rather than after somebody notices.</p>



<h2 class="wp-block-heading">Failure family two: throttling that is not rate limiting</h2>



<p class="wp-block-paragraph">Most engineers instrument Zoho as if it were a requests-per-minute API. It is not, and that mismatch produces dashboards that look fine right up until calls start failing.</p>



<p class="wp-block-paragraph">Zoho CRM meters API usage in <strong>credits</strong>, not calls. Different operations deduct different amounts, and bulk operations are metered per batch of records rather than per request. The budget resets on a rolling twenty-four hour window measured from each call, not at midnight. If your dashboard has a daily reset at 00:00, it is drawing a line that does not exist.</p>



<h3 class="wp-block-heading">The gauge that appears halfway through the tank</h3>



<p class="wp-block-paragraph">Here is the detail that makes credit monitoring awkward. Zoho returns remaining credits in an <code>X-API-CREDITS-REMAINING</code> response header, but only once usage has reached fifty percent or more of the daily allowance. Below that threshold the header simply is not there.</p>



<p class="wp-block-paragraph">Read that again in monitoring terms. An absent header means either &#8220;you are comfortably under half your budget&#8221; or &#8220;your instrumentation is broken and you have no idea.&#8221; Those two states look identical, and one of them is fine while the other is a blind spot. So record both facts separately:</p>



<ul class="wp-block-list">
<li>A counter for responses where the header was <em>present</em>, which effectively tells you when you crossed the halfway mark.</li>

<li>A gauge of the value itself, which only has meaning while the header exists.</li>

<li>Your own client-side count of calls made, which works regardless and is the only thing you control.</li>
</ul>



<p class="wp-block-paragraph">The first crossing of that threshold is a useful early warning in its own right. Alert on the transition, not just on a low remaining value.</p>



<h3 class="wp-block-heading">Concurrency is a separate ceiling</h3>



<p class="wp-block-paragraph">Alongside credits, Zoho applies a concurrency limit: the number of calls active simultaneously per org. There is also a sub-concurrency limit covering the expensive operations such as COQL queries, composite requests, lead conversion, and inserts or updates above a small record count. Exceeding either returns <code>TOO_MANY_REQUESTS</code>.</p>



<p class="wp-block-paragraph">Two consequences matter for monitoring. First, since the limit is on simultaneous calls rather than calls per unit time, a requests-per-minute graph will never predict a concurrency failure. You need a gauge of in-flight requests, incremented before the call and decremented in a <code>finally</code> block. Second, the sub-concurrency ceiling is the same across every CRM edition, so upgrading the plan raises the general limit but does not buy headroom on the expensive calls. Batch those and queue them rather than fanning out.</p>



<p class="wp-block-paragraph">One more source of consumption that surprises people: integration tasks inside Deluge functions consume credits from the same org budget as your external API calls. If your credit graph climbs while your service is idle, someone&#8217;s workflow rule is spending your budget. Zoho&#8217;s API Dashboard breaks consumption down by application and function, which is where to look before blaming your own code.</p>



<h2 class="wp-block-heading">Failure family three: successful calls that did nothing</h2>



<p class="wp-block-paragraph">Bulk and multi-record operations return a per-record status inside the response body. The HTTP status can be perfectly healthy while individual records were rejected for validation rules, mandatory fields, duplicate checks or picklist values that do not exist in the target layout.</p>



<p class="wp-block-paragraph">If your client only checks the HTTP status, you will log a success, increment a success counter, and move on. Weeks later somebody asks why the CRM is missing a slice of records that your logs insist were written.</p>



<p class="wp-block-paragraph">The fix is to stop counting requests and start counting records, on both sides:</p>



<ul class="wp-block-list">
<li><strong>Records submitted</strong> versus <strong>records confirmed</strong> in the per-record response status.</li>

<li>A breakdown of rejection reasons as a low-cardinality label, so a spike in one specific validation error is visible.</li>

<li>A periodic reconciliation job comparing source counts against Zoho counts for a bounded window, which is the only thing that catches drift you failed to anticipate.</li>
</ul>



<p class="wp-block-paragraph">In LogQL, the divergence between the two counters is the panel that actually earns its place on the wall:</p>



<pre class="wp-block-code"><code>sum(count_over_time({job="zoho-sync"} | json | status = "submitted" [1h]))
  -
sum(count_over_time({job="zoho-sync"} | json | status = "confirmed" [1h]))</code></pre>



<p class="wp-block-paragraph">A healthy integration holds that near zero. A number that grows steadily is drift, and it is visible weeks before anyone in sales notices.</p>



<h2 class="wp-block-heading">Deciding which signal goes where</h2>



<p class="wp-block-paragraph">Running CloudWatch, Grafana and Loki together only helps if each one has a job. The split I reach for first:</p>



<ul class="wp-block-list">
<li><strong>CloudWatch</strong> holds the numbers you alarm on, especially where the workload already runs in AWS and EMF gives you metrics for free out of log output.</li>

<li><strong>Loki</strong> holds the full request and response context: which record, which module, which validation error, which channel. This is what you actually read at three in the morning.</li>

<li><strong>Grafana</strong> is the single pane over both, with CloudWatch and Loki added as separate data sources so one dashboard can show the credit gauge next to the rejection reasons.</li>
</ul>



<p class="wp-block-paragraph">The trap that spans both backends is cardinality, and it bites in the same way for different reasons. In CloudWatch, each unique dimension combination becomes a billable custom metric. In Loki, each unique label combination becomes a separate stream, and Loki was designed for a small number of long-lived streams. Record IDs, email addresses and timestamps belong in neither.</p>



<p class="wp-block-paragraph">Loki&#8217;s answer is structured metadata: high-cardinality fields attached to log lines without creating streams or entering the index. It needs enabling per tenant:</p>



<pre class="wp-block-code"><code>limits_config:
  allow_structured_metadata: true
  volume_enabled: true</code></pre>



<p class="wp-block-paragraph">Put <code>module</code>, <code>environment</code> and <code>job</code> in labels. Put the Zoho record ID and the correlation ID in structured metadata. Query with a label selector first and filter on the rest.</p>



<p class="wp-block-paragraph">On collection, one scheduling note that does age: Promtail reached end of life on 2 March 2026, with its collection capabilities folded into Grafana Alloy. If you are still shipping through Promtail, that migration is now maintenance debt rather than an optional upgrade, and Grafana published a conversion tool for existing configurations. New pipelines should start on Alloy.</p>



<p class="wp-block-paragraph">Where you run all this is a real decision. Self-hosting Loki on a VPS from a provider like Contabo or InterServer is cheap and entirely workable for a single integration&#8217;s log volume, and it keeps the data under your control. Grafana Cloud or Amazon Managed Grafana removes the operational surface if you would rather not babysit an ingester. Both are defensible. The honest trade-off is that self-hosted Loki is easy to stand up and less easy to keep healthy once retention and compaction start to matter.</p>



<h2 class="wp-block-heading">Troubleshooting the monitoring itself</h2>



<ul class="wp-block-list">
<li><strong>EMF metrics never appear.</strong> The JSON must be a single line and structurally valid. A pretty-printed or truncated log line is ingested as an ordinary log event with no error, so check the raw event in the log stream rather than the metrics console.</li>

<li><strong>The alarm never fires despite obvious silence.</strong> Look at the alarm state. Sitting in <code>INSUFFICIENT_DATA</code> means the missing-data treatment was left at its default.</li>

<li><strong><code>absent_over_time</code> returns nothing.</strong> That is the success case. It emits a value only when the stream is empty, which makes it feel broken while everything is working.</li>

<li><strong>Loki queries slow to a crawl.</strong> Check stream count before blaming resources. A label added &#8220;temporarily&#8221; during debugging is the usual cause.</li>

<li><strong>Webhook deliveries stop after a deployment.</strong> Check whether the subscription script reran and created a channel without an expiry. An hour later it will look identical to a network problem.</li>

<li><strong>Credit consumption does not match your call count.</strong> Expected. Operations deduct different credit amounts, and Deluge integration tasks draw from the same pool.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Alerting on error rate alone, which is blind to every failure described above.</li>

<li>Creating notification channels without an explicit <code>channel_expiry</code> and inheriting the one-hour default.</li>

<li>Assuming an absent <code>X-API-CREDITS-REMAINING</code> header means healthy usage rather than sub-threshold usage.</li>

<li>Building a daily credit dashboard when the limit is a rolling window.</li>

<li>Putting record IDs into EMF dimensions or Loki labels.</li>

<li>Treating an HTTP 200 on a bulk write as confirmation that every record landed.</li>

<li>Running alert rules only in the Grafana UI, so nothing evaluates if that instance is down.</li>
</ul>



<h2 class="wp-block-heading">Best practices for monitoring Zoho integrations</h2>



<ol class="wp-block-list">
<li>Start with a deadman alert per channel. If you build one thing, build this.</li>

<li>Set <code>channel_expiry</code> explicitly, renew on a schedule, and monitor the renewal job as carefully as the integration.</li>

<li>Count records, not requests, on both sides of every write.</li>

<li>Keep a client-side call counter so you are never dependent on a header that may not be sent.</li>

<li>Gauge in-flight requests to see concurrency pressure before it becomes <code>TOO_MANY_REQUESTS</code>.</li>

<li>Attach a correlation ID to every record as structured metadata so one identifier joins logs and metrics.</li>

<li>Evaluate critical rules server-side, in the Loki ruler or as CloudWatch alarms.</li>

<li>Test by disabling a channel in a sandbox org and confirming the alert fires. An untested deadman alert is decoration.</li>
</ol>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Why did my Zoho webhook stop working on its own?</h3>



<p class="wp-block-paragraph">Almost always channel expiry. Zoho CRM notification channels last a maximum of one week, and if the expiry is omitted or set beyond that maximum, the channel defaults to one hour. Nothing is logged on your side because the deliveries simply stop arriving.</p>



<h3 class="wp-block-heading">Can I list my active Zoho notification channels?</h3>



<p class="wp-block-paragraph">There is no endpoint for enumerating active channels, which is a long-standing complaint in Zoho&#8217;s developer community. Keep the inventory in your own configuration and treat the receipt of events as the only reliable liveness signal.</p>



<h3 class="wp-block-heading">Do I need both CloudWatch and Loki?</h3>



<p class="wp-block-paragraph">No. Either can carry the whole job. The pairing makes sense when your integration already runs on AWS, because EMF gives you alarmable metrics from log output at no extra instrumentation cost, while Loki holds richer request context more cheaply. If you are starting fresh outside AWS, one backend is simpler and simpler is worth a lot.</p>



<h3 class="wp-block-heading">How do I monitor Zoho API credits properly?</h3>



<p class="wp-block-paragraph">Record the <code>X-API-CREDITS-REMAINING</code> header when it appears, alert on the moment it first appears in a window, and maintain your own call counter as the fallback. Model the budget as a rolling twenty-four hour window rather than a daily reset.</p>



<h3 class="wp-block-heading">What causes TOO_MANY_REQUESTS if I am under my credit limit?</h3>



<p class="wp-block-paragraph">Concurrency, not consumption. Credits and concurrency are independent ceilings. Expensive operations such as COQL queries and composite requests share a tighter sub-concurrency pool, so a burst of parallel queries can trip it while your credit balance is untouched.</p>



<h3 class="wp-block-heading">Should I still use Promtail for shipping integration logs?</h3>



<p class="wp-block-paragraph">No. Promtail reached end of life on 2 March 2026 and its functionality moved into Grafana Alloy. Existing deployments will keep running, but they are unsupported, and Grafana provides a tool to convert Promtail configuration to Alloy.</p>



<h3 class="wp-block-heading">How long should the deadman alert window be?</h3>



<p class="wp-block-paragraph">Base it on the quietest legitimate period for that module. A leads webhook that goes quiet overnight and at weekends needs a longer window than an order sync, or you will train the team to ignore it. Measure the real gap distribution for a fortnight before picking a number.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Monitoring Zoho integrations is mostly the discipline of detecting absence. Expired channels, exhausted credits and silently rejected records all present as nothing happening, and nothing happening is invisible to the instrumentation most teams reach for first.</p>



<p class="wp-block-paragraph">So invert the default. Alarm when the count is too low rather than when the errors are too high. Set <code>--treat-missing-data breaching</code>. Use <code>absent_over_time</code>. Count records on both sides and watch the gap. If you build only one thing from this post, build the deadman alert on notification delivery, because that is the failure that will otherwise be reported to you by a salesperson wondering where the leads went.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help instrumenting a Zoho integration?</h2>



<p class="wp-block-paragraph">Most of the work here is judgement about what to measure rather than volume of code. If you are staring at a sync that keeps going quiet, this is the sort of thing I take on:</p>



<ul class="wp-block-list">
<li>Building deadman alerting for Zoho notification channels, including renewal jobs that monitor themselves.</li>

<li>Instrumenting credit and concurrency headroom so throttling is visible before it becomes <code>TOO_MANY_REQUESTS</code>.</li>

<li>Designing EMF dimensions and Loki label schemas that will not turn into a cardinality bill.</li>

<li>Adding reconciliation jobs that compare record counts across systems and surface drift early.</li>

<li>Standing up Loki and Grafana Alloy, or migrating an existing Promtail pipeline that is now unsupported.</li>

<li>Reviewing an existing Grafana dashboard and cutting the panels that look reassuring but detect nothing.</li>
</ul>



<p class="wp-block-paragraph">Send me a sample webhook payload, a redacted response header set, or a screenshot of the dashboard you currently trust, and I will tell you what it would miss.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/monitoring-zoho-integrations/">Monitoring Zoho Integrations: Catching the Failures That Never Throw an Error</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Your CRM Is Not a Database: Where GoHighLevel Should Stop</title>
		<link>https://john-nessime.com/blog/data-engineering/gohighlevel-as-a-database/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 26 Sep 2026 06:00:23 +0000</pubDate>
				<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Enterprise Integration]]></category>
		<category><![CDATA[SaaS Engineering]]></category>
		<category><![CDATA[Architecture Debt]]></category>
		<category><![CDATA[CRM Audit]]></category>
		<category><![CDATA[Custom Objects]]></category>
		<category><![CDATA[Data Modeling]]></category>
		<category><![CDATA[GoHighLevel]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Marketing Automation]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Reverse ETL]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Snapshots]]></category>
		<category><![CDATA[Sub-Account Management]]></category>
		<category><![CDATA[System of Record]]></category>
		<category><![CDATA[Vendor Lock-In]]></category>
		<category><![CDATA[Webhooks]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=649</guid>

					<description><![CDATA[<p>A CRM never pushes back when you add one more field, which is exactly why the schema drifts until someone asks a question that needs a join. Four boundaries for deciding what belongs in GoHighLevel, what Custom Objects actually change, and the CRM-in-front, system-of-record-behind pattern I default to.</p>
<p>The post <a href="https://john-nessime.com/blog/data-engineering/gohighlevel-as-a-database/">Your CRM Is Not a Database: Where GoHighLevel Should Stop</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">&#8220;Can you pull every job we did last quarter, by technician, with the margin on each one?&#8221;</p>



<p class="wp-block-paragraph">The answer was no. Not &#8220;no, that&#8217;s expensive.&#8221; Just no. The jobs weren&#8217;t records. Each one was a couple of tags on a contact, three custom fields added at different times by different people, an opportunity someone had dragged into Won, and a note cut off mid-sentence. There was no job. There was a person that a job had happened to.</p>



<p class="wp-block-paragraph">Nobody made a bad decision to get there. &#8220;Can we just add a field for that?&#8221; is a five-second request and the answer is always yes, because adding a field is easy. That&#8217;s the trap. The CRM never pushes back, so the schema drifts for eighteen months and nobody notices until someone asks a question that needs a join.</p>



<p class="wp-block-paragraph">This post is about where that line sits: not whether to use the platform, but where <strong>GoHighLevel as a database</strong> stops being a shortcut and starts being architecture debt. I&#8217;ll cover the failure that stays invisible, the four boundaries that decide it, what Custom Objects actually change, and the arrangement I default to when the CRM isn&#8217;t the right home.</p>



<h2 class="wp-block-heading">The failure mode nobody sees coming</h2>



<p class="wp-block-paragraph">Databases push back. Declare a column as a date and it refuses to hold &#8220;next Tuesday.&#8221; Declare a foreign key and it refuses to orphan the row. Half the value of a schema isn&#8217;t storage, it&#8217;s the set of things it won&#8217;t let you do.</p>



<p class="wp-block-paragraph">A custom field holds whatever you type. That&#8217;s a feature when a rep jots down a competitor&#8217;s name. It&#8217;s a slow disaster when the field is called <code>contract_value</code> and it contains &#8220;1,250.00&#8221;, &#8220;$1250&#8221;, &#8220;1250 (pending)&#8221; and &#8220;ask Dave&#8221; across four hundred records.</p>



<p class="wp-block-paragraph">Here&#8217;s why it stays invisible: nothing breaks. Workflows keep firing. Dashboards keep rendering, because they count records rather than summing values. The failure surfaces the day someone needs an aggregate, and by then you have four hundred rows of ambiguity and no way to tell which reading was right. You can&#8217;t retroactively decide whether &#8220;1250 (pending)&#8221; was invoiced.</p>



<p class="wp-block-paragraph">History is the other half. A CRM field is a current value. Overwrite it and the old one is gone. Anything touching money, entitlements or consent eventually needs to prove what the state was and when, and the platform discarded that silently, months earlier.</p>



<h2 class="wp-block-heading">What the platform is genuinely good at</h2>



<p class="wp-block-paragraph">An argument against a tool isn&#8217;t credible unless it starts with the tool&#8217;s real case. GoHighLevel is very good at a specific set of things, most of them expensive to build yourself.</p>



<ul class="wp-block-list">
<li><strong>Conversation state across channels.</strong> One thread per person spanning SMS, email and calls, with delivery status attached. Rebuilding that is months of work and a compliance surface you don&#8217;t want.</li>

<li><strong>Sequenced outreach with human interrupts.</strong> Wait steps, conditions, and a reply pulling someone out of a sequence. Easy to describe, tedious to get right.</li>

<li><strong>Tenant isolation by default.</strong> Each sub-account is its own location, and nothing crosses unless you deliberately copy it.</li>

<li><strong>Non-engineers can change it.</strong> A workflow a client can edit on a Tuesday afternoon beats an elegant pipeline only you can touch.</li>
</ul>



<p class="wp-block-paragraph">Every one of those is about people and communication. That&#8217;s the pattern: when the thing you&#8217;re storing is a person, or something that happened to a person, the CRM is the right home. When it has a life independent of any person, you&#8217;re outside the design.</p>



<h2 class="wp-block-heading">Four boundaries that decide it for you</h2>



<p class="wp-block-paragraph">&#8220;Is this a database?&#8221; isn&#8217;t a useful question. These four are decidable, and any one of them crossing is usually enough.</p>



<h3 class="wp-block-heading">1. The question you need to ask requires a join</h3>



<p class="wp-block-paragraph">Not &#8220;show me contacts tagged X.&#8221; I mean revenue by technician by quarter, or every account with more than three open items where the last payment was over sixty days ago: grouping, filtering across related records, and arithmetic on typed values, all at once. Smart lists answer a narrow, predefined set of questions well, but they aren&#8217;t a query engine, and the gap between the two is where most people discover they&#8217;ve built a database by accident.</p>



<h3 class="wp-block-heading">2. More than one system writes the same field</h3>



<p class="wp-block-paragraph">This is the one that quietly costs the most. A stock level or invoice status lives in a CRM field, and both an automation and an external sync can write it. There&#8217;s no transaction and no compare-and-set. Last write wins, and &#8220;last&#8221; is whichever request happened to land second. You see it as an occasional wrong value you can&#8217;t reproduce, which is the worst kind of bug to be handed.</p>



<p class="wp-block-paragraph">The fix isn&#8217;t better retry logic. The field has the wrong owner. Exactly one system should write any given piece of state, and if that system isn&#8217;t the CRM, the CRM holds a copy for display rather than the value itself.</p>



<h3 class="wp-block-heading">3. Someone will need the history</h3>



<p class="wp-block-paragraph">Anything where you may later have to prove a prior state needs append-only storage. The workaround people reach for is a note per change. It doesn&#8217;t hold: notes aren&#8217;t structured, aren&#8217;t queryable, and the export path treats them as an afterthought, which leads straight to the next boundary.</p>



<h3 class="wp-block-heading">4. The exit shape doesn&#8217;t match what you put in</h3>



<p class="wp-block-paragraph">People check this last and should check it first. Contact export lands as CSV, and only the most recent note travels with it, truncated to 255 characters. A year of notes exports as one, clipped. Custom object data isn&#8217;t part of a direct export at all: the documented route is a snapshot from one sub-account to another, which is a copy inside the platform, not a file you keep. Export permissions are role-gated, so a client who believes they own their data may not be able to pull it themselves.</p>



<p class="wp-block-paragraph">None of that makes it a bad choice. It makes it a bad <em>sole</em> home for anything you&#8217;d be in trouble without.</p>



<h2 class="wp-block-heading">What Custom Objects change, and what they don&#8217;t</h2>



<p class="wp-block-paragraph">Custom Objects are a real improvement and worth using. Properties, policies, vehicles, courses: things that aren&#8217;t people, modelled as first-class records with their own fields and associations back to contacts, usable in workflows. For years the only option was squashing everything into a contact, and that was worse. They don&#8217;t turn the CRM into a database, though, and the reasons are specific rather than philosophical.</p>



<ul class="wp-block-list">
<li><strong>Ten objects per sub-account, documented as a hard cap.</strong> That&#8217;s your entire entity budget. Fifteen nouns worth modelling means five go back into custom fields.</li>

<li><strong>Uniqueness is limited and one-way.</strong> Up to ten unique fields per object, and only single line text, multi line text, number and phone types can be unique. It&#8217;s enforced across the sub-account and every entry point including the API, which is genuinely useful. But downgrade a unique field to non-unique and you cannot make it unique again.</li>

<li><strong>Write access is admin-only.</strong> Creating, updating and deleting object records is restricted to location admins; regular users are read-only. Check this before assuming the front desk will maintain records.</li>

<li><strong>Records count against storage quota.</strong> Object records and attached files consume the same account storage as everything else, so high-volume machine-generated data is the wrong shape.</li>

<li><strong>The exit is a snapshot, not an export.</strong> Objects, fields and associations move between sub-accounts via snapshots, subject to the same cap in the target account.</li>
</ul>



<p class="wp-block-paragraph">Read those as design constraints rather than complaints. They tell you the intended scale: dozens of entity types, no. Millions of rows, no. A handful of business nouns with clean identity and moderate volume, yes, and it works well there.</p>



<h2 class="wp-block-heading">The API budget is a design constraint too</h2>



<p class="wp-block-paragraph">If you&#8217;re treating the platform as a datastore behind an integration, the rate limits shape what&#8217;s possible. On the v2 API a marketplace app gets a burst allowance of 100 requests per 10 seconds per resource, and 200,000 requests per day per resource, where a resource means a location or company. Version 1 is end of support, so anything new belongs on v2.</p>



<p class="wp-block-paragraph">Every v2 request carries a version header alongside the bearer token. It&#8217;s easy to forget and produces a confusing failure when missing:</p>



<pre class="wp-block-code"><code>curl -H "Authorization: Bearer $TOKEN" 
     -H "Version: 2021-04-15" 
     -H "Accept: application/json" 
     "$BASE_URL/&lt;resource&gt;?locationId=$LOCATION_ID"</code></pre>



<p class="wp-block-paragraph">More useful than the numbers is that responses carry their own accounting, so you never have to guess where you are in the window: <code>X-RateLimit-Remaining</code> and <code>X-RateLimit-Max</code> cover the burst interval, <code>X-RateLimit-Daily-Remaining</code> and <code>X-RateLimit-Limit-Daily</code> cover the day.</p>



<p class="wp-block-paragraph">Pace against the remaining count rather than reacting to failures, and watch the daily counter separately: burst limits recover in seconds, a daily allowance doesn&#8217;t, and a backfill that exhausts it takes the rest of the day&#8217;s normal traffic with it. The consequence is that polling is expensive and webhooks are cheap. Push where you can; poll only to reconcile.</p>



<h2 class="wp-block-heading">The shape I reach for: CRM in front, system of record behind</h2>



<p class="wp-block-paragraph">When something crosses one of those boundaries, this is the arrangement I default to. It&#8217;s deliberately boring.</p>



<ol class="wp-block-list">
<li><strong>Pick one owner per piece of state and write it down.</strong> Contact details and conversation history belong to the CRM. Money, inventory and anything with an audit requirement belong to the external store. No field gets two owners.</li>

<li><strong>Give every record a stable external ID on both sides.</strong> Do this on day one. Retrofitting identity onto records matched by email is genuinely painful.</li>

<li><strong>Push changes out with webhooks, not polling.</strong> Webhook to a queue, queue to a worker, and make the worker idempotent, because you will receive duplicates and you&#8217;ll need to replay after an outage.</li>

<li><strong>Reconcile on a schedule anyway.</strong> Webhooks get dropped. A nightly sweep comparing counts and updated timestamps, re-pulling only the drift, catches what the push path missed without burning the daily allowance.</li>

<li><strong>Push a flattened summary back for display.</strong> The CRM doesn&#8217;t need the ledger, just the two or three values a human wants on the contact record. Treat those fields as read-only and never let a workflow write them.</li>
</ol>



<p class="wp-block-paragraph">The external side doesn&#8217;t need to be impressive. For a small engagement it&#8217;s usually managed Postgres, or plain Postgres on a modest VPS from somewhere like Contabo or InterServer, sat behind Cloudflare so nothing is directly exposed. The glue is often n8n or Make rather than anything custom. What matters is that constraints exist:</p>



<pre class="wp-block-code"><code>CREATE TABLE job (
  id             bigserial PRIMARY KEY,
  crm_contact_id text        NOT NULL,
  technician_id  bigint      NOT NULL REFERENCES technician(id),
  completed_on   date        NOT NULL,
  value_cents    bigint      NOT NULL CHECK (value_cents &gt;= 0),
  created_at     timestamptz NOT NULL DEFAULT now()
);</code></pre>



<p class="wp-block-paragraph">Nothing clever there. But <code>value_cents</code> can&#8217;t hold &#8220;ask Dave&#8221;, <code>technician_id</code> can&#8217;t point at a technician who doesn&#8217;t exist, and the quarterly margin question becomes one query instead of a conversation about why it can&#8217;t be answered.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<ul class="wp-block-list">
<li><strong>&#8220;We&#8217;re too small for a second system.&#8221;</strong> Sometimes right, and worth taking seriously. But the cost of adding a store later isn&#8217;t the store, it&#8217;s the reconstruction, and some of that history is unrecoverable.</li>

<li><strong>&#8220;We&#8217;ll clean the data up later.&#8221;</strong> Cleanup needs to know what the value should have been. Free-text fields with no history don&#8217;t retain that. This is the one I&#8217;d push back on hardest.</li>

<li><strong>&#8220;Custom Objects solved this.&#8221;</strong> They solved modelling non-person entities, which was the biggest gap. They didn&#8217;t add joins, transactions, versioning or an export path.</li>

<li><strong>&#8220;The client wants everything in one place.&#8221;</strong> They want one place to <em>look</em>, which is a UI requirement, not a storage requirement. A summary pushed back to the contact record satisfies it.</li>

<li><strong>&#8220;We can export whenever we want.&#8221;</strong> Test it before you rely on it. Export the notes. Export a custom object. Do it as the client&#8217;s own user, not as an admin.</li>
</ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<ol class="wp-block-list">
<li><strong>Name the thing.</strong> If it&#8217;s a noun that isn&#8217;t a person, it wants to be a record, not a field on a contact.</li>

<li><strong>Write out the hardest question anyone will ask of it.</strong> The actual sentence. If it needs grouping, arithmetic, or two entity types at once, the CRM isn&#8217;t the home.</li>

<li><strong>Count the writers.</strong> More than one system writing the same field means the CRM holds a copy, not the value.</li>

<li><strong>Decide whether history matters.</strong> Assume it does for anything involving money or consent.</li>

<li><strong>Do the export before you commit.</strong> Not the docs, the actual file. Whatever doesn&#8217;t come out cleanly is data you don&#8217;t really own.</li>

<li><strong>Check the entity budget.</strong> Ten objects per sub-account. If you&#8217;re modelling toward that ceiling, decide now which nouns get a slot.</li>
</ol>



<p class="wp-block-paragraph">Steps two and five take about twenty minutes together and prevent most of what I&#8217;ve described.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I use GoHighLevel as a database for my business data?</h3>



<p class="wp-block-paragraph">For people, conversations, bookings and pipeline state, yes, and it&#8217;s a good one. For anything needing joins, transactions, versioned history or guaranteed export, no. Using <strong>GoHighLevel as a database</strong> for that second category works right up until someone asks a question the platform can&#8217;t express, and by then the data is already ambiguous.</p>



<h3 class="wp-block-heading">How many Custom Objects can a sub-account have?</h3>



<p class="wp-block-paragraph">Ten per location, documented as a hard cap. Each object supports up to ten unique fields, and only single line text, multi line text, number and phone types can be marked unique. Treat those slots as a fixed budget when you plan the model.</p>



<h3 class="wp-block-heading">Can I export Custom Object data out of the platform?</h3>



<p class="wp-block-paragraph">Not through a direct export the way contacts go to CSV. The documented path is a snapshot between sub-accounts, which carries fields and associations but stays inside the platform. The API does support CRUD on objects and records, so a scripted pull is the practical route to a copy you keep.</p>



<h3 class="wp-block-heading">Why do my exported notes look truncated?</h3>



<p class="wp-block-paragraph">Contact export includes only the most recent note, cut to 255 characters. That&#8217;s expected behaviour, not a bug. If notes hold anything you&#8217;d need later, pull them through the API and store them somewhere that keeps all of them.</p>



<h3 class="wp-block-heading">What usually trips the API rate limits?</h3>



<p class="wp-block-paragraph">Backfills, and polling loops that scale with contact count. The limits are 100 requests per 10 seconds and 200,000 per day, both per resource. Read the rate limit headers on each response and pace against them rather than retrying into a wall.</p>



<h3 class="wp-block-heading">Does a sub-account count as a real tenancy boundary?</h3>



<p class="wp-block-paragraph">For operational separation, yes: each is its own location and nothing crosses unless you copy it. The consequence is that cross-client reporting has to happen outside the platform, because no query spans locations. If portfolio-level reporting is a requirement, plan for an external store from the start.</p>



<h3 class="wp-block-heading">Should I migrate off the CRM entirely?</h3>



<p class="wp-block-paragraph">Usually not. A full migration means rebuilding the conversation and outreach layer, which is the part it does best. Moving one category of data out, with a clear owner and a summary pushed back for display, is smaller and reversible.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The CRM will never tell you that you&#8217;ve gone too far. There&#8217;s no constraint to violate, no error to raise, no migration to fail. That&#8217;s exactly what makes the boundary worth drawing on purpose, before the field gets added rather than after four hundred records have been through it.</p>



<p class="wp-block-paragraph">The test I&#8217;d keep: write down the hardest question anyone will ever ask of this data, then check whether the platform can express it. If it can&#8217;t, that&#8217;s not a reporting gap a better dashboard will close. It&#8217;s the boundary telling you where <strong>GoHighLevel as a database</strong> should stop and a real system of record should start. The identity mapping, the webhooks, the reconciliation sweep: all of that is plumbing once the decision is made.</p>



<h2 class="wp-block-heading">Working on this with me</h2>



<p class="wp-block-paragraph">Most of what I do here is unglamorous and ends with the reporting question getting answered:</p>



<ul class="wp-block-list">
<li>Auditing an existing setup and mapping which custom fields have quietly become load-bearing, with an owner assigned to each.</li>

<li>Drawing the boundary: what stays in the CRM, what moves to Custom Objects, what needs an external system of record.</li>

<li>Building the sync layer: webhooks into a queue, idempotent workers, stable ID mapping both ways, and a reconciliation job that catches dropped events without burning the API allowance.</li>

<li>Standing up the store: schema design with constraints that hold, on managed Postgres or a small VPS, with backups you&#8217;ve tested restoring rather than assumed.</li>

<li>Rescuing free-text fields into typed columns, including the judgement calls about what&#8217;s recoverable.</li>

<li>Handover packaging: a real export, documented, so a client can leave with their data whether or not they keep working with you.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;re weighing this up, send me something concrete: a list of your custom field names, a sample export, or the query you can&#8217;t currently answer. That&#8217;s usually enough to tell you where the line sits.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/data-engineering/gohighlevel-as-a-database/">Your CRM Is Not a Database: Where GoHighLevel Should Stop</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Six Signs GoHighLevel Needs a Developer, Not Another Workflow</title>
		<link>https://john-nessime.com/blog/workflow-automation/gohighlevel-custom-development-vs-workflows/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 17 Sep 2026 06:00:00 +0000</pubDate>
				<category><![CDATA[Enterprise Integration]]></category>
		<category><![CDATA[SaaS Engineering]]></category>
		<category><![CDATA[Workflow Automation]]></category>
		<category><![CDATA[Architecture Debt]]></category>
		<category><![CDATA[CRM Audit]]></category>
		<category><![CDATA[Error Handling]]></category>
		<category><![CDATA[Execution Logs]]></category>
		<category><![CDATA[Exponential Backoff]]></category>
		<category><![CDATA[GoHighLevel]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[iPaaS]]></category>
		<category><![CDATA[Marketing Automation]]></category>
		<category><![CDATA[n8n]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Over-Engineering]]></category>
		<category><![CDATA[Private Integration]]></category>
		<category><![CDATA[Project Scoping]]></category>
		<category><![CDATA[Race Conditions]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[Reconciliation]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Sub-Account Management]]></category>
		<category><![CDATA[Webhooks]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=647</guid>

					<description><![CDATA[<p>Adding another workflow is usually the cheap answer, until it isn't. Six signals that a GoHighLevel account has outgrown the builder, the honest case for staying in it, and the three levels of code to try before committing to a marketplace app.</p>
<p>The post <a href="https://john-nessime.com/blog/workflow-automation/gohighlevel-custom-development-vs-workflows/">Six Signs GoHighLevel Needs a Developer, Not Another Workflow</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The account had forty-one workflows. Two were named &#8220;Lead Follow-Up v3&#8221; and &#8220;Lead Follow-Up v3 (USE THIS ONE)&#8221;. When someone asked what happens to a contact who submits the form twice in one afternoon, nobody could answer. Not the agency owner, not the person who built it.</p>



<p class="wp-block-paragraph">That is the moment. Not a crash, not an error banner. A plain question about the system that nobody can answer from the canvas.</p>



<p class="wp-block-paragraph">GoHighLevel is good at what it was built for and most accounts never need anything else. But there is a point where the next workflow makes the account worse instead of better, and past it, <strong>GoHighLevel custom development</strong> costs less than another six months of patching. Below: the failure mode hiding under all the others, six signals you have crossed that line, the honest case for staying in the builder, and three levels of code to try before committing to a full app.</p>



<h2 class="wp-block-heading">The failure hiding under all the others</h2>



<p class="wp-block-paragraph">The workflow builder has no failure semantics. Everything below is a consequence of that.</p>



<p class="wp-block-paragraph">When a step works, the contact moves on. When a step fails, the contact usually also moves on. No transaction, no rollback, no retry budget you control, no way to say &#8220;this contact is in a bad state, stop and tell someone&#8221;. A Custom Webhook that gets a 500 back does not park the contact for review. It becomes a line in an execution log nobody reads, and the contact continues as though the other system had answered.</p>



<p class="wp-block-paragraph">Fine when the step sends a text message. Not fine when the step is the only thing keeping two systems in agreement. So the useful test is not &#8220;is this hard to build in the builder&#8221;, because plenty of hard things build fine. The test is: <em>what happens when this step fails, and would I find out?</em></p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Signal one: you are storing state in tags</h2>



<p class="wp-block-paragraph">Tags start as labels. Then someone needs to know whether a lead was called, so there is a <code>called</code> tag. Then whether they were called twice, so <code>called-2</code>. Then a workflow that removes <code>called</code> when a new form arrives, and another that puts it back if an appointment was booked first.</p>



<p class="wp-block-paragraph">Those are no longer labels. That is a state machine with no diagram, no validation, and no guarantee that only one workflow writes to it at a time. Two workflows fired by the same event will both read the old value and both write, and whichever finishes last wins. That is a race condition, and it hides during testing because you test one contact at a time. The tell is a tag name with a number in it, or a tag whose meaning needs the word &#8220;unless&#8221; to explain.</p>



<h2 class="wp-block-heading">Signal two: the data has to be right, not eventually right</h2>



<p class="wp-block-paragraph">Marketing automation tolerates drift. One extra nurture email hurts nobody. The moment GoHighLevel holds something that must reconcile with another system, that tolerance is gone. Order status from a store, invoice state from an accounting package, availability shared with a clinic system.</p>



<p class="wp-block-paragraph">Those need idempotency, meaning a repeated message produces the same result as a single one, and reconciliation, meaning something periodically compares both sides and reports the difference. The builder gives you neither: no natural home for an idempotency key, no scheduled job that reads both systems and complains. You can fake reconciliation with a workflow on a timer, but that is a batch job written in a UI designed for follow-up sequences, and the first time it half-runs you will not hear about it.</p>



<h2 class="wp-block-heading">Signal three: per-execution billing became a per-lead tax</h2>



<p class="wp-block-paragraph">The actions that make workflows powerful are premium actions, billed per execution against the agency wallet. Custom Webhook, Inbound Webhook, Custom Code and the workflow AI actions all sit in that category. HighLevel gives each sub-account a small free allowance when premium actions are first switched on, publishes volume tiers above that, and lets you rebill sub-accounts individually instead of absorbing the charge. Take the current rate and tiers from HighLevel&#8217;s billing documentation rather than any blog post, including this one.</p>



<p class="wp-block-paragraph">The mechanism is the point. You pay per execution, not per useful outcome. A workflow calling out three times to enrich, score and push a lead is three executions per lead. Multiply by volume and sub-accounts, and the plumbing line can pass the cost of a small service doing all three in one request.</p>



<p class="wp-block-paragraph">Two caveats. Consolidating three calls into one endpoint is usually cheaper than moving the whole thing to code. And whatever you move still costs something to run, whether that is a VPS from a provider like Contabo or InterServer, a hosted platform such as Make or n8n, or a serverless function. Compare both sides.</p>



<h2 class="wp-block-heading">Signal four: the same logic lives in twenty sub-accounts</h2>



<p class="wp-block-paragraph">Snapshots are the platform&#8217;s answer and better than people expect. You can refresh a snapshot and push selected assets to sub-accounts that previously imported it, choosing both the assets and the recipients, and the push is selective rather than destructive.</p>



<p class="wp-block-paragraph">Two limits. Push Update only reaches sub-accounts inside your own agency, so anything shared externally has to be re-imported through a new link. And HighLevel documents that when a snapshot refresh deletes steps, contacts waiting on those steps in workflows created from that snapshot are removed. Fine for a copy change, destructive for a structural one.</p>



<p class="wp-block-paragraph">Snapshots distribute structure. They do not give you versioning, a rollback path, a diff between intended and running state, or a test that fails before the change reaches a client. Once a change has to land correctly in twenty accounts on one day, you want those, and they come from putting the logic behind an API you control and calling it from a thin step in each account.</p>



<h2 class="wp-block-heading">Signal five: the integration must survive the other side being down</h2>



<p class="wp-block-paragraph">A Custom Webhook fires the moment the contact reaches the step. If your endpoint is slow, restarting or rate limited, that single firing is what you get. No queue in front, no backoff behind.</p>



<p class="wp-block-paragraph">The API side has numbers worth knowing. HighLevel documents a burst limit of 100 requests per 10 seconds and a daily limit of 200,000 requests, counted per marketplace app per resource, where a resource is a location or the company. Every response carries headers telling you where you stand, so backoff can follow what the server says instead of a guessed sleep.</p>



<pre class="wp-block-code"><code>curl -sS -D - -o /dev/null 
  -H "Authorization: Bearer $GHL_TOKEN" 
  -H "Version: $GHL_API_VERSION" 
  "$GHL_API_BASE/contacts/" 
  | grep -i '^x-ratelimit'</code></pre>



<p class="wp-block-paragraph"><code>-D -</code> writes response headers to stdout while <code>-o /dev/null</code> discards the body, so you read the rate limit state without dumping contact data into your shell history. Set the base URL and the required <code>Version</code> value from the current API reference. Watch <code>X-RateLimit-Max</code> and <code>X-RateLimit-Remaining</code> for the burst window, <code>X-RateLimit-Interval-Milliseconds</code> for its length, and <code>X-RateLimit-Limit-Daily</code> with <code>X-RateLimit-Daily-Remaining</code> for the day.</p>



<p class="wp-block-paragraph">The daily budget sounds enormous until a migration. Reading a contact, upserting it, then writing custom fields is three calls each, so forty thousand contacts is 120,000 calls before live traffic touches the account. Batch the reads, skip writes where nothing changed, run backfills overnight. Watch for loops too: writes made through the API can themselves fire workflow triggers, so a receiver that writes back has built a cycle.</p>



<h2 class="wp-block-heading">Signal six: someone needs a screen, not a form</h2>



<p class="wp-block-paragraph">Forms collect. They do not display. Once a client asks to see a filtered view of their own data, approve something in two clicks, or work a queue that is not a pipeline, you are looking at an interface.</p>



<p class="wp-block-paragraph">The platform supports this. Marketplace apps can add Custom Pages, which render inside HighLevel in an embedded iframe loading a URL you host, appearing on the app details page or in the left navigation. HighLevel publishes an app template to start from, plus an SSO mechanism so your page can identify the user instead of asking them to log in again. That is still a real application with hosting, auth and deployment: the biggest step on this list, the one I would defer longest, and the only honest answer when the requirement is a screen.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Where the workflow builder genuinely wins</h2>



<p class="wp-block-paragraph">A one-sided argument is not worth reading, so here is the case against writing code.</p>



<ul class="wp-block-list">
<li><strong>The person who owns the outcome can change it.</strong> An account manager adjusts send timing on a Friday with no deploy and no developer awake.</li>



<li><strong>Messaging and scheduling are solved.</strong> SMS, email, calendars, reminders and the delivery infrastructure behind them are the platform&#8217;s core. Rebuilding any of it is a bad trade.</li>



<li><strong>Timezone handling is built in.</strong> Time-based steps run in the account timezone or the contact&#8217;s own, falling back to the account when a contact has none.</li>



<li><strong>No pipeline to maintain.</strong> No repository, no secrets rotation, no dependency upgrades, no server needing patched at an awkward hour.</li>



<li><strong>Execution history is right there.</strong> For a linear sequence, reading what happened to a contact beats correlating logs across two systems.</li>
</ul>



<p class="wp-block-paragraph">If the automation is a follow-up sequence with branching, the builder is correct and code is the expensive mistake.</p>



<h2 class="wp-block-heading">Three levels before you build an app</h2>



<p class="wp-block-paragraph">&#8220;Needs a developer&#8221; is not one decision. It is three, and most accounts only need the first.</p>



<h3 class="wp-block-heading">Level one: the Custom Code action</h3>



<p class="wp-block-paragraph">The builder has a Custom Code premium action that runs JavaScript inside the workflow. Values from earlier steps are mapped in as named input properties and arrive in an <code>InputData</code> object. The action returns a JavaScript object or an array of objects for later steps to reference, and <code>console.log</code> output is captured so you can see what ran.</p>



<pre class="wp-block-code"><code>// Normalise a phone number before it reaches a branch step.
// rawPhone and country are input properties mapped from earlier steps.
const digits = String(InputData.rawPhone || "").replace(/D/g, "");
const cc = InputData.country === "US" ? "1" : "";

if (!digits) {
  console.log("no phone on contact, routing to manual branch");
  return { e164: "", usable: false };
}

return {
  e164: "+" + (digits.startsWith(cc) ? digits : cc + digits),
  usable: digits.length &gt;= 10
};</code></pre>



<p class="wp-block-paragraph">This kills a lot of branch sprawl. Ten If/Else steps existing only to reshape a value collapse into one action returning a clean value and a boolean. Still inside the platform, still billed per execution, still no retries. But logic smeared across a canvas now reads in one place.</p>



<h3 class="wp-block-heading">Level two: a private integration and a script you own</h3>



<p class="wp-block-paragraph">For one account you do not need a marketplace app. Create a Private Integration inside the sub-account, select only the scopes the job needs, and use the token as a bearer credential. No OAuth handshake, no refresh cycle, no listing.</p>



<p class="wp-block-paragraph">This is where reconciliation jobs, nightly audits, migrations and bulk cleanups belong. Run it as a cron job on a small VPS, behind a Cloudflare Tunnel if it needs an inbound endpoint, with the token in a secrets store rather than a config file. Treat that token as a credential with a rotation plan, not a string you paste once and forget.</p>



<h3 class="wp-block-heading">Level three: a marketplace app</h3>



<p class="wp-block-paragraph">You need this when the thing spans many accounts, has a user interface, or is a product you intend to sell. It means OAuth 2.0 on the authorization code grant, token storage and refresh per installed location, webhook handling, and somewhere to run it. Note the version boundary: the v1 API is end of life and unmaintained, and new builds target the OAuth-based v2. If a tutorial hands you a permanent location API key, it is describing the old world.</p>



<p class="wp-block-paragraph">Marketplace apps can also register their own workflow triggers and actions, which appear in the builder for sub-accounts that installed the app. That is the pattern I reach for first: the hard logic lives in your service, and the person who owns the campaign still drags a step onto a canvas. It requires the workflows read scope, and those custom actions are themselves premium actions billed per execution.</p>



<h2 class="wp-block-heading">Arguments that do not survive contact</h2>



<ul class="wp-block-list">
<li><strong>&#8220;We will just add one more workflow.&#8221;</strong> The cost is not building it. It is that every future workflow now has to be reasoned about alongside it. Ten workflows listening for one tag is not ten problems, it is the interactions between them.</li>



<li><strong>&#8220;No-code means no developer.&#8221;</strong> It means no deployment. Someone still holds the whole model in their head, and when they leave, an undocumented canvas is harder to inherit than a repository with a README.</li>



<li><strong>&#8220;Zapier or Make will handle it.&#8221;</strong> Often true and worth trying first. What they do not give you is one place where business logic lives. Split rules across three tools and debugging means three log formats.</li>



<li><strong>&#8220;It works, so it is fine.&#8221;</strong> Every silent failure worked right up until someone counted. Absence of errors in a system with no error reporting is not evidence.</li>



<li><strong>&#8220;Custom code is lock-in.&#8221;</strong> Backwards. Logic on a canvas is bound to one vendor&#8217;s UI and exports only as a snapshot. Logic in a repository moves.</li>
</ul>



<h2 class="wp-block-heading">How I would decide</h2>



<ol class="wp-block-list">
<li><strong>Write down what happens when each external call fails.</strong> If the answer anywhere is &#8220;the contact continues as if it succeeded&#8221; and that matters, you have found your requirement.</li>



<li><strong>Count the workflows writing to the same tag or field.</strong> More than two means you have a state machine. Draw it before touching anything.</li>



<li><strong>Count premium executions per lead, then multiply by monthly volume.</strong> Compare against a few hours a month maintaining a small service. Sometimes the builder still wins. Do the arithmetic instead of assuming.</li>



<li><strong>Ask how a change reaches every account.</strong> If it involves opening sub-accounts one at a time, that is the real cost, and it grows with every client.</li>



<li><strong>Try the smallest level that fits, and leave the trigger in the builder.</strong> Custom Code action, then private integration script, then marketplace app. Whatever moves out, keep the entry point as a workflow step so the marketing side can still change timing and audience without you.</li>
</ol>



<p class="wp-block-paragraph">Whatever you build, monitor it. An uptime check on your receiver and an alert when the reconciliation job reports a non-zero difference will catch more real problems than any amount of careful workflow naming.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need a marketplace app to use the GoHighLevel API?</h3>



<p class="wp-block-paragraph">No. For a single sub-account, create a Private Integration inside that account, grant only the scopes you need, and use the token as a bearer credential. Marketplace apps and OAuth are for anything spanning accounts or distributed to other people.</p>



<h3 class="wp-block-heading">How much automation should stay in workflows?</h3>



<p class="wp-block-paragraph">The parts a non-developer should be able to change: timing, audience, message copy, branching on obvious contact attributes. Move out the parts where being wrong is expensive and being wrong is silent.</p>



<h3 class="wp-block-heading">Why does my workflow show as successful when the integration clearly failed?</h3>



<p class="wp-block-paragraph">A step firing and a step achieving something are different events, and the builder records the first. A webhook returning an error still counts as having run. If the outcome matters, verify it explicitly rather than trusting the log.</p>



<h3 class="wp-block-heading">Should I avoid premium workflow actions on cost grounds?</h3>



<p class="wp-block-paragraph">Not on principle. They cost per execution because they are the expensive verbs. The problem is fragmentation: three separate calls where one would do triples the bill for the same outcome. Consolidate before considering a move off the platform.</p>



<h3 class="wp-block-heading">What does a GoHighLevel custom development project usually start with?</h3>



<p class="wp-block-paragraph">An audit, not a build. Map which workflows write to which fields, where external calls happen, and what the failure path is for each. Often that document alone removes the need for the project.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Ask what happens when a step fails. That question separates automation belonging in the builder from automation needing code, more reliably than counting steps, counting workflows, or judging how clever the branching has become.</p>



<p class="wp-block-paragraph">If failure is visible and cheap, stay in the builder and enjoy that your client can edit it themselves. If failure is silent and expensive, GoHighLevel custom development stops being an upgrade and becomes maintenance you already deferred. Then start small: a Custom Code action beats a marketplace app you will not want to maintain.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Working on this with me</h2>



<p class="wp-block-paragraph">I work on the engineering side of GoHighLevel accounts, usually the parts failing quietly rather than the parts that look broken.</p>



<ul class="wp-block-list">
<li>Auditing an account for silent failures: which workflows write to which tags and fields, and what happens on every external call that returns an error.</li>



<li>Replacing branch sprawl with Custom Code actions, so logic reads as one block instead of fifteen steps on a canvas.</li>



<li>Webhook receivers and sync services that retry, deduplicate and reconcile, with the rate limit headers driving backoff instead of a fixed sleep.</li>



<li>Migrations and backfills planned against the daily API budget, batched and run in the account&#8217;s timezone so live traffic keeps working.</li>



<li>Private integration setup with least-privilege scopes, token storage and a rotation plan that is written down.</li>



<li>Marketplace app work when genuinely warranted: OAuth handling, custom workflow actions, and Custom Pages hosted somewhere you control.</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion, send something concrete: a screenshot of the workflow you are least sure about, an execution log entry that makes no sense, or the response body your webhook endpoint is returning. That is a faster conversation than a scoping call.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/workflow-automation/gohighlevel-custom-development-vs-workflows/">Six Signs GoHighLevel Needs a Developer, Not Another Workflow</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>One Customer, Four Systems: Unifying Shopify, Stripe, CRM and Support Data</title>
		<link>https://john-nessime.com/blog/technical-guides/unify-shopify-stripe-crm-support-data/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 29 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Customer 360]]></category>
		<category><![CDATA[Customer Support Analytics]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Data Warehouse]]></category>
		<category><![CDATA[dbt]]></category>
		<category><![CDATA[Ecommerce Analytics]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Event-Driven Architecture]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Identity Resolution]]></category>
		<category><![CDATA[PII Redaction]]></category>
		<category><![CDATA[Reverse ETL]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[Stripe]]></category>
		<category><![CDATA[Webhooks]]></category>
		<category><![CDATA[Zendesk]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=350</guid>

					<description><![CDATA[<p>Three CRM records, one person, and nothing in the logs to explain it. A practical walkthrough of what actually breaks when you unify Shopify, Stripe, CRM and support data: guest-checkout identity, duplicate and out-of-order webhooks, revenue that never ties to payouts, deletion requests that miss half your copies, and reverse ETL write loops. Organized by failure family, with schema and handler patterns you can apply directly.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/unify-shopify-stripe-crm-support-data/">One Customer, Four Systems: Unifying Shopify, Stripe, CRM and Support Data</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The project usually starts with a screenshot. Someone in support pastes three CRM records into a channel, all carrying the same person&#8217;s name, and asks which one is real. One has the order history. One has the subscription. The third has four tickets attached and no orders at all.</p>



<p class="wp-block-paragraph">Nobody broke anything. Shopify, Stripe, the CRM and the helpdesk each created a customer record using whatever identifier they had at the moment they needed one, and none of those identifiers agreed. The systems are all correct in isolation. The person is the thing that got lost.</p>



<p class="wp-block-paragraph">This post covers what actually breaks when you unify Shopify, Stripe, CRM and support data, and how to build the join so it survives guest checkouts, duplicate webhooks, refunds, currency conversion and deletion requests. It&#8217;s organized by failure family rather than by tool, because the tools get acquired and renamed every couple of years and the failures never change.</p>



<h2 class="wp-block-heading">Failure family one: the join key is not the email address</h2>



<p class="wp-block-paragraph">Every one of these systems will happily give you an email address, which is exactly why so many pipelines join on it and quietly produce garbage.</p>



<p class="wp-block-paragraph">Start with where identity actually lives in each system:</p>



<ul class="wp-block-list">
<li><strong>Shopify</strong> has a customer ID, but an order does not have to be attached to one. Guest checkout produces an order with an email on it and no durable customer record behind it.</li>

<li><strong>Stripe</strong> has a customer ID, but only if a Customer object was created. Checkout Sessions that don&#8217;t create one are associated with guest customers instead, which Stripe groups in the Dashboard based on the same card, email or phone. That grouping is read-only and it is not an ID you can join on.</li>

<li><strong>The CRM</strong> keys on email in almost every default configuration, and cheerfully creates a second contact when the email differs by one character.</li>

<li><strong>The helpdesk</strong> keys on the requester&#8217;s email, which is whatever the person typed into a contact form or sent the ticket from. It is very often not the address on the order.</li>
</ul>



<p class="wp-block-paragraph">Here&#8217;s the part that stays invisible until someone finally checks: nothing errors. Every dashboard still returns a number. Lifetime value gets silently split across three profiles, so your LTV looks lower and your repeat-purchase rate looks worse than it is. Support metrics look <em>better</em> than they are, because a customer coming back with the same unresolved problem shows up as a brand new contact. You can run like that for a year without a single failed job.</p>



<h3 class="wp-block-heading">Build an identifier table, not a fuzzy match</h3>



<p class="wp-block-paragraph">The fix is boring and it works: a surrogate person key, and a separate table of every identifier ever observed for that person, with the source that reported it.</p>



<pre class="wp-block-code"><code>-- One row per real human, generated by you, owned by you.
CREATE TABLE person (
  person_key      uuid PRIMARY KEY,
  created_at      timestamptz NOT NULL DEFAULT now(),
  merged_into     uuid NULL REFERENCES person(person_key)
);

-- One row per identifier per source. This is the whole trick.
CREATE TABLE person_identifier (
  person_key      uuid NOT NULL REFERENCES person(person_key),
  source          text NOT NULL,   -- shopify | stripe | crm | helpdesk
  id_type         text NOT NULL,   -- email | phone | shopify_customer_id | stripe_customer_id
  id_value        text NOT NULL,   -- already normalized
  first_seen_at   timestamptz NOT NULL,
  last_seen_at    timestamptz NOT NULL,
  PRIMARY KEY (source, id_type, id_value)
);</code></pre>



<p class="wp-block-paragraph">Normalization happens before the row is written, not in the join. Lowercase and trim emails, store phone numbers in E.164, strip whitespace from external IDs. Resist the urge to get clever with Gmail&#8217;s dot and plus handling. Stripping <code>+tags</code> is a business policy decision with real consequences, not a technical detail, and the same rule applied to a domain that treats plus addresses as distinct mailboxes will merge two genuinely different people.</p>



<p class="wp-block-paragraph">Deterministic matching first, always. Exact matches on shared identifiers, transitively closed. Probabilistic matching on name plus address plus device is worth reaching for only when you have exhausted the deterministic keys, and it has an asymmetric failure mode: a household sharing one card gets collapsed into one person, and unmerging afterwards is expensive because every downstream system has already copied the merged key into its own records. Merges are cheap to make and painful to undo. Bias toward leaving two records separate.</p>



<p class="wp-block-paragraph">One thing worth doing early on the Stripe side: when you create a Customer, write your own identifier into <code>metadata</code>. Stripe allows fifty key-value pairs per object, with keys up to forty characters and values up to five hundred, and it only returns metadata for requests made with a secret key. That&#8217;s enough room for a person key and an order reference, which turns a future fuzzy join into an exact one. Never put anything sensitive there.</p>



<h2 class="wp-block-heading">Failure family two: webhooks arrive twice, out of order, or not at all</h2>



<p class="wp-block-paragraph">Both platforms are explicit about this, and both get treated as if they weren&#8217;t.</p>



<p class="wp-block-paragraph">Stripe delivers at least once, does not guarantee ordering, and retries failed deliveries with exponential backoff for up to three days. The event <code>id</code> (the one starting <code>evt_</code>) stays identical across every retry, which makes it your deduplication key. Shopify sends <code>X-Shopify-Webhook-Id</code> for the same purpose, signs the raw body into <code>X-Shopify-Hmac-Sha256</code>, and states plainly that ordering isn&#8217;t guaranteed within a topic or across topics for the same resource. Shopify&#8217;s own documentation goes further and tells you not to rely on webhooks as your only source of data.</p>



<p class="wp-block-paragraph">For a customer-data pipeline specifically, out-of-order delivery is nastier than duplication. Consider a <code>customers/update</code> that changes someone&#8217;s email arriving before the <code>orders/create</code> that was placed under the old address. Process those in arrival order and you attach the order to a person who doesn&#8217;t exist yet, or worse, to the wrong one. Order your writes by the event&#8217;s own timestamp, using <code>X-Shopify-Triggered-At</code> or the <code>updated_at</code> in the payload, and make profile updates last-writer-wins on that timestamp rather than on insertion time.</p>



<h3 class="wp-block-heading">Do almost nothing in the handler</h3>



<p class="wp-block-paragraph">Verify the signature, claim the event ID, write the raw body to durable storage, return 2xx. That&#8217;s it. Identity resolution, CRM lookups and enrichment calls all belong in a worker reading from that store. A handler that tries to resolve identity inline will eventually be slow enough to time out, which triggers a retry, which arrives while the first attempt is still running.</p>



<p class="wp-block-paragraph">Make the claim atomic rather than check-then-act, because two retries genuinely can land concurrently:</p>



<pre class="wp-block-code"><code>-- Atomic claim. If this inserts zero rows, someone already has it.
INSERT INTO webhook_event (source, event_id, topic, received_at, payload)
VALUES ($1, $2, $3, now(), $4)
ON CONFLICT (source, event_id) DO NOTHING
RETURNING event_id;</code></pre>



<p class="wp-block-paragraph">Return a 2xx for a duplicate. A duplicate is a success from the sender&#8217;s point of view, and returning an error just schedules more of them.</p>



<p class="wp-block-paragraph">Signature verification has one recurring trap on the Shopify side: it runs against the <em>raw</em> body. Any JSON body parser that runs first will change the bytes and every signature will fail. Capture the raw body before parsing, and compare in constant time.</p>



<pre class="wp-block-code"><code>import base64, hashlib, hmac

def verify_shopify(raw_body: bytes, header_value: str, client_secret: str) -&gt; bool:
    digest = hmac.new(
        client_secret.encode("utf-8"),
        raw_body,
        hashlib.sha256,
    ).digest()
    expected = base64.b64encode(digest).decode("utf-8")
    return hmac.compare_digest(expected, header_value)</code></pre>



<p class="wp-block-paragraph">Worth knowing before you spend an afternoon on it: after rotating an app&#8217;s client secret, Shopify notes it can take up to an hour before signatures are generated with the new one. If verification starts failing right after a rotation, that&#8217;s usually why.</p>



<p class="wp-block-paragraph">And because delivery isn&#8217;t guaranteed, webhooks alone are never enough. Run a reconciliation job that pulls orders and customers modified since the last watermark and fills the gaps. This is the job that saves you during an outage on your side, and it&#8217;s the one people skip. If you&#8217;d rather not build the buffering and replay layer, managed webhook infrastructure like Hookdeck exists for exactly this, and self-hosting a small queue and worker on a modest VPS from somewhere like Contabo or InterServer is perfectly reasonable too. The point is that something has to catch what the webhook missed.</p>



<h2 class="wp-block-heading">Failure family three: &#8220;revenue&#8221; means four different things</h2>



<p class="wp-block-paragraph">This is the one that turns into a meeting. Finance says the number is wrong. Analytics says the number is right. Both are correct, because they&#8217;re measuring different things and nobody wrote down which.</p>



<ul class="wp-block-list">
<li>Shopify reports gross sales at the moment the order is placed, in the store&#8217;s currency and the store&#8217;s timezone.</li>

<li>Stripe pays out in batches, net of processing fees, refunds and disputes, on its own schedule, in the settlement currency of the account.</li>

<li>A refund issued in Shopify and a refund issued in Stripe are two different events that may or may not be the same underlying money movement.</li>

<li>A dispute lands days or weeks after the order and reverses revenue in a period that has already been reported.</li>
</ul>



<p class="wp-block-paragraph">There&#8217;s a structural trap here too. If the store runs Shopify Payments, there is no Stripe account for you to query. Shopify Payments is built on Stripe&#8217;s processing infrastructure, but it&#8217;s Shopify&#8217;s gateway, and payouts, fees and disputes live in the Shopify admin rather than behind a Stripe API key. Plenty of businesses end up with Shopify Payments handling storefront orders and a separate Stripe account handling subscriptions or invoicing, which means two payment systems, two payout schedules and one customer who has no idea any of this exists.</p>



<p class="wp-block-paragraph">Do not build a single <code>revenue</code> column. Build separate fact tables at their natural grain (orders, order line items, payments, refunds, disputes, payouts) plus a bridging table that links a payout back to the transactions inside it. Then let each team define its own metric on top. Finance gets net settled cash. Marketing gets gross booked revenue. Neither has to be wrong for the other to be right.</p>



<p class="wp-block-paragraph">Store every timestamp in UTC and keep the original timezone as a separate column. Store every amount in minor units as an integer alongside its currency code, and store the FX rate that was actually applied rather than recalculating it later from a rates table. Recalculated FX is the single most common source of &#8220;the number moved and nobody changed anything.&#8221;</p>



<p class="wp-block-paragraph">On ingestion, Stripe Data Pipeline can push Stripe data directly into Snowflake, Redshift or Databricks as a data share, which removes a connector from the picture if you&#8217;re already on one of those. Otherwise a managed ELT tool like Fivetran or a self-hosted Airbyte will handle Shopify, Stripe, the CRM and the helpdesk with off-the-shelf connectors. Both routes are fine. The connector is genuinely the easy part.</p>



<h2 class="wp-block-heading">Failure family four: deletion is the requirement nobody models</h2>



<p class="wp-block-paragraph">If you&#8217;re building this as a Shopify app, three compliance webhooks are mandatory: <code>customers/data_request</code>, <code>customers/redact</code> and <code>shop/redact</code>. You acknowledge with a 200 and complete the action within thirty days. <code>shop/redact</code> fires forty-eight hours after uninstall and covers the shop&#8217;s data. These are enforced at app review, so you&#8217;ll find out about them either way.</p>



<p class="wp-block-paragraph">The interesting problem isn&#8217;t receiving the webhook. It&#8217;s that by the time it arrives, that person&#8217;s email exists in your raw landing zone, your staging tables, your identity table, your customer view, a BI extract, the CRM, the helpdesk, and a Slack message from a debugging session six months ago. A delete that only touches the golden record is not a delete.</p>



<p class="wp-block-paragraph">The design decision that makes this tractable is to hold personal data in exactly one place. Every downstream table carries <code>person_key</code> and nothing else. Facts, aggregates, models, extracts and syncs all reference the key. When a redaction request lands, you redact one row set in the identifier table and the rest of the warehouse is already anonymous. Getting this right at schema-design time costs an afternoon; retrofitting it costs a quarter.</p>



<p class="wp-block-paragraph">Practical pieces that make it real:</p>



<ul class="wp-block-list">
<li>Tag every table and column that can contain an identifier, in your catalog or in dbt metadata, and treat the tag list as the deletion scope.</li>

<li>Partition the raw landing zone so you can rewrite a partition rather than scan an entire bucket looking for one email.</li>

<li>Keep a deletion queue with a status per downstream system, including the CRM and the helpdesk, which have their own APIs and their own idea of what deletion means.</li>

<li>Write down what you retain and why. Financial records generally have to survive a deletion request for tax and dispute reasons. Say so rather than promising a hard delete you can&#8217;t perform.</li>
</ul>



<h2 class="wp-block-heading">Failure family five: pushing the unified view back out</h2>



<p class="wp-block-paragraph">A customer view nobody can act on is a report. The value shows up when a support agent opens a ticket and can see subscription status and lifetime order count without switching tabs.</p>



<p class="wp-block-paragraph">That&#8217;s reverse ETL, and the market is well served: Hightouch, Census (now part of Fivetran), Polytomic, RudderStack, and iPaaS-style tools like Workato if you need branching logic rather than field syncs. The warehouse-native argument is straightforward and mostly correct: your models already live there, and you avoid maintaining a second copy of the customer profile inside a vendor&#8217;s rigid schema. The counter-argument is also real. Reverse ETL is batch by nature, so if you need sub-minute reaction to an event, a packaged CDP or a direct event pipeline will serve you better, and you&#8217;ll be running several vendors plus warehouse compute plus the people to maintain the models.</p>



<p class="wp-block-paragraph">Three things go wrong here more or less every time.</p>



<p class="wp-block-paragraph"><strong>The write loop.</strong> You compute a churn-risk field and sync it into the CRM. Your ELT connector reads the CRM back into the warehouse. Now the computed field is an input to the model that computes it. This is genuinely hard to spot because nothing errors, the value just drifts. Namespace every synced field with a consistent prefix and explicitly exclude that namespace from ingestion.</p>



<p class="wp-block-paragraph"><strong>The resync stampede.</strong> Incremental syncs push diffs and stay small. Change a model definition and the next run wants to update every record, which will meet a helpdesk API rate limit at speed. Cap the batch size, and treat a full resync as a planned operation rather than something a merge to main can trigger.</p>



<p class="wp-block-paragraph"><strong>Syncing logic instead of keys.</strong> Push <code>person_key</code> into a custom field on the CRM contact and the helpdesk user. It makes the key searchable by agents, it gives the next pipeline a deterministic join, and it stops three teams from each rebuilding their own version of your matching rules.</p>



<h2 class="wp-block-heading">A pipeline shape that holds up</h2>



<p class="wp-block-paragraph">Put together, the order of operations matters more than the tool choice:</p>



<ol class="wp-block-list">
<li>Land raw. Every webhook body and every API page goes to immutable storage first, keyed by source and event ID, before anything interprets it.</li>

<li>Deduplicate on the provider&#8217;s event ID, atomically, with a retention window longer than the provider&#8217;s retry window.</li>

<li>Normalize identifiers into the identifier table as rows arrive, with the source recorded.</li>

<li>Resolve identity deterministically into <code>person_key</code>, as a scheduled job, never inside a request handler.</li>

<li>Build fact tables per system at their own grain, keyed on <code>person_key</code>, with no raw PII.</li>

<li>Model the customer view on top, in dbt or whatever your team already uses. The Fivetran <code>dbt_customer360</code> package is a reasonable starting point if you want to see how someone else structured the identity step.</li>

<li>Activate a small, named, namespaced set of fields back into the CRM and helpdesk.</li>

<li>Run reconciliation and freshness checks on a schedule, and alert on them like you would any other production job.</li>
</ol>



<h2 class="wp-block-heading">Troubleshooting: symptom to root cause</h2>



<p class="wp-block-paragraph">These are the ones that come up repeatedly, and the check that usually settles them.</p>



<ul class="wp-block-list">
<li><strong>Warehouse customer count is much higher than Shopify&#8217;s.</strong> Guest orders are minting a new person on every purchase. Check how many <code>person</code> rows have exactly one identifier and one order.</li>

<li><strong>Warehouse count is much lower.</strong> An over-eager merge rule. Look for person keys with an implausible number of distinct emails or shipping addresses attached.</li>

<li><strong>BI revenue doesn&#8217;t match Stripe payouts.</strong> Grain mismatch, not a data loss. Reconcile a single payout against its constituent transactions before touching anything else.</li>

<li><strong>A CRM field keeps flipping between two values.</strong> Write loop. Check whether the field&#8217;s namespace is excluded from your ELT source configuration.</li>

<li><strong>Recent orders missing, older ones fine.</strong> Webhook delivery gap. Confirm the reconciliation job ran, and check its watermark rather than trusting its exit code.</li>

<li><strong>Every Shopify signature suddenly fails.</strong> Either a body parser is running before verification, or the client secret was rotated within the last hour.</li>

<li><strong>Tickets attached to the wrong customer after a merge.</strong> The merge happened in the warehouse but the synced key in the helpdesk still points at the old person. Merges need to propagate outward, not just inward.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Joining on email in the model layer instead of resolving identity once, upstream.</li>

<li>Treating webhooks as the system of record instead of as a low-latency hint, with a reconciliation pull behind them.</li>

<li>Doing identity resolution, enrichment or CRM writes inside the webhook handler.</li>

<li>Ordering profile updates by arrival time rather than by the event&#8217;s own timestamp.</li>

<li>One <code>revenue</code> column that finance and marketing are both told to use.</li>

<li>Storing amounts as floats, or recalculating historical FX from a rates table.</li>

<li>Copying PII into every downstream table because it was convenient at the time.</li>

<li>Reaching for probabilistic matching before the deterministic keys have been exhausted.</li>

<li>Assuming a Shopify store on Shopify Payments has a Stripe account you can query.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>Own the surrogate key. Never let a vendor&#8217;s internal ID become the primary identity of a person in your systems.</li>

<li>Write your <code>person_key</code> into Stripe <code>metadata</code> and Shopify customer metafields at creation time, so future joins are exact rather than inferred.</li>

<li>Keep raw payloads immutable and replayable. Every model bug you&#8217;ll ever have is cheaper to fix if you can reprocess from raw.</li>

<li>Version your matching rules and record which version produced each merge, so you can audit and reverse a bad rule.</li>

<li>Alert on freshness and on row-count deltas per source, not just on job success. A connector returning zero rows exits cleanly.</li>

<li>Reconcile one payout end to end by hand before you trust the automated version. It&#8217;s the fastest way to find a grain error.</li>

<li>Document the metric definitions next to the models. Most &#8220;the data is wrong&#8221; tickets are definition disputes.</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Do I need a CDP to unify Shopify, Stripe, CRM and support data?</h3>



<p class="wp-block-paragraph">Not necessarily. If you already run a warehouse and have someone comfortable with dbt, ELT plus modelling plus reverse ETL covers the same ground with more control over the data model. A packaged CDP earns its cost when you need real-time behavioural segmentation, when nobody on the team can maintain models, or when marketing needs to ship changes without waiting on data engineering.</p>



<h3 class="wp-block-heading">How do I handle guest checkout customers?</h3>



<p class="wp-block-paragraph">Create a person from the order&#8217;s email anyway, and mark the record as identity-weak. When that email later appears attached to a real Shopify customer or Stripe Customer object, the deterministic match links them and the order history follows. The mistake is discarding guest orders because they have no customer ID, which is exactly the revenue you most want attributed.</p>



<h3 class="wp-block-heading">Should I sync everything into the CRM or keep it in the warehouse?</h3>



<p class="wp-block-paragraph">Keep the full picture in the warehouse and sync a deliberately small set of fields outward. The test is whether an agent or a rep will act on the field during a conversation. Lifetime value, subscription status, open ticket count and churn risk pass. A hundred-column profile does not, and it turns every model change into a rate-limit incident.</p>



<h3 class="wp-block-heading">Why don&#8217;t my Shopify sales match my Stripe payouts?</h3>



<p class="wp-block-paragraph">Because they&#8217;re measuring different things at different times. Shopify shows gross sales at order time; Stripe pays out batched amounts net of fees, refunds and disputes, and the timing differs by days. Add currency conversion and a timezone mismatch and the numbers will never tie without an explicit bridging table linking payouts to the transactions inside them.</p>



<h3 class="wp-block-heading">Can I use email as the primary key if my store requires account creation?</h3>



<p class="wp-block-paragraph">You can get away with it longer, but it&#8217;s still a bad primary key. People change email addresses, use one address for orders and another for support, and share addresses within a household. A surrogate key with an identifier table costs very little up front and means an email change is a new row rather than a broken join and a duplicate customer.</p>



<h3 class="wp-block-heading">How do I test the identity resolution before trusting it?</h3>



<p class="wp-block-paragraph">Run it in shadow mode. Compute person keys without activating them, then have support pick twenty customers they know are duplicates and twenty they know are distinct, and check what the resolver did. That exercise finds normalization bugs and over-eager merge rules faster than any aggregate metric, because the people who answer the tickets already know where the duplicates are.</p>



<h3 class="wp-block-heading">What&#8217;s the minimum viable version of this?</h3>



<p class="wp-block-paragraph">Raw landing with deduplication, an identifier table with deterministic matching, one fact table per source, and a single synced field into the helpdesk. That&#8217;s a couple of weeks of work and it removes most of the pain. Everything above it is refinement.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Conclusion: unify Shopify, Stripe, CRM and support data around a key you own</h2>



<p class="wp-block-paragraph">If you take one thing from this: the hard part of unifying Shopify, Stripe, CRM and support data is not moving the data. Connectors are commodity. The hard part is deciding what a customer <em>is</em>, writing that decision down as a key you own, and then refusing to let any downstream system quietly invent its own version of it.</p>



<p class="wp-block-paragraph">Do the identity work first, land everything raw so you can reprocess when you get it wrong, and keep personal data in one place so deletion stays a single operation. The dashboards will follow. Skip that order and you get exactly the screenshot this post opened with, eighteen months later and with a lot more rows behind it.</p>



<h2 class="wp-block-heading">Need help wiring this together?</h2>



<p class="wp-block-paragraph">I work on exactly this kind of plumbing: the unglamorous layer between an ecommerce platform, a payment processor and the tools your team actually opens all day. Things I can help with here:</p>



<ul class="wp-block-list">
<li>Designing the identity model (surrogate keys, identifier tables, deterministic matching rules) and auditing an existing one for over-merging.</li>

<li>Building idempotent, signature-verified webhook intake for Shopify and Stripe, with a dead-letter path and a reconciliation job behind it.</li>

<li>Setting up ELT into Snowflake, BigQuery, Redshift or Postgres, whether with Fivetran, self-hosted Airbyte, or Stripe Data Pipeline where it fits.</li>

<li>Modelling orders, payments, refunds, disputes and payouts so finance and marketing can each get a number they trust.</li>

<li>Reverse ETL back into the CRM and helpdesk without write loops, rate-limit incidents or a hundred junk fields.</li>

<li>Making deletion and data-subject requests actually work across the warehouse, the raw landing zone and every downstream copy.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;re stuck on a specific piece, send me the concrete thing: a webhook payload, a schema dump, the query that disagrees with your Stripe payout report. It&#8217;s much easier to say something useful about a real artifact than about a diagram.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/unify-shopify-stripe-crm-support-data/">One Customer, Four Systems: Unifying Shopify, Stripe, CRM and Support Data</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Grafana Alert Notifications That Actually Arrive: Email, Slack and Telegram</title>
		<link>https://john-nessime.com/blog/devops/grafana-alert-notifications-email-slack-telegram/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 27 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Alertmanager]]></category>
		<category><![CDATA[Contact Points]]></category>
		<category><![CDATA[Email Deliverability]]></category>
		<category><![CDATA[Grafana]]></category>
		<category><![CDATA[Incident Response]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Notification Policies]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[Prometheus]]></category>
		<category><![CDATA[Self Hosting]]></category>
		<category><![CDATA[Slack]]></category>
		<category><![CDATA[SMTP]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[Telegram]]></category>
		<category><![CDATA[Webhooks]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=258</guid>

					<description><![CDATA[<p>The rule went red at 02:04 and nobody found out until 09:15. Rule evaluation and notification delivery are two different systems, and only one of them is visible in the UI. A practical setup for Grafana alert notifications to email, Slack and Telegram, plus how to detect the day delivery quietly stops.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/grafana-alert-notifications-email-slack-telegram/">Grafana Alert Notifications That Actually Arrive: Email, Slack and Telegram</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The disk filled at 02:00. The rule went red at 02:04. Nobody found out until 09:15, when someone opened the alert list, saw a Firing rule with a five-hour age on it, and asked the obvious question: why didn&#8217;t anyone get paged?</p>



<p class="wp-block-paragraph">The rule was fine. The query was fine. The evaluation ran on schedule and the state transition happened exactly when it should have. What failed was everything after that, and none of it made a sound.</p>



<p class="wp-block-paragraph">That is the failure mode worth designing around. Rule evaluation and notification delivery are two different systems, and a red rule only proves the first one works. Grafana alert notifications run against SMTP servers, Slack&#8217;s API and Telegram&#8217;s bot endpoint, all of which fail in their own quiet ways. A rejected login, a revoked webhook, a message over a length limit: each produces a log line and nothing else. The alert list still shows the rule firing, so everything looks healthy from the one place you&#8217;d think to check.</p>



<p class="wp-block-paragraph">This post covers the practical setup for the three channels most small and mid-sized teams actually use: email, Slack and Telegram. It covers what breaks in each of them, how to route alerts so the right ones interrupt people and the rest don&#8217;t, how to provision the whole thing as files instead of clicking through the UI, and how to detect the case where delivery has stopped entirely.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Where a firing alert actually goes</h2>



<p class="wp-block-paragraph">Before touching any integration, it helps to know how many hops sit between &#8220;the rule fired&#8221; and &#8220;my phone buzzed&#8221;.</p>



<ul class="wp-block-list">
<li>The rule evaluates on its own schedule and produces alert instances, each carrying labels.</li>
<li>Those instances go to an Alertmanager. Grafana ships its own and can also use an external Prometheus Alertmanager.</li>
<li>The Alertmanager applies mute timings, groups instances, and walks the notification policy tree for a match.</li>
<li>The matched policy names a contact point, which holds one or more integrations.</li>
<li>Each integration makes its own outbound call to a third-party service.</li>
</ul>



<p class="wp-block-paragraph">Steps one and two are visible in the Grafana UI. Steps three through five are where things go wrong, and they are visible only in logs. That asymmetry is the whole problem.</p>



<p class="wp-block-paragraph">One thing worth knowing early: a contact point can hold several integrations at once, so a single contact point named <code>critical</code> can fire email, Slack and Telegram together. Building one contact point per channel instead makes routing much harder than it needs to be.</p>



<h2 class="wp-block-heading">Email: the channel everyone assumes works</h2>



<p class="wp-block-paragraph">Email is the one people configure first and verify least. The catch is that the email contact point holds only the recipient addresses. The actual SMTP credentials live in <code>grafana.ini</code>, so a perfectly valid contact point can sit there doing nothing because the server underneath was never configured.</p>



<pre class="wp-block-code"><code>[smtp]
enabled = true
host = smtp.example.com:587
user = alerts@example.com
# Wrap the password in triple quotes if it contains # or ;
password = """your-smtp-password"""
from_address = alerts@example.com
from_name = Grafana Alerts
skip_verify = false
# Default is OpportunisticStartTLS
startTLS_policy = MandatoryStartTLS</code></pre>



<p class="wp-block-paragraph">Several details in that block matter more than they look.</p>



<ul class="wp-block-list">
<li><strong>The port is part of <code>host</code>.</strong> There is no separate port setting. Port 587 with STARTTLS is the normal choice; 465 is implicit TLS and behaves differently, which is usually why one fails cleanly and the other hangs.</li>
<li><strong><code>startTLS_policy</code> decides how hard Grafana insists on encryption.</strong> The default is opportunistic: it upgrades the connection if the server offers it and quietly continues in the clear if not. On a relay you control, mandatory turns a silent downgrade into a visible failure.</li>
<li><strong><code>from_address</code> has to be something the relay accepts.</strong> Managed relays reject mail from unverified domains <em>after</em> the connection succeeds, which is why the Grafana side looks fine.</li>
<li><strong>Leave <code>skip_verify</code> false.</strong> It exists for internal relays with self-signed certificates. Turning it on to clear an error removes the check that was telling you the certificate is wrong.</li>
</ul>



<p class="wp-block-paragraph">Changes to this file need a Grafana restart. Nothing in the UI reloads SMTP settings.</p>



<h3 class="wp-block-heading">Deliverability is a separate problem from delivery</h3>



<p class="wp-block-paragraph">Sending directly from a VPS is the fastest way to get alerts filed as spam. A fresh IP range from a provider like Contabo or InterServer has no sending reputation, and a mail server that emits one message a week during incidents never builds one.</p>



<p class="wp-block-paragraph">Point Grafana at a transactional relay instead. Amazon SES, Postmark, Mailgun and Brevo all do the same job at this volume. Whichever you pick, publish SPF and DKIM records for the sending domain and set a DMARC policy, then send a real alert to a Gmail address and a Microsoft 365 address and check the spam folder. That is the only test that tells you anything.</p>



<h2 class="wp-block-heading">Slack: webhook or bot token</h2>



<p class="wp-block-paragraph">Grafana supports two ways of posting to Slack, and the choice has consequences beyond setup time.</p>



<p class="wp-block-paragraph"><strong>Incoming webhooks</strong> are the quick path. Slack generates a URL under <code>hooks.slack.com/services/</code> that is bound to a single channel, and Grafana posts to it. There&#8217;s no OAuth flow and no scopes to reason about. The trade-off is that the URL <em>is</em> the credential and it maps to exactly one channel, so a team with six alert channels ends up managing six secrets with no central view of them.</p>



<p class="wp-block-paragraph"><strong>A Slack app with a bot token</strong> takes longer. You create the app, add the <code>chat:write.public</code> scope so it can post to public channels without being invited to each one, then copy the bot user OAuth token, which starts with <code>xoxb-</code>. In Grafana you supply that token plus the channel ID as the recipient. One credential covers every channel, permissions are visible in one place, and revoking it kills all posting at once.</p>



<p class="wp-block-paragraph">Webhooks win when you have one or two channels and want to be done in five minutes. The bot token wins as soon as channels start multiplying or someone asks what the integration is permitted to do. I reach for the webhook on small setups and regret it around the fourth channel.</p>



<p class="wp-block-paragraph">Two constraints either way. A Slack integration targets one channel, so several channels means several integrations. And notification templates control the message title and body text, not its visual structure, so rich block layouts aren&#8217;t on the table.</p>



<h2 class="wp-block-heading">Telegram: cheap paging, one nasty edge</h2>



<p class="wp-block-paragraph">Telegram is genuinely useful for out-of-hours alerts because it pushes to a phone without anyone paying for a paging product. The setup is two values: a bot API token and a chat ID.</p>



<ol class="wp-block-list">
<li>Message BotFather and send <code>/newbot</code>. The username has to end in <code>bot</code> or <code>_bot</code>.</li>
<li>Copy the HTTP API token it returns. That&#8217;s the bot API token Grafana wants.</li>
<li>Create a group for alerts and add the bot to it. A bot cannot message a chat it has never seen.</li>
<li>Send a slash command in the group, such as <code>/start@your_bot_name</code>. An explicit command is the reliable way to make the group visible to the bot.</li>
<li>Fetch the chat ID from the Telegram API.</li>
</ol>



<pre class="wp-block-code"><code>curl -s "https://api.telegram.org/bot&lt;YOUR_BOT_TOKEN&gt;/getUpdates" | jq '.result[].message.chat'</code></pre>



<p class="wp-block-paragraph">The <code>id</code> field in that output is what goes in Grafana&#8217;s Chat ID field. Group chat IDs are negative. Copy the minus sign; leaving it off is the single most common reason a Telegram contact point tests as failing.</p>



<p class="wp-block-paragraph">Now the edge case, and it&#8217;s the reason Telegram belongs in a post about invisible failures. Telegram caps messages at 4096 UTF-8 characters. If you&#8217;re using a parse mode other than none, truncating at that boundary can leave the markup unbalanced, and Telegram rejects the whole message as malformed. You don&#8217;t get a shortened alert. You get nothing.</p>



<p class="wp-block-paragraph">This is exactly backwards, because the message most likely to blow the limit is a large grouped alert during a real outage. The channel works perfectly for months of single alerts, then goes quiet the first time forty instances fire together. Three ways to deal with it, in the order I&#8217;d try them:</p>



<ul class="wp-block-list">
<li>Write a deliberately short notification template for Telegram: alert name, severity, instance count and a dashboard link. Treat it as a pager, not a report.</li>
<li>Narrow <code>group_by</code> on the policy that feeds Telegram so a single notification covers fewer instances.</li>
<li>Set the parse mode to none, which makes truncation harmless at the cost of formatting.</li>
</ul>



<h2 class="wp-block-heading">Routing: split by severity, not by tool</h2>



<p class="wp-block-paragraph">The most common design mistake is a policy tree organised around channels: a Slack policy, an email policy, a Telegram policy. That gives you no way to express the thing you actually care about, which is that some alerts should wake someone up and most should not.</p>



<p class="wp-block-paragraph">Put severity on the alert rule as a label and route on that instead. Contact points then become bundles of channels chosen to match urgency.</p>



<ul class="wp-block-list">
<li><code>severity=critical</code> goes to a contact point holding Telegram and Slack. Something is broken and someone needs to look now.</li>
<li><code>severity=warning</code> goes to Slack only, in a channel people read during working hours.</li>
<li>Everything else falls through to the default policy and lands in email, where it accumulates without interrupting anyone.</li>
</ul>



<p class="wp-block-paragraph">Four timing settings control how noisy this is, and each one does something specific:</p>



<ul class="wp-block-list">
<li><strong><code>group_by</code></strong> decides what counts as the same notification. Grouping by <code>alertname</code> and a service label means one cascading failure produces one message instead of forty.</li>
<li><strong><code>group_wait</code></strong> (default 30 seconds) is how long to hold the first notification for a new group, so related instances that fire a few seconds apart arrive together.</li>
<li><strong><code>group_interval</code></strong> (default 5 minutes) is the minimum gap before a group that has already notified sends again because new instances joined it.</li>
<li><strong><code>repeat_interval</code></strong> (default 4 hours) is how often an unresolved alert re-notifies with nothing new to say.</li>
</ul>



<p class="wp-block-paragraph">The temptation is to drop <code>repeat_interval</code> to fifteen minutes so nothing gets forgotten. Resist it. Teams that do this stop reading the channel within a month, which converts a routing problem into a much worse cultural one. Longer intervals plus a real escalation path beats short intervals every time.</p>



<p class="wp-block-paragraph">Grafana also offers a simpler path: pick a contact point directly on the alert rule form and skip label matching entirely. Reasonable when a rule has one obvious destination. The trade-off is that routing knowledge moves into individual rules, so changing where a class of alerts goes means editing every rule instead of one policy.</p>



<h2 class="wp-block-heading">Provision it as files, not clicks</h2>



<p class="wp-block-paragraph">Contact points built through the UI are invisible to review, easy to delete by accident, and impossible to recreate after a bad restore. If Grafana alert notifications matter enough to build, they matter enough to keep in Git. Grafana reads alerting configuration from files in the <code>provisioning/alerting</code> directory under its provisioning path at startup.</p>



<p class="wp-block-paragraph">Before writing any YAML by hand, build one contact point in the UI, test it, then use the export option on the contact points list to dump it as YAML. That gives you the exact setting keys for that integration type instead of the ones you half-remember from a blog post. Setting names are not consistent between integration types, and a guessed key produces a config that loads without error and sends nothing.</p>



<pre class="wp-block-code"><code>apiVersion: 1

contactPoints:
  - orgId: 1
    name: critical-oncall
    receivers:
      # Telegram receiver goes here. Export it from the UI first:
      # the setting key names differ from what you would guess.
      - uid: critical-slack
        type: slack
        settings:
          url: https://hooks.slack.com/services/REPLACE/WITH/YOURS
        disableResolveMessage: false

  - orgId: 1
    name: warnings-slack
    receivers:
      - uid: warnings-slack
        type: slack
        settings:
          url: https://hooks.slack.com/services/REPLACE/WITH/YOURS

  - orgId: 1
    name: email-digest
    receivers:
      - uid: email-digest
        type: email
        settings:
          addresses: ops@example.com
          singleEmail: false</code></pre>



<p class="wp-block-paragraph">And the policy tree that uses them:</p>



<pre class="wp-block-code"><code>apiVersion: 1

policies:
  - orgId: 1
    receiver: email-digest
    group_by: [grafana_folder, alertname]
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - receiver: critical-oncall
        matchers:
          - severity = critical
        group_wait: 10s
        repeat_interval: 1h
      - receiver: warnings-slack
        matchers:
          - severity = warning
        repeat_interval: 12h</code></pre>



<p class="wp-block-paragraph">Two things to know about provisioned resources. By default they become read-only in the UI, which is the point but surprises people the first time. And a contact point that a policy references cannot be deleted until the policy stops referencing it, so removals have to happen in the right order.</p>



<p class="wp-block-paragraph">Keep tokens out of the files themselves. Grafana supports environment variable substitution in provisioning files, which means the YAML can live in Git and the secrets can come from your existing secret store.</p>



<h2 class="wp-block-heading">Detecting the silence</h2>



<p class="wp-block-paragraph">Everything above still leaves the original problem unsolved. If the Slack app gets revoked or the SMTP password rotates, notifications stop and Grafana does not tell you.</p>



<p class="wp-block-paragraph">The fix is a heartbeat alert, and it&#8217;s worth the twenty minutes. Create a rule whose condition is always true, give it a low severity label, route it to each contact point with a short <code>repeat_interval</code>, and point the resulting notification at an external dead-man&#8217;s-switch service. Healthchecks.io, Cronitor and Better Stack all offer this, and Healthchecks.io can be self-hosted if you&#8217;d rather not add a dependency. The service expects a signal on a schedule and alerts you when it stops arriving.</p>



<p class="wp-block-paragraph">The logic is inverted on purpose. You aren&#8217;t monitoring infrastructure here, you&#8217;re monitoring the path that tells you about your infrastructure, and it has to be watched by something that isn&#8217;t itself.</p>



<p class="wp-block-paragraph">As a second layer, Grafana exposes its own internal metrics on a Prometheus-compatible endpoint, including counters for notification attempts and failures. If you&#8217;re already scraping Grafana, alert on that failure counter increasing. It won&#8217;t catch a credential that works but delivers to a dead mailbox, which is why the heartbeat comes first.</p>



<h2 class="wp-block-heading">Troubleshooting: the rule fires, nothing arrives</h2>



<p class="wp-block-paragraph">Work down this list in order. Each step rules out one hop, and stopping early is how people spend an afternoon on the wrong layer.</p>



<ol class="wp-block-list">
<li><strong>Test the contact point in isolation.</strong> The Test button sends a notification without touching rules or policies. If it fails, the problem is credentials or network and nothing else matters yet.</li>
<li><strong>Watch the Grafana logs during the test.</strong> On a systemd install, run <code>journalctl -u grafana-server -f</code> while you click Test. SMTP, Slack and Telegram delivery errors all surface here, usually specific enough to fix directly.</li>
<li><strong>Compare the instance&#8217;s labels against your matchers.</strong> A policy matching <code>severity = critical</code> never sees <code>severity = Critical</code>. Matchers are exact and case-sensitive. Read the firing instance&#8217;s real labels, not the ones you think the rule sets.</li>
<li><strong>Look for a policy earlier in the tree that swallowed it.</strong> Evaluation stops at the first match unless the policy continues. A broad route near the top silently absorbs everything below it.</li>
<li><strong>Check active silences and mute timings.</strong> A silence created during last month&#8217;s maintenance with a generous matcher is the classic. So is a mute timing in the wrong timezone.</li>
<li><strong>Check whether you&#8217;re inside <code>repeat_interval</code>.</strong> A red alert now does not mean a notification is due now. This looks identical to a broken integration and isn&#8217;t.</li>
<li><strong>For email, read the relay&#8217;s own logs.</strong> Grafana handing the message off is not delivery. Bounces and spam filing happen after Grafana&#8217;s involvement ends.</li>
</ol>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Configuring the email contact point and never touching <code>grafana.ini</code>.</strong> The contact point is only a recipient list. Without the SMTP block, it has nothing to send with.</li>
<li><strong>Routing by channel instead of severity.</strong> It feels tidy and it makes urgency impossible to express.</li>
<li><strong>Dropping the minus sign from a Telegram group chat ID.</strong> Group IDs are negative and the field takes the value literally.</li>
<li><strong>Sending verbose templates to Telegram.</strong> The 4096-character limit turns your biggest incident into your quietest one.</li>
<li><strong>Testing with the Test button and calling it done.</strong> Test bypasses the policy tree entirely. It proves the credential works, not that a real alert would ever reach it.</li>
<li><strong>Leaving resolved notifications on for every channel.</strong> Resolution messages in a pager channel double the volume and train people to ignore it.</li>
<li><strong>Storing webhook URLs and bot tokens in the UI only.</strong> They exist in one place, are invisible to review, and disappear with the database.</li>
</ul>



<h2 class="wp-block-heading">Best practices for Grafana alert notifications</h2>



<ul class="wp-block-list">
<li>Set a consistent <code>severity</code> label on every rule from day one. Retrofitting labels across dozens of rules is miserable work.</li>
<li>Use one contact point per urgency level, holding as many integrations as that level needs.</li>
<li>Put a dashboard or runbook link in an annotation your templates render. An alert that says what broke but not where to look costs the responder five minutes every time.</li>
<li>Write channel-specific templates: short for Telegram, richer for email, in between for Slack.</li>
<li>Provision contact points and policies from files, with secrets injected from the environment, and run a heartbeat through every channel.</li>
<li>Re-test every integration after credential rotation, Slack workspace changes or a Grafana upgrade. Those are the three events that quietly break delivery.</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Can one contact point send to email, Slack and Telegram at the same time?</h3>



<p class="wp-block-paragraph">Yes. A contact point holds multiple integrations and every one of them fires when that contact point is selected. This is the right way to build an escalation tier: one contact point per severity, with the channel mix chosen to match how urgently that severity needs attention.</p>



<h3 class="wp-block-heading">Why does the Test button work but real alerts never arrive?</h3>



<p class="wp-block-paragraph">Test sends straight to the contact point and skips the policy tree. If it works and real alerts don&#8217;t, the break is in routing: matchers that don&#8217;t match, a broader policy earlier in the tree absorbing the alert, an active silence, or a mute timing. Open the firing instance, read its real labels, and compare them character by character against your matchers.</p>



<h3 class="wp-block-heading">Do I need Grafana Cloud for any of this?</h3>



<p class="wp-block-paragraph">No. Contact points, notification policies, templates and all three integrations ship in the open source build. Grafana Cloud handles hosting and adds on-call tooling on top, which matters if you need rotations and escalation chains. For a team routing alerts to three channels, a self-hosted instance on a modest VPS does everything described here.</p>



<h3 class="wp-block-heading">Should I use a Slack webhook or a bot token?</h3>



<p class="wp-block-paragraph">Webhook for one or two channels where speed of setup matters. Bot token once you have several channels, want one revocable credential, or need to see in one place what the integration is allowed to do. Both deliver identically; the difference is entirely in credential management.</p>



<h3 class="wp-block-heading">Why did my Telegram alerts stop during a big outage?</h3>



<p class="wp-block-paragraph">Almost certainly the 4096-character limit. A large grouped notification exceeded it, truncation broke the message markup, and Telegram rejected the send outright. Shorten the template for that channel, group more narrowly, or drop the parse mode. It&#8217;s the failure mode most worth guarding against, because it only shows up when you need the channel most.</p>



<h3 class="wp-block-heading">Can I keep using the Prometheus Alertmanager I already run?</h3>



<p class="wp-block-paragraph">Yes. Grafana can send to an external Alertmanager instead of its built-in one, and the contact point selector at the top of the alerting pages lets you choose which one you&#8217;re configuring. If you already have a mature Alertmanager routing config, keeping it and pointing Grafana at it is usually less work than migrating the tree.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">A firing rule proves evaluation works. It proves nothing about whether anyone was told. Grafana alert notifications cross three or four systems after the rule turns red, and every one of them fails quietly by default.</p>



<p class="wp-block-paragraph">So build the channels carefully, route by severity rather than by tool, then run a heartbeat through every path and let something outside Grafana watch it. The alternative is finding out during the incident, which is the one time you have no attention to spare.</p>



<h2 class="wp-block-heading">Need alerting that reaches people?</h2>



<p class="wp-block-paragraph">Most of the Grafana setups I get asked to look at have good dashboards and alerting nobody trusts. Work I take on here:</p>



<ul class="wp-block-list">
<li>Fixing email delivery end to end: SMTP config, relay selection, SPF, DKIM and DMARC so alerts stop landing in spam.</li>
<li>Setting up Slack and Telegram contact points properly, including bot apps, scoped tokens and channel-appropriate templates.</li>
<li>Designing a policy tree around severity labels, with grouping and timing tuned so people keep reading the channel.</li>
<li>Converting click-built alerting into provisioned YAML under version control, with secrets kept out of Git.</li>
<li>Adding heartbeat monitoring so a broken notification path announces itself instead of waiting for an outage.</li>
<li>Auditing alert rules for missing labels, missing runbook links, and rules that fire constantly and are therefore ignored.</li>
</ul>



<p class="wp-block-paragraph">Send me your notification policy export and the output of <code>journalctl -u grafana-server</code> from a period when an alert should have gone out, and I&#8217;ll tell you which hop is dropping it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/grafana-alert-notifications-email-slack-telegram/">Grafana Alert Notifications That Actually Arrive: Email, Slack and Telegram</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Monitoring n8n Workflow Failures in Grafana: Why Your Green Dashboard Is Lying</title>
		<link>https://john-nessime.com/blog/devops/monitoring-n8n-workflow-failures-grafana/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 25 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[Cardinality]]></category>
		<category><![CDATA[Docker Compose]]></category>
		<category><![CDATA[Grafana]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[n8n]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[Prometheus]]></category>
		<category><![CDATA[PromQL]]></category>
		<category><![CDATA[Pushgateway]]></category>
		<category><![CDATA[Self Hosting]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[Webhooks]]></category>
		<category><![CDATA[Workflow Automation]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=276</guid>

					<description><![CDATA[<p>n8n's Prometheus endpoint tells you the process is alive and busy. It does not tell you a workflow stopped producing correct output. This post walks through the three failure families that slip past a default n8n dashboard, the PromQL that catches each one, and how to close the gap with a second source of truth from the executions API.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/monitoring-n8n-workflow-failures-grafana/">Monitoring n8n Workflow Failures in Grafana: Why Your Green Dashboard Is Lying</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone asks why the nightly invoice sync hasn&#8217;t posted anything since Tuesday. You open Grafana. The n8n scrape target is UP, the instance has been running for weeks without a restart, every panel is green. Nothing on the dashboard says a thing is wrong, and yet four days of records are missing.</p>



<p class="wp-block-paragraph">That gap is the entire problem. Monitoring n8n workflow failures in Grafana is not hard because the tooling is missing. It&#8217;s hard because the built-in metrics endpoint answers a question you didn&#8217;t ask. It tells you the process is alive and busy. It does not tell you that a workflow stopped producing correct output.</p>



<p class="wp-block-paragraph">This post covers what n8n&#8217;s Prometheus endpoint actually exposes, the three families of failure it handles badly, and the specific PromQL and alert rules that close each gap. There&#8217;s also a troubleshooting section for the case where <code>/metrics</code> returns nothing at all, which is the first wall most people hit.</p>



<h2 class="wp-block-heading">What the n8n metrics endpoint actually gives you</h2>



<p class="wp-block-paragraph">n8n uses the <code>prom-client</code> library and serves metrics on the same port as the editor, at <code>/metrics</code>. The endpoint is off by default, and almost everything interesting inside it is off by default too, behind its own environment variable.</p>



<pre class="wp-block-code"><code>N8N_METRICS=true
N8N_METRICS_INCLUDE_MESSAGE_EVENT_BUS_METRICS=true
N8N_METRICS_INCLUDE_WORKFLOW_ID_LABEL=true
N8N_METRICS_INCLUDE_NODE_TYPE_LABEL=true
N8N_METRICS_INCLUDE_QUEUE_METRICS=true</code></pre>



<p class="wp-block-paragraph">The first line opens the endpoint. The second turns on the workflow and node event counters, which is the part people assume is included and is not. The third attaches a <code>workflow_id</code> label to workflow metrics, without which you get a single instance-wide number and no way to tell which automation broke.</p>



<p class="wp-block-paragraph">Those labels are opt-in for a reason worth understanding rather than working around. Every distinct label value creates a separate time series. Turn on <code>workflow_id</code> and you get one series per workflow per counter; add <code>node_type</code> and node counters fan out across every integration you use. On a few dozen workflows that&#8217;s fine. On several hundred with per-node labels, you&#8217;re signing up for a cardinality problem that surfaces later as slow queries and a Prometheus process that keeps getting OOM-killed.</p>



<p class="wp-block-paragraph">With the event bus metrics enabled, the counters you get include <code>n8n_workflow_started_total</code>, <code>n8n_workflow_success_total</code>, <code>n8n_node_started_total</code> and <code>n8n_node_finished_total</code>. Queue mode adds <code>n8n_scaling_mode_queue_jobs_active</code>, <code>_completed</code>, <code>_failed</code> and <code>_waiting</code>, gathered from Bull and exposed on the main instance only.</p>



<p class="wp-block-paragraph">Prometheus side, the scrape job is unremarkable:</p>



<pre class="wp-block-code"><code>scrape_configs:
  - job_name: n8n
    static_configs:
      - targets:
          - n8n-main:5678
          - n8n-worker-1:5678
    metrics_path: /metrics</code></pre>



<p class="wp-block-paragraph">Note both main and workers. Each process exposes its own endpoint and its own counters. Scrape only the main and you lose everything the workers did, which in queue mode is nearly all the real execution work.</p>



<p class="wp-block-paragraph">One thing the n8n docs are emphatic about, and they&#8217;re right: don&#8217;t put <code>/metrics</code> on the public internet. It leaks workflow IDs, execution volumes and instance topology to anyone who asks. Keep the scrape path on a private network segment, a WireGuard tunnel, or your provider&#8217;s internal networking. If you&#8217;re running the whole stack on a single VPS from somewhere like Contabo or InterServer, binding n8n to the Docker network and scraping it by container name is enough. Publishing port 5678 to the host and firewalling it afterwards is not, because you will forget.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: the execution that never happened</h2>



<p class="wp-block-paragraph">This is the one that costs the most and shows up the least. A schedule trigger gets deactivated during a deploy. A credential expires and the workflow is switched off rather than fixed. An upstream service stops calling a webhook. Nothing errors. Nothing increments. The workflow simply stops existing as far as your metrics are concerned.</p>



<p class="wp-block-paragraph">Every error-rate alert you write is blind to this, because an error rate needs a denominator. Zero executions divided by zero executions is not an alert, it&#8217;s a NaN.</p>



<p class="wp-block-paragraph">The instinct is to write this:</p>



<pre class="wp-block-code"><code>sum by (workflow_id) (increase(n8n_workflow_started_total[6h])) == 0</code></pre>



<p class="wp-block-paragraph">It looks correct and it does not work. When a workflow stops running, its series stops being exported. Prometheus marks it stale after a few missed scrapes and instant queries stop returning it. There is no series left for <code>== 0</code> to match against. The alert never fires, because the thing you wanted to alert on is exactly the thing that removed the data.</p>



<p class="wp-block-paragraph">What you want is a function that returns something when a series is absent:</p>



<pre class="wp-block-code"><code>absent_over_time(
  n8n_workflow_started_total{workflow_id="AbC123XyZ"}[6h]
)</code></pre>



<p class="wp-block-paragraph"><code>absent_over_time</code> returns 1 if the matched series produced no samples in the window and nothing at all if it did. That inversion is the point. The alert fires on silence.</p>



<p class="wp-block-paragraph">The cost is that <code>absent_over_time</code> needs an exact matcher, so you can&#8217;t write it once for everything. That&#8217;s more acceptable than it sounds. You don&#8217;t need a heartbeat on all two hundred workflows, you need one on the six a human would notice within an hour. Write those six by hand and treat the list as part of the runbook.</p>



<p class="wp-block-paragraph">One Grafana-specific trap. When a query returns nothing, Grafana&#8217;s alert rule enters a <strong>No Data</strong> state, which is configurable separately from the alert condition and defaults to firing in most versions. Set it deliberately. For an <code>absent_over_time</code> rule, No Data means the whole scrape target is gone, which is a different incident from one workflow going quiet, and you probably want it routed differently.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family two: the failure you can see but can&#8217;t attribute</h2>



<p class="wp-block-paragraph">You have the counters. Something is failing. Which workflow?</p>



<p class="wp-block-paragraph">Without <code>N8N_METRICS_INCLUDE_WORKFLOW_ID_LABEL</code>, the answer is: no idea. The counters are instance-wide totals. You know the number moved and that&#8217;s it. That&#8217;s a fine signal for capacity planning and useless during an incident.</p>



<p class="wp-block-paragraph">Turn the label on and you get IDs, which are opaque strings like <code>AbC123XyZ</code>. A dashboard full of those is technically attributable and practically unreadable at 3am. Newer n8n versions solve this with a gauge that maps IDs to names, enabled with <code>N8N_METRICS_INCLUDE_WORKFLOW_INFO</code>:</p>



<pre class="wp-block-code"><code>n8n_workflow_info{workflow_id="VaQPuPmx9tPpo6BX",workflow_name="Invoice sync"} 1</code></pre>



<p class="wp-block-paragraph">It&#8217;s a constant-1 gauge that exists only to be joined against. The pattern is a standard Prometheus vector match:</p>



<pre class="wp-block-code"><code>sum by (workflow_name) (
  rate(n8n_workflow_started_total[5m])
  * on(workflow_id) group_left(workflow_name)
  n8n_workflow_info
)</code></pre>



<p class="wp-block-paragraph">The multiplication is arithmetically a no-op because the gauge is always 1. Its only job is to carry the <code>workflow_name</code> label across with <code>group_left</code>. Do this once in a Grafana variable or a recording rule rather than in every panel, or you&#8217;ll be copy-pasting the join for the rest of your life.</p>



<p class="wp-block-paragraph">On an older build without the info gauge, the fallback is a hand-maintained relabel config or a Grafana value mapping. Both drift, and the relabel version drifts silently.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family three: the failure that counts as a success</h2>



<p class="wp-block-paragraph">This is the invisible one, and it&#8217;s why a green n8n dashboard is not evidence of anything.</p>



<p class="wp-block-paragraph">n8n gives you several deliberate ways to make a node&#8217;s failure not be an execution failure. They&#8217;re all reasonable features. Together they mean the success counter and &#8220;the automation worked&#8221; are different statements.</p>



<ul class="wp-block-list">
<li><strong>Continue on error.</strong> A node set to continue passes its error downstream instead of stopping the run. The execution finishes. It finishes as a success.</li>

<li><strong>Error output branches.</strong> A node with a separate error output routes failures to a different path. That path might log and stop. The execution still completes normally.</li>

<li><strong>Never Error on HTTP responses.</strong> The HTTP Request node can be configured to treat any status code as a valid response. A run of 500s from an upstream API becomes a workflow that processed zero useful records, successfully.</li>

<li><strong>Empty result sets.</strong> A query that returns nothing is not an error. A sync that syncs nothing, every night, looks exactly like a sync that has nothing to do.</li>
</ul>



<p class="wp-block-paragraph">There are also user reports on the n8n community forum of started and success counters matching even when a node failed outright, in queue mode. I have not reproduced that and I&#8217;d treat it as unconfirmed rather than a known behavior. But it&#8217;s the right instinct: don&#8217;t assume the success counter is a strict complement of failure. Verify it on your own instance before you build alerting on the assumption.</p>



<p class="wp-block-paragraph">For webhook-triggered workflows, newer n8n versions give you a real answer here. The <code>n8n_webhook_request_duration_seconds</code> histogram carries a <code>status_code</code> label, so you can alert on what the caller actually received rather than on what n8n thought about the run:</p>



<pre class="wp-block-code"><code>sum by (workflow_id) (
  rate(n8n_webhook_request_duration_seconds_count{status_code!~"2.."}[5m])
)</code></pre>



<p class="wp-block-paragraph">Worth flagging: the n8n documentation shows this example with <code>status_code!="2.."</code>, using the equality operator. In PromQL, <code>!=</code> compares against a literal string, so that matcher excludes only a status code whose literal value is the three characters <code>2..</code>, which never occurs. It matches everything, including your 200s. Use <code>!~</code> for the regex form. Test any label matcher you copy from documentation by running it in Prometheus and checking the series count against what you expect.</p>



<h3 class="wp-block-heading">Getting a second source of truth</h3>



<p class="wp-block-paragraph">For everything that isn&#8217;t a webhook, the metrics endpoint alone won&#8217;t close this gap. You need a second signal that reads execution outcomes from n8n&#8217;s own store.</p>



<p class="wp-block-paragraph">The public REST API exposes exactly that. Executions can be filtered by status, and the failed ones are one request away:</p>



<pre class="wp-block-code"><code>curl -s -H "X-N8N-API-KEY: $N8N_API_KEY" 
  "https://n8n.example.com/api/v1/executions?status=error&amp;limit=50"</code></pre>



<p class="wp-block-paragraph">Two ways to get that into Grafana. Write a small exporter that polls the API on a timer and serves its own <code>/metrics</code>, or run a scheduled job that pushes counts to a Prometheus Pushgateway. The exporter is more work and behaves correctly. Pushgateway is faster to build and carries the usual caveat: pushed metrics persist until deleted, so a job that stops running leaves a stale value sitting there looking healthy. That reintroduces failure family one through the back door.</p>



<p class="wp-block-paragraph">Two caveats on the API. There&#8217;s an open report that the <code>status=crashed</code> list filter returns executions whose stored status is something else entirely when re-fetched by ID, which would make crash alerting built on the list endpoint unreliable. Check the behavior on your version before depending on it. And n8n&#8217;s log streaming feature, which forwards execution events to syslog or a webhook in real time, is an Enterprise capability. If you&#8217;re on the community edition, the API poll is your path.</p>



<p class="wp-block-paragraph">The third option, and often the best one for a small setup, is an Error Trigger workflow. n8n lets you designate a workflow that runs whenever another workflow fails. Point it at Pushgateway, at Loki, or at a Grafana webhook contact point. It&#8217;s native, it needs no API key, and it gives you the error message rather than just a count. The trade-off is that it only fires on real execution failures, so it inherits every blind spot in this section.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The Grafana panels that earn their space</h2>



<p class="wp-block-paragraph">Four panels cover most of what you need. Everything past that is usually decoration.</p>



<ol class="wp-block-list">
<li><strong>Executions per workflow, by name.</strong> A time series of <code>rate(n8n_workflow_started_total[5m])</code> joined to <code>n8n_workflow_info</code>. Its real job is showing you the lines that flatten to zero.</li>

<li><strong>Success ratio.</strong> <code>rate(n8n_workflow_success_total[15m]) / rate(n8n_workflow_started_total[15m])</code>. Anything below 1 is worth a look. Anything above 1 means your window is catching a restart and the counters reset.</li>

<li><strong>Queue depth and failed jobs.</strong> <code>n8n_scaling_mode_queue_jobs_waiting</code> next to <code>increase(n8n_scaling_mode_queue_jobs_failed[1h])</code>. A waiting count that climbs and never drains means workers are dead or starved, which looks identical to &#8220;no failures&#8221; on every other panel.</li>

<li><strong>Webhook non-2xx rate.</strong> The query from the previous section. This is the only panel that reports what the outside world experienced.</li>
</ol>



<p class="wp-block-paragraph">Skip the node-level breakdown unless you have a specific reason. It is the most expensive thing you can enable in cardinality terms and it answers a question you can answer faster by opening the execution in the n8n UI.</p>



<h3 class="wp-block-heading">Alert rules worth paging on</h3>



<ol class="wp-block-list">
<li><strong>Named workflow silent.</strong> <code>absent_over_time</code> on each business-critical workflow, window set to roughly twice its expected interval. This is the rule that catches the invoice sync.</li>

<li><strong>Success ratio below threshold.</strong> Sustained for 15 minutes, not instantaneous. A single failed run is not a page.</li>

<li><strong>Queue waiting count rising.</strong> Compare the current value to fifteen minutes ago rather than to a fixed number, so the threshold survives traffic growth.</li>

<li><strong>Webhook error rate.</strong> Per workflow, so a single noisy endpoint doesn&#8217;t mask the rest.</li>

<li><strong>Scrape target down.</strong> The <code>up</code> metric. Obvious, and routinely forgotten, and it invalidates every other rule on this list.</li>
</ol>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting: the endpoint returns nothing useful</h2>



<ul class="wp-block-list">
<li><strong>&#8220;Cannot GET /metrics&#8221;.</strong> <code>N8N_METRICS</code> isn&#8217;t set on the process you&#8217;re hitting, or it&#8217;s set as a string that n8n doesn&#8217;t read as true. In Docker Compose, quote it as <code>"true"</code> and confirm with <code>docker compose exec n8n env | grep N8N_METRICS</code>.</li>

<li><strong>Endpoint responds but has no n8n_ counters.</strong> You have the default Node.js process metrics and nothing else. Set <code>N8N_METRICS_INCLUDE_MESSAGE_EVENT_BUS_METRICS=true</code> and restart.</li>

<li><strong>Counters exist but no workflow_id label.</strong> <code>N8N_METRICS_INCLUDE_WORKFLOW_ID_LABEL</code> is off. Restart is required; it&#8217;s read at startup.</li>

<li><strong>Queue metrics are zero or missing.</strong> They&#8217;re gathered from Bull and exposed on the main instance, not the workers. Querying the worker endpoint gives you nothing. Confirm you&#8217;re looking at the right target.</li>

<li><strong>Counters look wrong after a deploy.</strong> They reset to zero when the process restarts. Always wrap them in <code>rate()</code> or <code>increase()</code>, which handle counter resets. Raw counter values on a dashboard will lie to you every time you deploy.</li>

<li><strong>Prometheus can&#8217;t reach the target.</strong> Usually a Docker network mismatch. Scrape by container name on the shared network rather than by published host port, and check the target page in Prometheus for the actual error rather than guessing.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating the success counter as proof the automation worked. It proves the run finished.</li>

<li>Building only error-rate alerts, so the workflows that stop entirely go unmonitored.</li>

<li>Enabling every <code>N8N_METRICS_INCLUDE_*</code> variable on day one and discovering the cardinality bill a month later.</li>

<li>Scraping only the main instance in queue mode and losing all worker-side execution data.</li>

<li>Copying label matchers out of documentation without running them and checking the returned series count.</li>

<li>Exposing <code>/metrics</code> publicly because it&#8217;s &#8220;just numbers&#8221;.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Decide which workflows are business-critical and write a silence alert for each one by name. Six good heartbeats beat two hundred generic panels.</li>

<li>Run a second source of truth. Metrics tell you about volume, the executions API tells you about outcomes. You need both.</li>

<li>Push the ID-to-name join into a recording rule so panels stay readable and cheap.</li>

<li>Break a staging workflow on purpose and confirm the alert fires. An untested alert is a decoration.</li>

<li>If you&#8217;d rather not run Prometheus and Grafana yourself, Grafana Cloud will scrape a remote endpoint and gives you alerting without a control plane to maintain. Weigh that against your metrics leaving your network.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Does n8n expose a failed-executions counter?</h3>



<p class="wp-block-paragraph">Not a reliable per-workflow one. You get started and success counters from the event bus, and a failed-jobs counter for queue mode. Deriving failures by subtracting success from started works only if you&#8217;ve verified on your own instance that a failing run genuinely doesn&#8217;t increment success, which is exactly the assumption that breaks when nodes are set to continue on error.</p>



<h3 class="wp-block-heading">Can I monitor n8n Cloud this way?</h3>



<p class="wp-block-paragraph">No. The <code>/metrics</code> endpoint isn&#8217;t available on n8n Cloud. On Cloud you&#8217;re limited to the public API and the built-in insights views, so the executions-API approach in this post is the only one that transfers.</p>



<h3 class="wp-block-heading">Why does my workflow error rate look like zero when I know things are failing?</h3>



<p class="wp-block-paragraph">Almost always one of three things: the event bus metrics aren&#8217;t enabled so you&#8217;re only seeing Node.js process metrics, the failures are being absorbed by continue-on-error settings and recorded as successes, or you&#8217;re scraping the main instance while the work happens on workers.</p>



<h3 class="wp-block-heading">How do I alert when a scheduled workflow stops running entirely?</h3>



<p class="wp-block-paragraph">Use <code>absent_over_time</code> against that workflow&#8217;s started counter with an exact <code>workflow_id</code> matcher, and set the window to about twice the expected interval. A threshold comparison won&#8217;t work, because a workflow that stops running stops producing the series you&#8217;d compare against.</p>



<h3 class="wp-block-heading">Is enabling the workflow_id label safe on a large instance?</h3>



<p class="wp-block-paragraph">It multiplies your n8n series count by the number of workflows. On a few dozen it&#8217;s negligible. Past a few hundred, combined with node-type labels, watch your Prometheus memory and head series count before and after. Enable the workflow label, leave the node label off, and only turn the node one on when you have a specific question it answers.</p>



<h3 class="wp-block-heading">Should I use OpenTelemetry instead of Prometheus for n8n?</h3>



<p class="wp-block-paragraph">They solve different problems. Metrics answer &#8220;how many, how often, how fast&#8221; cheaply and forever. Traces answer &#8220;where did this run spend its time and what did it call&#8221;, which is better for debugging a slow multi-service workflow. Tracing n8n usually means a custom image with the OpenTelemetry SDK loaded before startup, so it&#8217;s a project rather than an environment variable. Start with metrics; add tracing when you have a latency question metrics can&#8217;t answer.</p>



<h3 class="wp-block-heading">Where do the n8n counters go when the container restarts?</h3>



<p class="wp-block-paragraph">They reset to zero. They&#8217;re in-process counters with no persistence. Prometheus handles this correctly as long as your queries use <code>rate()</code> or <code>increase()</code>. Any panel showing a raw counter value will appear to lose all its history on every deploy.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">Monitoring n8n workflow failures in Grafana works, but only once you stop asking the metrics endpoint a question it can&#8217;t answer. It reports activity. Activity and correctness are different things, and the space between them is where the expensive outages live.</p>



<p class="wp-block-paragraph">So build for absence first. A silence alert on the handful of workflows a human would miss will catch more real incidents than any error-rate panel, because errors are loud and stopped schedules are not. Then add attribution with the workflow label, then close the false-success gap with the executions API or an Error Trigger workflow. In that order, because each one is cheap and the value drops off fast after the third.</p>



<h2 class="wp-block-heading">Need help wiring this up?</h2>



<p class="wp-block-paragraph">I build and maintain observability stacks for self-hosted automation platforms, and n8n is a common one. If any of the above sounds like a week you&#8217;d rather not spend, here&#8217;s what I can take off your plate:</p>



<ul class="wp-block-list">
<li>Enabling and securing the n8n metrics endpoint on a private network path, including queue-mode setups with multiple workers</li>

<li>Building Prometheus scrape configs, recording rules and the ID-to-name join so your Grafana panels are readable rather than a wall of hashes</li>

<li>Writing silence and error-rate alert rules that survive counter resets, Grafana No Data states and traffic growth</li>

<li>Adding a second source of truth from the n8n executions API, either as a small exporter or an Error Trigger workflow feeding Loki or Pushgateway</li>

<li>Auditing an existing setup for cardinality problems, stale Pushgateway series and alerts that quietly stopped being able to fire</li>

<li>Running a failure drill against staging so you know which alerts actually work before you need them</li>
</ul>



<p class="wp-block-paragraph">Send me your Compose file, your <code>prometheus.yml</code>, or a screenshot of the panel that looks fine while things are on fire, and I&#8217;ll tell you what&#8217;s missing.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/monitoring-n8n-workflow-failures-grafana/">Monitoring n8n Workflow Failures in Grafana: Why Your Green Dashboard Is Lying</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Streaming Shopify Events into AWS Without Losing Orders</title>
		<link>https://john-nessime.com/blog/devops/streaming-shopify-events-into-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 16 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon SQS]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Dead Letter Queue]]></category>
		<category><![CDATA[DynamoDB]]></category>
		<category><![CDATA[Ecommerce Analytics]]></category>
		<category><![CDATA[Event-Driven Architecture]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Partner Event Source]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Reliability Engineering]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[Terraform]]></category>
		<category><![CDATA[Webhooks]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=221</guid>

					<description><![CDATA[<p>Wiring Shopify webhooks into Amazon EventBridge takes an afternoon. Keeping every order is the hard part. A walk through the five failure families that actually bite when streaming Shopify events into AWS: the partner source that silently drops everything, duplicate and out-of-order deliveries, rule patterns that match nothing, targets that fail without a dead-letter queue, and the 64 KB metering rule that quietly inflates the bill.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/streaming-shopify-events-into-aws/">Streaming Shopify Events into AWS Without Losing Orders</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The partner event source in the EventBridge console said <code>Pending</code>. It had said <code>Pending</code> for six days.</p>



<p class="wp-block-paragraph">Nobody noticed, because nothing errored. No 5xx in a log. No failed delivery in Shopify&#8217;s dashboard. No alarm. Shopify had been publishing order events the entire time, and AWS had been throwing every single one of them on the floor.</p>



<p class="wp-block-paragraph">That behaviour is documented, in one short note in the AWS docs: events published to a partner event source that has not been associated with an event bus are dropped immediately and are not persisted at rest. There is no retry for that. There is no buffer. The events are gone, and the only way to get the data back is to go ask the Shopify Admin API for it after the fact.</p>



<p class="wp-block-paragraph">That is the shape of most of the pain in this integration. Streaming Shopify events into AWS is easy to stand up and easy to get quietly wrong, and every one of the quiet failures looks identical from the outside: everything is green, and some of your data isn&#8217;t there.</p>



<p class="wp-block-paragraph">This post walks the five failure families that actually cost you records, plus the reconciliation layer that most teams only build after the first incident. It assumes you can read a rule pattern and an IAM policy. It does not assume you have shipped this before.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">What the pipe actually looks like</h2>



<p class="wp-block-paragraph">Four moving parts, and only two of them live in your account.</p>



<ol class="wp-block-list"><li>A Shopify app holds the webhook subscriptions. Each subscription has a topic and a delivery method. For this path the delivery method is EventBridge and the address is an ARN, not a URL.</li><li>Shopify creates a <strong>partner event source</strong> inside your AWS account, in the region you nominated.</li><li>You associate that source with a <strong>partner event bus</strong>. This is the step everyone forgets.</li><li>Rules on that bus match events and push them at targets: Lambda, SQS, Step Functions, Firehose, whatever fits.</li></ol>



<p class="wp-block-paragraph">The ARN trips people up more than anything else in the setup. Shopify wants the <em>event source</em> ARN, not the event bus ARN. They look similar and only one of them works:</p>



<pre class="wp-block-code"><code># Correct - the event source ARN. Note the empty account field.
arn:aws:events:eu-west-1::event-source/aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;

# Wrong - this is the bus, and Shopify will reject it
arn:aws:events:eu-west-1:123456789012:event-bus/aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;</code></pre>



<p class="wp-block-paragraph">Associating the source is a single call, and both the name and the source name are the same string:</p>



<pre class="wp-block-code"><code># Create the partner event bus that accepts the source
aws events create-event-bus 
  --name "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;" 
  --event-source-name "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;" 
  --region eu-west-1

# Confirm it flipped from PENDING to ACTIVE
aws events describe-event-source 
  --name "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;" 
  --region eu-west-1</code></pre>



<p class="wp-block-paragraph">The same architecture applies if you are not on Shopify. BigCommerce and commercetools both publish to EventBridge as partner sources, and the failure families below are identical because they come from EventBridge&#8217;s semantics, not the store&#8217;s.</p>



<h2 class="wp-block-heading">Failure family one: events that never existed</h2>



<p class="wp-block-paragraph">This is the one from the opening, and it is the most expensive because it is completely silent on both sides.</p>



<p class="wp-block-paragraph">Shopify considers the delivery successful. It handed the event to the partner source, which is its contract. AWS considers nothing to have happened, because an unassociated source has no bus to write to, and EventBridge does not persist events at rest before a bus exists. Your CloudWatch metrics show nothing, because metrics are emitted per bus and per rule, and you have neither.</p>



<p class="wp-block-paragraph">The same class of hole opens up in two other ways:</p>



<ul class="wp-block-list"><li><strong>Region mismatch.</strong> The source is created in the region you gave Shopify. Your bus, your rules, your targets and your dead-letter queues all have to be in that region. A rule in the right account but the wrong region matches nothing, forever, without complaint.</li><li><strong>Environment drift.</strong> A staging store pointed at a production source, or a source created against an account ID that belonged to an old sandbox. Nothing errors. Events just land somewhere you are not looking.</li></ul>



<p class="wp-block-paragraph">The fix is boring and it works: treat the source state as a monitored asset. A scheduled job that calls <code>describe-event-source</code> and alarms if <code>State</code> is anything other than <code>ACTIVE</code> costs you twenty minutes and covers the entire failure family. Put it next to your other synthetic checks, not inside the pipeline it is watching.</p>



<p class="wp-block-paragraph">The second half of that check is a heartbeat on volume. If a bus that normally sees a few thousand events a day sees zero for an hour, that is an incident even when every component reports healthy. Alarm on <code>MatchedEvents</code> hitting zero, not just on errors.</p>



<h2 class="wp-block-heading">Failure family two: events that arrive twice, or backwards</h2>



<p class="wp-block-paragraph">EventBridge is at-least-once. Shopify&#8217;s webhooks are at-least-once. Neither one promises ordering. Put those together and you get two distinct bugs that people usually try to fix with one patch.</p>



<p class="wp-block-paragraph">The duplicate is the obvious one. The same <code>orders/create</code> arrives twice, and if your handler posts to a fulfilment provider or sends a customer email, you have just done it twice. The dedupe key is sitting in the envelope: Shopify puts <code>X-Shopify-Webhook-Id</code> into <code>detail.metadata</code>, and it identifies the delivery. Write it into DynamoDB with a conditional put and a TTL of a few days, and drop the event if the write fails.</p>



<p class="wp-block-paragraph">The out-of-order case is the one that costs you money quietly. An <code>orders/updated</code> carrying a cancelled status arrives before the <code>orders/updated</code> carrying the address change, and your database ends up holding the older state because it was written last. Nothing failed. The row is just wrong, and it will stay wrong until someone complains.</p>



<p class="wp-block-paragraph">The envelope carries what you need for this too. <code>detail.metadata</code> includes <code>X-Shopify-Triggered-At</code>, and the resource in <code>detail.payload</code> carries its own <code>updated_at</code>. Compare before you write, and refuse to apply an update whose timestamp is older than the one already stored.</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow"><p>The dedupe key stops you from doing the work twice. The version check stops you from doing the work backwards. They solve different problems and you need both.</p></blockquote>



<p class="wp-block-paragraph">One thing you can skip on this path: HMAC verification. On the HTTPS delivery method you must verify the signature, because anyone can POST to your endpoint. On the EventBridge path, only the partner account behind the event source is permitted to publish to that bus, and the AWS docs are explicit that adding your own resource policy to a partner bus is rejected. The signature header still rides along in the metadata, but the trust boundary is enforced by AWS rather than by your code.</p>



<h2 class="wp-block-heading">Failure family three: rules that match nothing</h2>



<p class="wp-block-paragraph">Every Shopify event, regardless of topic, arrives with the same <code>detail-type</code>. That single fact invalidates the routing instinct most people bring from AWS service events.</p>



<pre class="wp-block-code"><code>{
  "version": "0",
  "id": "1b8e2e75-b771-e964-f0e6-fbca6a21dad8",
  "detail-type": "shopifyWebhook",
  "source": "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;",
  "account": "123456789012",
  "time": "2022-07-02T12:47:58Z",
  "region": "eu-west-1",
  "resources": [],
  "detail": {
    "payload": {
      "id": 1234567890,
      "title": "Columbia Las Hermosas"
    },
    "metadata": {
      "Content-Type": "application/json",
      "X-Shopify-Topic": "products/update",
      "X-Shopify-Shop-Domain": "example.myshopify.com",
      "X-Shopify-Hmac-SHA256": "...",
      "X-Shopify-Webhook-Id": "...",
      "X-Shopify-API-Version": "...",
      "X-Shopify-Triggered-At": "2022-07-02T12:47:57.989779121Z"
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Two things to take from that envelope. The resource body is nested under <code>detail.payload</code>, not at the top of <code>detail</code>, so a pattern copied from an HTTPS handler will match nothing. And the topic lives in <code>detail.metadata</code>, which is where all your routing has to happen.</p>



<pre class="wp-block-code"><code>// Exact topic match
{
  "detail-type": ["shopifyWebhook"],
  "detail": {
    "metadata": {
      "X-Shopify-Topic": ["orders/create"]
    }
  }
}

// Every orders topic, one rule
{
  "detail-type": ["shopifyWebhook"],
  "detail": {
    "metadata": {
      "X-Shopify-Topic": [{ "prefix": "orders/" }]
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Do not deploy a pattern you have not tested against a real envelope. <code>test-event-pattern</code> answers in a second and saves an afternoon:</p>



<pre class="wp-block-code"><code>aws events test-event-pattern 
  --event-pattern file://pattern.json 
  --event file://sample-event.json</code></pre>



<h3 class="wp-block-heading">One rule or thirty?</h3>



<p class="wp-block-paragraph">There is a real argument for a single catch-all rule that pushes everything into one queue and lets your consumer branch on the topic. It is less infrastructure, it deploys faster, and adding a topic does not require a Terraform run.</p>



<p class="wp-block-paragraph">What you give up is per-topic visibility. <code>MatchedEvents</code>, <code>FailedInvocations</code> and the dead-letter queue are all scoped to the rule. Collapse thirty topics into one rule and you can no longer tell that inventory events stopped three days ago, because the aggregate number still looks fine.</p>



<p class="wp-block-paragraph">The split I reach for first: a dedicated rule for each topic that touches money or fulfilment, and one catch-all for everything else. You get precise alarms where the cost of being wrong is high and low overhead everywhere else.</p>



<h2 class="wp-block-heading">Failure family four: events that arrive and die at the target</h2>



<p class="wp-block-paragraph">By default EventBridge keeps retrying a failed target invocation for up to a day, with exponential backoff and jitter. That is generous, and it is also the reason people assume they do not need a dead-letter queue. They do, for two reasons.</p>



<p class="wp-block-paragraph">First, a whole class of errors gets <em>no</em> retries at all. Missing permissions on the target, a target that no longer exists, an address that will not resolve. EventBridge does not retry those, because retrying cannot help. It sends them straight to the DLQ if one is configured, and drops them if one is not.</p>



<p class="wp-block-paragraph">Second, a day of retries is not much when the failure is a bad deploy discovered on a Friday evening.</p>



<pre class="wp-block-code"><code>aws events put-targets 
  --rule shopify-orders-create 
  --event-bus-name "aws.partner/shopify.com/&lt;id&gt;/&lt;source-name&gt;" 
  --targets '[{
    "Id": "order-processor",
    "Arn": "arn:aws:lambda:eu-west-1:123456789012:function:order-processor",
    "RetryPolicy": {
      "MaximumRetryAttempts": 20,
      "MaximumEventAgeInSeconds": 3600
    },
    "DeadLetterConfig": {
      "Arn": "arn:aws:sqs:eu-west-1:123456789012:shopify-orders-dlq"
    }
  }]'</code></pre>



<p class="wp-block-paragraph">Lowering the retry window is deliberate here. Twenty-four hours of retries against a genuinely broken consumer buys you nothing and hides the problem; a shorter window pushes failures into the DLQ where they are visible and countable.</p>



<h3 class="wp-block-heading">The DLQ permission trap</h3>



<p class="wp-block-paragraph">This one catches almost everyone who manages infrastructure as code. Configure a DLQ through the console and AWS attaches the queue policy for you. Configure it through <code>PutTargets</code> — which is what Terraform, CloudFormation and the CLI all do — and you must attach it yourself. Miss it, and you have a dead-letter queue that cannot receive dead letters.</p>



<pre class="wp-block-code"><code>{
  "Sid": "Dead-letter queue permissions",
  "Effect": "Allow",
  "Principal": { "Service": "events.amazonaws.com" },
  "Action": "sqs:SendMessage",
  "Resource": "arn:aws:sqs:eu-west-1:123456789012:shopify-orders-dlq",
  "Condition": {
    "ArnEquals": {
      "aws:SourceArn": "arn:aws:events:eu-west-1:123456789012:rule/shopify-orders-create"
    }
  }
}</code></pre>



<p class="wp-block-paragraph">The metric that catches this is <code>InvocationsFailedToBeSentToDlq</code>. If it is ever non-zero, your safety net has a hole in it and events are being lost at the exact moment you were counting on it. Alarm on it at a threshold of one. It only reports when it is non-zero, so it costs nothing the rest of the time.</p>



<p class="wp-block-paragraph">Two more constraints worth knowing before you design around a DLQ: it must be a standard SQS queue, not FIFO, and it must live in the same region as the rule. Each message carries the error code, the exhausted retry condition, the retry count and both ARNs as message attributes, which is usually enough to triage without opening the payload.</p>



<h2 class="wp-block-heading">Failure family five: the bill</h2>



<p class="wp-block-paragraph">EventBridge does not meter one event as one event. It meters in 64 KB chunks, so an event larger than that bills as multiple events. Rates change and vary by region, so check the current pricing page rather than trusting any number you read in a blog post, but the mechanism is stable and it is what determines your bill.</p>



<p class="wp-block-paragraph">This matters more for commerce than for most event sources. A product update is small. An order with thirty line items, per-item discount allocations, tax lines, shipping lines, note attributes and a stack of metafields is not. Wholesale and subscription stores routinely produce order payloads that cross the chunk boundary, and the same order updated eight times through its lifecycle multiplies that.</p>



<p class="wp-block-paragraph">Three levers, roughly in order of how much they return:</p>



<ul class="wp-block-list"><li><strong>Trim at the subscription.</strong> Shopify&#8217;s webhook subscription API lets you restrict which fields are included in the payload and which metafield namespaces come along. Fields you never read cost you at ingestion, at archive and again at replay. This is the only lever that stops paying for the data before it enters AWS.</li><li><strong>Subscribe to fewer topics.</strong> Broad topics like <code>orders/updated</code> fire on changes you do not care about. If you only act on fulfilment state, subscribe to the fulfilment topics instead of filtering a firehose after you have paid for it.</li><li><strong>Archive selectively, and set retention.</strong> Archives bill for processing, for storage and again for replay. An archive with no retention period grows forever. Archive the topics you would genuinely replay and let the rest go.</li></ul>



<p class="wp-block-paragraph">One structural limit to design around: EventBridge caps the total size of a single event. A payload that exceeds it does not get truncated in a helpful way — the publish fails. Trimming at the subscription protects you here as well as on cost.</p>



<h2 class="wp-block-heading">The layer nobody builds until they need it</h2>



<p class="wp-block-paragraph">Archive and replay is genuinely useful, and it is also routinely misunderstood. Replay re-delivers events that <em>reached the bus</em>. It does nothing at all for the failure family at the top of this post, where the events never reached the bus in the first place. Replay fixes bugs in your consumer. It does not fix gaps in your ingestion.</p>



<p class="wp-block-paragraph">For that you need reconciliation: a scheduled job that queries the Shopify Admin API for resources changed since a stored watermark and compares them against what you hold. It is unglamorous, it is the thing that catches the outage you did not know about, and it is worth building before you need it rather than during the incident.</p>



<ul class="wp-block-list"><li>Run it hourly for orders and fulfilments, daily for products and customers. The cadence should track how expensive being wrong is, not how much data there is.</li><li>Store a watermark per topic and advance it only after a successful full pass. A partial pass that advances the watermark creates the exact gap you built the job to find.</li><li>Compare counts first, records second. A count mismatch is cheap to compute and tells you whether to bother with the expensive comparison.</li><li>Emit the drift as a metric, not just a log line. &#8220;Orders in Shopify but not in our store, last hour&#8221; is a graph worth putting on a dashboard, and it should normally read zero.</li></ul>



<p class="wp-block-paragraph">The reconciliation worker does not need to live in Lambda. It is a long, paginated, rate-limited crawl, which is an awkward fit for a function timeout and a comfortable fit for a small VPS you already run. If you have a box at InterServer or Hetzner sitting there for other jobs, a cron entry and a script is a perfectly respectable answer.</p>



<h2 class="wp-block-heading">Troubleshooting by symptom</h2>



<p class="wp-block-paragraph">Work these in order. Each one is cheap and rules out a whole branch.</p>



<h3 class="wp-block-heading">Nothing is arriving at all</h3>



<ol class="wp-block-list"><li>Run <code>describe-event-source</code>. If <code>State</code> is not <code>ACTIVE</code>, stop here. Everything published so far is gone and you need the reconciliation path.</li><li>Confirm the region of the bus matches the region in the source ARN.</li><li>List your webhook subscriptions through the Admin API and confirm the address is the event-source ARN, not the bus ARN.</li><li>Confirm the subscriptions belong to the app whose access token you are using. Registering with a token from a different app is a common and confusing dead end.</li><li>Check <code>MatchedEvents</code> on the bus with no rule dimension. Non-zero means events are landing and your rules are the problem, not the plumbing.</li></ol>



<h3 class="wp-block-heading">Some topics arrive, order or customer topics do not</h3>



<p class="wp-block-paragraph">This is almost always scopes rather than infrastructure. Order and customer topics sit behind protected customer data access, which is a separate approval in the app configuration on top of the read scopes. Without it, product events flow perfectly and order events silently do not — which looks exactly like a broken rule and is not.</p>



<h3 class="wp-block-heading">The rule matches but the target does nothing</h3>



<p class="wp-block-paragraph">Compare <code>MatchedEvents</code> against <code>SuccessfulInvocationAttempts</code> on the rule. A gap sends you to <code>FailedInvocations</code> and to the DLQ. Check the target&#8217;s resource policy, and check <code>InvocationsFailedToBeSentToDlq</code> before you trust that the DLQ is catching anything.</p>



<h3 class="wp-block-heading">Events arrive, but late</h3>



<p class="wp-block-paragraph">Look at <code>ThrottledRules</code> and at <code>IngestionToInvocationSuccessLatency</code>. Sustained throttling usually means an invocation quota rather than a rule problem, and it shows up first during flash sales, which is the worst possible time to discover it. Load-test the path before a peak event, not after.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list"><li>Creating the partner event source and never associating it with a bus. Silent, total, unrecoverable data loss for the whole window.</li><li>Registering the event bus ARN instead of the event source ARN, then debugging Shopify&#8217;s rejection for an hour.</li><li>Writing rule patterns against the resource shape from an HTTPS webhook, forgetting that the body sits under <code>detail.payload</code>.</li><li>Routing on <code>detail-type</code>. Every Shopify event carries the same one, so a pattern that matches on it alone matches everything.</li><li>Configuring a DLQ through Terraform without the queue policy, and only finding out when you needed it.</li><li>Assuming replay covers ingestion gaps. It replays what reached the bus and nothing else.</li><li>Deduplicating on the resource ID instead of the webhook ID, so legitimate subsequent updates get discarded as duplicates.</li><li>Skipping reconciliation because the pipeline &#8220;works&#8221;. It works right up until it doesn&#8217;t, and that is precisely when you need the other path.</li></ul>



<h2 class="wp-block-heading">Best practices for streaming Shopify events into AWS</h2>



<ul class="wp-block-list"><li>Alarm on the event source state and on <code>MatchedEvents</code> reaching zero. Absence of events is a signal, and it is the only signal you get for the worst failure.</li><li>Dedupe on <code>X-Shopify-Webhook-Id</code> and version-check on <code>X-Shopify-Triggered-At</code>. Two mechanisms, two problems.</li><li>Give every target a DLQ and a retry window you chose deliberately, rather than inheriting the default.</li><li>Keep dedicated rules for money and fulfilment topics so their metrics stay legible; batch the rest behind a catch-all.</li><li>Trim payloads at the Shopify subscription rather than in a Lambda. Filtering after ingestion means you already paid for the bytes.</li><li>Define the whole thing in Terraform or CloudFormation, including the queue policies. This stack has too many one-time console clicks to survive being hand-built twice.</li><li>Point your observability platform at the same bus. Datadog and New Relic are both EventBridge partners, so business events and infrastructure telemetry can share one pipeline instead of two.</li><li>Build reconciliation before your first peak trading period, not after your first missing-order ticket.</li></ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I still need to verify the HMAC signature on the EventBridge path?</h3>



<p class="wp-block-paragraph">No. Only the partner account behind the event source can publish to a partner event bus, and AWS actively rejects attempts to add your own resource policy granting anyone else access. The signature header is still present in the metadata, but the trust boundary is enforced by AWS rather than by your handler. On the HTTPS delivery method, verification remains mandatory.</p>



<h3 class="wp-block-heading">Should I use EventBridge or plain HTTPS webhooks?</h3>



<p class="wp-block-paragraph">HTTPS is simpler, works with any host, and is easier to debug because you can curl your own endpoint. It also puts you on the hook for absorbing burst traffic within a short response deadline, and for keeping the endpoint up well enough that Shopify does not remove the subscription after persistent failures. EventBridge moves that burst absorption to AWS and gives you native fan-out. If your consumers already live in AWS, the operational maths favours EventBridge. If they do not, a dedicated reliability layer such as Hookdeck in front of an HTTPS endpoint is a reasonable alternative and a much smaller change.</p>



<h3 class="wp-block-heading">Can I use one partner event source for multiple stores?</h3>



<p class="wp-block-paragraph">Events from every shop that installed your app flow through the source associated with that app, and the shop is identified by <code>X-Shopify-Shop-Domain</code> in the metadata. You can route per-shop with rule patterns matching that field. For genuine tenant isolation — separate accounts, separate blast radius — you want separate apps and separate sources, because a single bus is a single failure domain.</p>



<h3 class="wp-block-heading">Why do I get duplicate order events even though nothing failed?</h3>



<p class="wp-block-paragraph">Because at-least-once means exactly that. Duplicates are normal operation, not a fault to be investigated. Separately, an order genuinely does change several times shortly after creation — payment capture, risk assessment, post-purchase upsells — so several <code>orders/updated</code> events for one order are expected and are not duplicates at all. Deduplicate on the webhook ID to tell the two apart.</p>



<h3 class="wp-block-heading">What happens to events published while my consumer is broken?</h3>



<p class="wp-block-paragraph">They reach the bus, match your rules, and EventBridge retries the target within your configured window. Once that window is exhausted they go to the DLQ if you have one and are discarded if you do not. The events themselves are not lost at the bus level as long as the source is associated — this is the failure family you can actually engineer your way out of.</p>



<h3 class="wp-block-heading">Can I archive and replay Shopify events?</h3>



<p class="wp-block-paragraph">Yes, with an archive on the partner event bus and an event pattern controlling what gets archived. Budget for three separate charges — processing into the archive, storage while it sits there, and the replay itself — and always set an explicit retention period, because an archive without one grows indefinitely.</p>



<h3 class="wp-block-heading">Does this work the same way for BigCommerce or commercetools?</h3>



<p class="wp-block-paragraph">The AWS half is identical: partner source, association step, bus, rules, targets, and every failure family in this post. What differs is the envelope shape and how you register subscriptions on the vendor side. The association gap in particular bites the same way regardless of which platform is publishing.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Almost everything about streaming Shopify events into AWS degrades loudly. Targets throw errors, retries show up as metrics, dead letters pile up in a queue you can see. Those are the failures you will handle correctly, because they announce themselves.</p>



<p class="wp-block-paragraph">The one that will actually hurt you is the one that reports success on both sides while dropping every event on the floor. Association state and event volume are the two signals that catch it, and neither one appears on any dashboard by default. Add them on day one, before you write the first rule. Everything else in this post can be fixed after the fact; that one cannot.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a hand with your event pipeline?</h2>



<p class="wp-block-paragraph">Most of my work on this stack is either standing it up properly the first time or working out where records went after someone else stood it up. Things I can help with:</p>



<ul class="wp-block-list"><li>Building the Shopify-to-EventBridge path end to end in Terraform, including the queue policies and retry configuration that the console quietly does for you.</li><li>Auditing an existing pipeline for silent loss: source association, region drift, missing DLQ permissions, rules that have been matching nothing since the day they shipped.</li><li>Designing the idempotency and ordering layer — dedupe store, TTLs, version checks — so replays and duplicates stop corrupting downstream state.</li><li>Writing the reconciliation job against the Admin API, with watermarks, drift metrics and alarms that fire before a customer does.</li><li>Cutting EventBridge spend by trimming payloads at the subscription and rationalising archive retention, without losing anything you actually query.</li><li>Load-testing the whole path ahead of a peak trading period so throttling shows up in a test window rather than on the day.</li></ul>



<p class="wp-block-paragraph">If you have a rule pattern that isn&#8217;t matching, a DLQ that&#8217;s mysteriously empty, or a bill that grew faster than your order volume, send me the pattern, the metric graph or the line item and I&#8217;ll tell you what I&#8217;d look at first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/streaming-shopify-events-into-aws/">Streaming Shopify Events into AWS Without Losing Orders</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Optimizing API Calls to Reduce SaaS Costs: Six Levers That Actually Move the Bill</title>
		<link>https://john-nessime.com/blog/devops/reduce-saas-api-costs/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Fri, 14 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[API Cost Optimization]]></category>
		<category><![CDATA[Batch Processing]]></category>
		<category><![CDATA[Cache Control]]></category>
		<category><![CDATA[Cache Stampede]]></category>
		<category><![CDATA[Caching]]></category>
		<category><![CDATA[Cloudflare]]></category>
		<category><![CDATA[Conditional Requests]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[ETag]]></category>
		<category><![CDATA[Exponential Backoff]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[HTTP Headers]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Nginx]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Polling]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[Redis]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Retry Budgets]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[Vendor Spend]]></category>
		<category><![CDATA[Webhooks]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=229</guid>

					<description><![CDATA[<p>Third-party API spend is the one production signal with no error rate attached to it, which is why it creeps up quietly. This is a working engineer's guide to reducing SaaS API costs by changing the shape of your calls: reading the billing unit before you optimise anything, killing pointless polling with conditional requests and webhooks, collapsing N+1 patterns, caching with stampede protection and per-tenant keys, stopping your own retry amplification, and attributing spend so you can prove the work paid off.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/reduce-saas-api-costs/">Optimizing API Calls to Reduce SaaS Costs: Six Levers That Actually Move the Bill</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually comes from finance, not from monitoring. A vendor line on the card went up again, nobody shipped a feature that month, and now somebody wants an explanation by Friday.</p>



<p class="wp-block-paragraph">Here is the awkward part: nothing was broken. No alert fired, no error rate moved, no dashboard turned red. Third-party API spend is one of the few production signals with no failure mode attached to it. A background job that polls a vendor every thirty seconds and gets back the exact same payload it got last time looks, from the outside, identical to a healthy integration. It just costs money every single time.</p>



<p class="wp-block-paragraph">This post is about how to reduce SaaS API costs by changing the shape of your calls rather than by renegotiating your contract. Six levers, in the order I&#8217;d pull them: read the billing unit, kill pointless polling, collapse N+1 patterns, cache honestly, stop your own retry amplification, and attribute spend so you can tell whether any of it worked.</p>



<h2 class="wp-block-heading">Start with the billing unit, not the call count</h2>



<p class="wp-block-paragraph">The single most common mistake I see is optimising HTTP requests when the vendor doesn&#8217;t bill HTTP requests. Before you touch any code, read the pricing page carefully enough to answer one question: <em>what is the countable thing?</em></p>



<p class="wp-block-paragraph">It varies more than people expect:</p>



<ul class="wp-block-list">
<li><strong>Per request.</strong> The obvious one. Geocoding, enrichment, most REST endpoints.</li>

<li><strong>Per session.</strong> Google Maps Platform bills Places Autocomplete this way when you pass a session token: the whole keystroke stream plus the terminating Place Details call is grouped and billed as one unit. Omit the token, or reuse one across sessions, and every keystroke request is billed individually.</li>

<li><strong>Per token issued.</strong> Auth0&#8217;s machine-to-machine quotas count access tokens obtained through the client credentials flow, not the API calls you make with them. A service that fetches a fresh token on every outbound request is burning the meter for no reason.</li>

<li><strong>Per input and output token.</strong> LLM providers. Output is typically priced higher than input, which makes &#8220;be concise&#8221; a cost control and not just a style note.</li>

<li><strong>Per record, event, or message.</strong> Twilio, SendGrid, most streaming and messaging platforms. One API call can carry many billable units.</li>

<li><strong>Not at all.</strong> Stripe doesn&#8217;t bill you per API call; the constraint there is a rate limit, not a line item. Optimising those calls buys you reliability headroom, not money.</li>
</ul>



<p class="wp-block-paragraph">Getting this wrong sends you down the wrong path for weeks. Batching requests against an endpoint that bills per record changes your request count and leaves the invoice exactly where it was. Read the unit first.</p>



<h2 class="wp-block-heading">Lever one: stop polling for things that rarely change</h2>



<p class="wp-block-paragraph">This is almost always the biggest single line, and it&#8217;s invisible because it&#8217;s steady. A sync job on a five-minute cron is 288 calls a day per resource. Multiply by resources, by environments, by the replicas you quietly scaled to three, and you get a number with no relationship at all to how often the data changed.</p>



<h3 class="wp-block-heading">Conditional requests, when the vendor honours them</h3>



<p class="wp-block-paragraph">If the API returns an <code>ETag</code> or <code>Last-Modified</code> header, store it and send it back on the next poll as <code>If-None-Match</code> or <code>If-Modified-Since</code>. If nothing changed you get a <code>304 Not Modified</code> with an empty body.</p>



<pre class="wp-block-code"><code># First call: capture the ETag from the response headers
curl -sS -D - -o /dev/null 
  -H "Authorization: Bearer $TOKEN" 
  https://api.github.com/repos/OWNER/REPO/pulls

# Second call: send it back and check the status code only
curl -sS -o /dev/null -w '%{http_code}n' 
  -H "Authorization: Bearer $TOKEN" 
  -H 'If-None-Match: "PASTE_THE_ETAG_HERE"' 
  https://api.github.com/repos/OWNER/REPO/pulls</code></pre>



<p class="wp-block-paragraph"><code>-D -</code> dumps response headers to stdout so you can see the <code>etag</code> value. <code>-w '%{http_code}'</code> on the second call prints just the status, which is all you care about: <code>304</code> means the resource is unchanged and you saved a payload.</p>



<p class="wp-block-paragraph">Whether that saves <em>money</em> depends on the vendor. GitHub documents that a conditional request returning 304 doesn&#8217;t count against your primary rate limit when it&#8217;s correctly authorised. Plenty of other vendors return 304 happily and bill it anyway. Test against your own invoice before you assume.</p>



<p class="wp-block-paragraph">Three things that bite people with ETags:</p>



<ul class="wp-block-list">
<li>They&#8217;re per-page on paginated endpoints. A 304 on page one says nothing about pages two through five, so store one ETag per page.</li>

<li>They can be scoped to the credential. Rotate a token and your stored ETags may stop matching, which quietly turns your conditional requests back into full ones.</li>

<li>GraphQL endpoints generally don&#8217;t do conditional requests at all. You cache those yourself, keyed on a hash of the query and its variables.</li>
</ul>



<h3 class="wp-block-heading">Webhooks, when they exist</h3>



<p class="wp-block-paragraph">The proper fix is to stop asking. If the vendor supports webhooks, the call count collapses from &#8220;one per interval&#8221; to &#8220;one per actual change&#8221;, which is usually a difference of two orders of magnitude on a slow-moving resource.</p>



<p class="wp-block-paragraph">Be honest about what it costs, though. Webhooks need a publicly reachable endpoint, signature verification, idempotent handling because vendors redeliver, a dead-letter path, and a reconciliation sweep for deliveries that never arrived. On a resource that changes every few seconds, polling is simpler and the savings are small. On one that changes twice a week, the webhook pays for itself immediately.</p>



<p class="wp-block-paragraph">A middle option people forget: many APIs expose a cheap &#8220;has anything changed&#8221; endpoint, a sync cursor, or a delta token. Poll that on a tight interval and only fetch the expensive payload when it moves.</p>



<h2 class="wp-block-heading">Lever two: collapse the N+1 pattern</h2>



<p class="wp-block-paragraph">You know this one from databases, and it shows up identically against third-party APIs. Fetch a list of 200 items, then fetch each item&#8217;s detail individually: that&#8217;s 201 billable calls where the vendor probably offers a way to do it in one or two. Look for:</p>



<ul class="wp-block-list">
<li><strong>Expansion or embed parameters.</strong> Many REST APIs let you inline related objects in the list response so the detail calls disappear entirely.</li>

<li><strong>Field selection.</strong> Asking for fewer fields is free on per-request billing but matters a lot where the field tier drives the price. Google Places is explicit about this: requesting a single higher-tier field upgrades the whole response to that tier&#8217;s pricing.</li>

<li><strong>Bulk and batch endpoints.</strong> Salesforce, Algolia, most CRMs and search platforms have one. They&#8217;re usually documented in a corner nobody reads.</li>

<li><strong>Larger pages.</strong> If the endpoint accepts a page size of 200 and you&#8217;re defaulting to 25, you&#8217;re paying eight times as many calls to fetch identical data.</li>
</ul>



<p class="wp-block-paragraph">The page-size one is the cheapest win in this entire post and it&#8217;s a one-line change. Check your client library defaults today.</p>



<h2 class="wp-block-heading">Lever three: cache with a boring, honest TTL</h2>



<p class="wp-block-paragraph">Caching third-party responses is the lever everyone reaches for first and gets subtly wrong. Three failure modes matter more than the cache itself.</p>



<p class="wp-block-paragraph"><strong>Stampede.</strong> A popular key expires, fifty concurrent requests all miss, and fifty identical calls go out to the vendor. You just paid fifty times for one refresh, at the worst possible moment.</p>



<p class="wp-block-paragraph"><strong>No negative caching.</strong> A lookup that returns &#8220;not found&#8221; is usually not cached at all, so the same failing lookup hammers the vendor forever. Cache your 404s. Short TTL, but cache them.</p>



<p class="wp-block-paragraph"><strong>Cache key design.</strong> If your key doesn&#8217;t include the credential or tenant, you will eventually serve one customer&#8217;s data to another. This is the one mistake in this post that turns a cost problem into an incident.</p>



<p class="wp-block-paragraph">You don&#8217;t necessarily need application code for this. An Nginx instance in front of the vendor handles all three, and it&#8217;s the approach I reach for first when several services hit the same API:</p>



<pre class="wp-block-code"><code>proxy_cache_path /var/cache/nginx/api levels=1:2 keys_zone=api_cache:20m
                 max_size=1g inactive=60m use_temp_path=off;

server {
    listen 127.0.0.1:8080;

    location / {
        proxy_pass https://api.vendor.example;
        proxy_set_header Host api.vendor.example;

        proxy_cache api_cache;
        proxy_cache_key "$request_method$request_uri$http_authorization";

        proxy_cache_valid 200 5m;
        proxy_cache_valid 404 30s;

        proxy_cache_lock on;
        proxy_cache_use_stale updating error timeout;
        proxy_cache_background_update on;

        add_header X-Cache-Status $upstream_cache_status;
    }
}</code></pre>



<p class="wp-block-paragraph">Reading that from the top down, because every directive is there for a reason:</p>



<ul class="wp-block-list">
<li><code>proxy_cache_key</code> includes <code>$http_authorization</code>, so responses are isolated per credential. That&#8217;s the tenant-leak fix.</li>

<li><code>proxy_cache_valid 404 30s</code> is the negative caching. Failed lookups stop being free traffic for the vendor.</li>

<li><code>proxy_cache_lock on</code> is the stampede fix. Only the first request for a given key goes upstream; the rest wait for it.</li>

<li><code>proxy_cache_use_stale updating</code> plus <code>proxy_cache_background_update on</code> serves the stale copy while a single background request refreshes it. Latency stays flat and you don&#8217;t pay a thundering herd at every expiry.</li>

<li><code>X-Cache-Status</code> gives you <code>HIT</code>, <code>MISS</code>, <code>STALE</code> or <code>UPDATING</code> in the response so you can actually measure your hit rate instead of guessing at it.</li>
</ul>



<p class="wp-block-paragraph">The trade-off is worth stating plainly: you&#8217;re now serving data that can be five minutes stale, and you own another moving part. Fine for a currency rate or a product catalogue. Not fine for an inventory count on a checkout page. Set the TTL from how wrong the data is allowed to be, never from how much you want to save.</p>



<p class="wp-block-paragraph">Where the caller is distributed, Redis with a short lock key does the same job in application code, and a small always-on VPS from something like InterServer or DigitalOcean is plenty to host either the proxy or the cache. At the edge, Cloudflare Workers with KV lets you cache vendor responses close to users, which helps when the calls originate from browsers rather than your backend.</p>



<h2 class="wp-block-heading">Lever four: stop paying for your own retries</h2>



<p class="wp-block-paragraph">This produces the sudden spike rather than the slow creep, and people miss it because retries feel like resilience.</p>



<p class="wp-block-paragraph">The vendor wobbles. Every client retries. With a fixed delay they retry in lockstep, so the vendor gets a synchronised wave on top of normal load, struggles longer, and generates more retries. AWS&#8217;s architecture write-up on exponential backoff and jitter is the canonical reference, and its central finding is worth repeating: backoff alone still leaves visible clusters of calls. Randomising each client&#8217;s delay is what decorrelates the herd, and in their simulation it cut total call count substantially.</p>



<pre class="wp-block-code"><code>import random, time

BASE = 0.5      # seconds
CAP  = 30.0     # never sleep longer than this

def sleep_before_retry(attempt):
    # capped exponential backoff, then full jitter
    window = min(CAP, BASE * (2 ** attempt))
    time.sleep(random.uniform(0, window))</code></pre>



<p class="wp-block-paragraph">The cap stops a long outage turning into hour-long sleeps. The <code>random.uniform(0, window)</code> is the jitter: two processes that failed at the same instant now wake at different times. Note that full jitter can occasionally produce a near-zero delay, which is fine for transient errors but too aggressive for throttling responses that reset on a per-minute window. For a <code>429</code>, honour the <code>Retry-After</code> header if the vendor sends one, and use a jitter formula with a guaranteed floor if they don&#8217;t.</p>



<p class="wp-block-paragraph">Three more rules that cost nothing to implement:</p>



<ol class="wp-block-list">
<li><strong>Never retry a 4xx except 429.</strong> A 400 or a 422 will fail identically forever. Every one of those retries is a billed call buying you nothing.</li>

<li><strong>Cap retries as a fraction of traffic.</strong> A retry budget (say, retries may not exceed some small percentage of total requests) is the only control that sees the system-wide picture. Per-client backoff can&#8217;t.</li>

<li><strong>Check your layers.</strong> If the SDK retries three times, your HTTP client retries three times, and your job runner retries three times, one logical operation is up to twenty-seven calls. This stacking is extremely common and almost never intentional.</li>
</ol>



<p class="wp-block-paragraph">For writes, retrying safely needs an idempotency key so the vendor deduplicates rather than creating two of whatever you asked for. Stripe&#8217;s <code>Idempotency-Key</code> header is the pattern most others copied. Without it, teams disable retries on writes entirely and then eat manual reconciliation work instead.</p>



<h2 class="wp-block-heading">Lever five: move latency-tolerant work off the synchronous path</h2>



<p class="wp-block-paragraph">Ask a blunt question about each integration: does a human wait on this response?</p>



<p class="wp-block-paragraph">Nightly enrichment, classification, report generation, bulk address validation, embedding backfills. Nobody is waiting. Several vendors, including the major LLM providers, offer asynchronous batch endpoints at a discount to the synchronous rate in exchange for a looser delivery window. Rates move, so check current terms rather than a number you read somewhere, but the mechanism is stable and the discount is usually substantial.</p>



<p class="wp-block-paragraph">Even without a batch endpoint, moving work behind a queue lets you coalesce. Ten events touching the same customer record within a minute become one enrichment call instead of ten. That deduplication window is often worth more than the batch discount.</p>



<h2 class="wp-block-heading">Lever six: attribute the spend or you&#8217;re guessing</h2>



<p class="wp-block-paragraph">Everything above is unfalsifiable until you can answer: which feature, which tenant, which job. The invoice gives you the total and never the cause. What I&#8217;d put in place, in order:</p>



<ol class="wp-block-list">
<li><strong>Route outbound calls through one place.</strong> A shared HTTP client wrapper or the caching proxy above. If every service instantiates its own client, you have no chokepoint and no measurement.</li>

<li><strong>Emit a counter per call</strong> with labels for vendor, endpoint, status class, and cache result. Keep the label set small; per-tenant labels will blow up your metrics cardinality and trade one bill for another.</li>

<li><strong>Build one unit-cost metric.</strong> Vendor spend per active tenant per month. It&#8217;s the only number that survives growth, because raw spend going up during a good quarter is not a problem.</li>

<li><strong>Alert on call rate, not on the bill.</strong> A rate alert fires in minutes. An invoice fires in thirty days. Grafana, Prometheus, Datadog, CloudWatch, whatever you already run is fine; the point is that the alert exists at all.</li>

<li><strong>Track cache hit rate as a first-class SLI.</strong> When someone changes a cache key format and the hit rate silently drops to zero, this is the graph that tells you before the invoice does.</li>
</ol>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting: the bill jumped and nothing shipped</h2>



<p class="wp-block-paragraph">Work down this list. In my experience the cause is usually in the first four.</p>



<ul class="wp-block-list">
<li><strong>Replica count changed.</strong> Autoscaling or a Kubernetes replica bump multiplies every cron-driven poller. Check whether your scheduled jobs have leader election, or whether all five pods are dutifully polling the same endpoint.</li>

<li><strong>Cache hit rate collapsed.</strong> A changed key format, a Redis eviction policy under memory pressure, or a TTL someone dropped &#8220;temporarily&#8221; during debugging. Graph the hit rate over the period, not the current value.</li>

<li><strong>Retry loop against a persistent 4xx.</strong> Look for a flat, high rate of identical failing calls. This has the distinctive signature of high call volume with zero successful responses.</li>

<li><strong>A non-production environment pointed at production credentials.</strong> Staging, a CI pipeline, or an integration test suite running on every commit. Check whether calls are arriving from IPs you don&#8217;t recognise.</li>

<li><strong>Session or token handling regressed.</strong> A refactor that dropped a session token parameter, or a service that stopped caching its auth token and now mints a new one per request.</li>

<li><strong>Legitimate growth in an expensive segment.</strong> A few large customers using a costly feature move the bill without moving call count much. Not a bug, but worth ruling in before you go hunting for one.</li>
</ul>



<p class="wp-block-paragraph">If the vendor exposes per-key usage breakdowns in their dashboard, split your credentials by service before your next incident. Separate API keys per environment and per major consumer turn a whodunnit into a two-minute lookup.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Optimising the endpoint that&#8217;s easy to change rather than the one that dominates the bill. Rank by spend first.</li>

<li>Caching without a per-credential key, and finding out the hard way in a multi-tenant system.</li>

<li>Treating 304 responses as free when your specific vendor bills them anyway.</li>

<li>Setting TTLs from a savings target rather than a staleness tolerance, and finding out during a customer escalation.</li>

<li>Leaving retry logic enabled at three separate layers and calling the result &#8220;defence in depth&#8221;.</li>

<li>Adding per-tenant labels to outbound-call metrics and moving the cost from the API vendor to the observability vendor.</li>

<li>Shipping the optimisation and never checking the next invoice against the prediction.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth keeping</h2>



<ul class="wp-block-list">
<li>One outbound HTTP client wrapper per service, with timeouts, retries, and metrics configured in exactly one place.</li>

<li>Separate API credentials per environment and per major consumer, so usage dashboards mean something.</li>

<li>Cache auth tokens until shortly before expiry. Refresh early, not on every call.</li>

<li>Add jitter to scheduled jobs as well as to retries. Everything set to run at the top of the hour arrives at the vendor as a spike.</li>

<li>Write down the staleness tolerance for each cached resource next to its TTL, so the next person doesn&#8217;t have to reverse-engineer your reasoning.</li>

<li>Re-check the pricing page after any vendor product update. Billing units change more often than endpoints do.</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">How do I find out which service is driving my API spend?</h3>



<p class="wp-block-paragraph">Start with the vendor&#8217;s own usage dashboard broken down by API key. If all your services share one key, that&#8217;s the first thing to fix. Failing that, log every outbound call at the client wrapper with the endpoint and calling service, and aggregate for a week. A week of honest logs beats a month of theorising.</p>



<h3 class="wp-block-heading">Do ETags actually reduce SaaS API costs, or just help with rate limits?</h3>



<p class="wp-block-paragraph">It depends entirely on the vendor. GitHub documents that an authorised conditional request returning 304 doesn&#8217;t count against the primary rate limit. Others count every request regardless of status. Run a controlled test: a known number of conditional calls over a billing period, then compare against the usage report. Don&#8217;t assume either way.</p>



<h3 class="wp-block-heading">Is caching third-party API responses allowed?</h3>



<p class="wp-block-paragraph">Usually yes, but not always, and some terms of service restrict how long you may store certain data. Mapping and place data is a common example of restricted caching. Check the terms for the specific vendor before you build a permanent store, particularly if you&#8217;re caching anything personally identifiable.</p>



<h3 class="wp-block-heading">How much can caching realistically cut an API bill?</h3>



<p class="wp-block-paragraph">Nobody can give you a number without knowing your traffic. Estimate it yourself: for a candidate endpoint, count distinct request keys versus total requests over a day. That ratio is roughly your ceiling for hit rate. Multiply by the endpoint&#8217;s share of spend. If the answer is a couple of percent, spend your time elsewhere.</p>



<h3 class="wp-block-heading">Should I always replace polling with webhooks?</h3>



<p class="wp-block-paragraph">No. The comparison is change frequency against poll frequency. If the resource changes almost as often as you poll, you save little and take on signature verification, redelivery handling, and a reconciliation job. Webhooks win decisively on slow-changing resources and on anything where you&#8217;re polling more than a handful of objects.</p>



<h3 class="wp-block-heading">Does rate limiting my own users reduce vendor costs?</h3>



<p class="wp-block-paragraph">Only when user actions map directly to vendor calls, such as a search box that fires an upstream request per keystroke. Debouncing input and enforcing per-plan quotas helps a lot there. For background processing driven by your own schedules, user rate limits change nothing.</p>



<h3 class="wp-block-heading">Should I disable retries to save money?</h3>



<p class="wp-block-paragraph">No. Fix them instead. Capped exponential backoff with jitter, no retries on non-429 client errors, a retry budget, and retry logic living at exactly one layer. Disabling retries entirely trades a cost problem for a reliability problem, and reliability problems are more expensive.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Most efforts to reduce SaaS API costs fail because they start with code. They should start with two questions: what is the vendor actually counting, and which of my calls returned information I already had?</p>



<p class="wp-block-paragraph">Answer those honestly and the work usually collapses to a handful of unglamorous changes. Raise a page size. Cache a token. Put a lock around a cache refresh. Delete a poller that was watching something that changes twice a week. None of it is clever. All of it shows up on the next invoice, which is the only review that counts.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help getting your API spend under control?</h2>



<p class="wp-block-paragraph">This is work I take on regularly. Typical engagements look like:</p>



<ul class="wp-block-list">
<li>Auditing outbound API traffic to find which endpoints, jobs and tenants are driving the bill, and ranking them by spend rather than by how obvious they are.</li>

<li>Building a caching layer in front of chatty vendor APIs, with Nginx, Redis or Cloudflare Workers, including stampede protection and per-tenant key isolation.</li>

<li>Replacing polling jobs with webhook receivers or delta-token syncs, with signature verification, idempotent handling and a dead-letter path.</li>

<li>Auditing timeout and retry configuration across services to remove stacked retries and add proper backoff, jitter and retry budgets.</li>

<li>Moving latency-tolerant workloads onto batch or queue-based paths, with deduplication windows that cut call volume before it reaches the vendor.</li>

<li>Setting up call-rate and cache-hit-rate dashboards and alerts in Prometheus, Grafana, Datadog or CloudWatch so the invoice is never the first signal.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;ve got a cost graph that&#8217;s climbing, a job you suspect is over-polling, or a client wrapper you&#8217;re not sure about, send it over and I&#8217;ll tell you what I&#8217;d look at first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/reduce-saas-api-costs/">Optimizing API Calls to Reduce SaaS Costs: Six Levers That Actually Move the Bill</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Shopify Sales Dashboard with AWS: Build One That Actually Reconciles</title>
		<link>https://john-nessime.com/blog/technical-guides/shopify-sales-dashboard-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 04 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Business Intelligence]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Ecommerce Analytics]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[Partition Projection]]></category>
		<category><![CDATA[QuickSight]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[SQL]]></category>
		<category><![CDATA[Webhooks]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=142</guid>

					<description><![CDATA[<p>Most Shopify dashboards built on AWS work perfectly for about three weeks, then quietly drift away from the numbers in the Shopify admin. Here is why that happens, and how to design the ingestion, storage and query layers so your totals still reconcile six months in.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/shopify-sales-dashboard-aws/">Shopify Sales Dashboard with AWS: Build One That Actually Reconciles</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone in the finance channel posts two screenshots side by side. On the left, the dashboard you built. On the right, the Shopify admin. The totals don&#8217;t match, and they&#8217;re not off by a rounding error either. They&#8217;re off by enough that nobody wants to use your dashboard for anything that matters.</p>



<p class="wp-block-paragraph">The frustrating part is that the pipeline is fine. Webhooks are arriving, Lambda is running clean, S3 has the files, Athena returns rows. Nothing is broken in the way monitoring understands &#8220;broken&#8221;. The pipeline is faithfully recording something that is no longer true.</p>



<p class="wp-block-paragraph">This post is about building a Shopify sales dashboard with AWS that survives that conversation. Not the wiring, which is well documented and mostly straightforward, but the design decisions that determine whether your numbers still hold up six months in. I&#8217;ll cover the three ingestion paths and when each one is the right call, why append-only pipelines drift, how to lay out S3 and Athena so recomputation is cheap, and what to do when the totals are already wrong.</p>



<h2 class="wp-block-heading">Why a Shopify sales dashboard with AWS drifts from the admin</h2>



<p class="wp-block-paragraph">Here&#8217;s the thing that catches almost everyone: <strong>a Shopify order is not an event, it&#8217;s a mutable record.</strong></p>



<p class="wp-block-paragraph">An event pipeline assumes facts are immutable once written. A payment happened. A shipment left. You append it, you never touch it again, and the sum of the log is the truth. That model is why streaming architectures are so clean, and it&#8217;s exactly wrong for order data.</p>



<p class="wp-block-paragraph">An order created on Monday can be edited on Tuesday, partially refunded on Friday, and fully refunded three weeks later. Every one of those changes belongs, financially, to Monday. If your pipeline appends the <code>orders/create</code> payload and never revisits it, Monday&#8217;s revenue is frozen at the moment of checkout and it will only ever be too high.</p>



<p class="wp-block-paragraph">This is the invisible failure. Nothing alerts. No queue backs up. Your dashboard is confidently wrong, and the gap widens roughly in proportion to your return rate. A store with a two percent return rate takes a long time to notice. A fashion store running thirty percent returns notices in about a month, usually via an angry accountant.</p>



<h3 class="wp-block-heading">The four adjustments that move historical numbers</h3>



<ul class="wp-block-list">
<li><strong>Refunds.</strong> Full or partial. A refund carries its own <code>created_at</code>, which is when the money moved back. The order it belongs to has a different, earlier date. You need both, and which one you attribute to depends on whether finance wants cash-basis or order-basis reporting. Ask before you build.</li>

<li><strong>Order edits.</strong> A merchant adds a line item or adjusts a quantity after the fact. The original payload is now stale. Shopify exposes both the original and the current totals precisely because of this.</li>

<li><strong>Cancellations.</strong> A cancelled order keeps existing in the API. If you filter only on payment status you will happily keep counting it.</li>

<li><strong>Test and draft orders.</strong> Test orders carry a flag marking them as such. Nobody remembers to filter these until a QA run during a quiet week produces a suspicious spike.</li>
</ul>



<p class="wp-block-paragraph">The design consequence is simple to state and annoying to implement: <strong>your pipeline must be able to recompute any past day.</strong> Every storage and partitioning decision below follows from that one requirement.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Getting data out of Shopify: three paths, three trade-offs</h2>



<p class="wp-block-paragraph">Before anything else: new Shopify apps are built on the GraphQL Admin API. The REST Admin API has been designated a legacy API and new public apps must use GraphQL. If you&#8217;re starting fresh, start there. If you inherited a REST integration, it probably still runs, but you&#8217;re on borrowed time and you should plan the migration rather than discover the deadline.</p>



<h3 class="wp-block-heading">Path 1: EventBridge partner event source</h3>



<p class="wp-block-paragraph">Shopify can deliver webhooks straight into an Amazon EventBridge partner event bus in your account. No public endpoint, no API Gateway, no HMAC verification code, because verification only applies to HTTPS deliveries. Shopify&#8217;s own docs confirm EventBridge and Pub/Sub deliveries skip it.</p>



<p class="wp-block-paragraph">You create the source in the Shopify app configuration using your AWS account ID, region and a source name, then associate it with an event bus in the EventBridge console and write rules to route it. The address you register with Shopify is the <em>partner event source</em> ARN, not the event bus ARN. That distinction accounts for a large share of the &#8220;I set it up and nothing arrives&#8221; threads on the Shopify forums.</p>



<p class="wp-block-paragraph">A rule matching everything from the Shopify partner source looks like this. Start broad, then narrow once you&#8217;ve seen the real shape of an event:</p>



<pre class="wp-block-code"><code>{
  "source": [ { "prefix": "aws.partner/shopify.com" } ]
}</code></pre>



<p class="wp-block-paragraph">Send that to an SQS queue with a dead-letter queue attached rather than straight to Lambda. Buffering gives you a replay buffer when a downstream deploy goes wrong, and the DLQ means a bad payload parks itself instead of poisoning the whole rule. This is the path I reach for first for anything already on AWS.</p>



<h3 class="wp-block-heading">Path 2: HTTPS webhooks into API Gateway and Lambda</h3>



<p class="wp-block-paragraph">The conventional route, and the right one if you need webhook delivery outside AWS too, or you want the payloads to pass through something you fully control. The cost is that you now own an internet-facing endpoint and the HMAC verification on it.</p>



<p class="wp-block-paragraph">Verify against the <strong>raw request body</strong>, before any JSON parsing. Re-serialising the payload changes byte-for-byte content and the signature will never match. Use a constant-time comparison so the check doesn&#8217;t leak timing information:</p>



<pre class="wp-block-code"><code>import base64, hashlib, hmac

def verify(raw_body: bytes, header_hmac: str, secret: str) -&gt; bool:
    digest = hmac.new(secret.encode(), raw_body, hashlib.sha256).digest()
    computed = base64.b64encode(digest).decode()
    return hmac.compare_digest(computed, header_hmac)</code></pre>



<p class="wp-block-paragraph">Shopify sends the signature in the <code>X-Shopify-Hmac-SHA256</code> header, base64-encoded, computed with your app&#8217;s client secret over the raw body. Store that secret in Secrets Manager or as an SSM SecureString parameter, not in a Lambda environment variable.</p>



<p class="wp-block-paragraph">Acknowledge fast. Shopify&#8217;s timeout is short and it retries with backoff over a finite window, so a handler that does real work inline will generate a wall of duplicate deliveries during a flash sale, exactly when you can least afford it. Return 2xx immediately, do the work asynchronously.</p>



<h3 class="wp-block-heading">Path 3: scheduled GraphQL bulk pull</h3>



<p class="wp-block-paragraph">Webhooks give you low latency. They do not give you completeness. Anything that fails past its retry window is gone, and Shopify will eventually remove a subscription that keeps failing. That&#8217;s a silent data loss mode with no local symptom at all.</p>



<p class="wp-block-paragraph">So run a scheduled reconciliation pull alongside the stream. Shopify&#8217;s GraphQL bulk operations are built for this: you submit a query, it runs asynchronously, and you fetch a JSONL result file when it finishes. That&#8217;s the right tool for backfills and nightly catch-up, rather than paginating thousands of pages against a points-based rate limiter and getting throttled halfway through.</p>



<p class="wp-block-paragraph">A nightly job that re-pulls the last seven to fourteen days and overwrites those partitions costs almost nothing and quietly fixes every category of drift described above. If you build one thing from this post, build that.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Decide what &#8220;revenue&#8221; means before you write a line of SQL</h2>



<p class="wp-block-paragraph">This is where most reconciliation arguments actually live, and it isn&#8217;t an engineering problem at all until you&#8217;ve had the conversation.</p>



<p class="wp-block-paragraph">Shopify&#8217;s own sales reporting builds total sales from gross sales, minus discounts, minus returns, plus taxes and shipping. Gift card sales sit outside that in a separate finance report. If your dashboard sums order totals and calls it revenue, you have built a different metric with the same name, and it will disagree with the admin forever no matter how good your pipeline is.</p>



<p class="wp-block-paragraph">Write the definition down. Put it in the dashboard as a tooltip. When someone challenges a number, you want the argument to be about the definition, not about whether your infrastructure works.</p>



<h3 class="wp-block-heading">The currency trap</h3>



<p class="wp-block-paragraph">If the store sells in more than one currency, the money fields split in two. Shopify exposes totals as a set containing both <code>shop_money</code> and <code>presentment_money</code>: the amount in the store&#8217;s base currency, and the amount the customer actually saw and paid.</p>



<p class="wp-block-paragraph">Sum the presentment amounts across a multi-currency store and you get a number with no meaning at all, euros and yen added together as if they were the same unit. For a single reporting figure you want the shop-currency side. Keep the presentment amount and its currency code in the table anyway, because the day someone asks &#8220;how much did we actually sell in Germany&#8221;, you&#8217;ll want it and it is painful to backfill.</p>



<p class="wp-block-paragraph">One caveat worth knowing: orders created through the API rather than through checkout can behave differently from native multi-currency checkout orders. If your store takes orders from an ERP or a marketplace integration, spot-check a few of those specifically.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Storage layout: partition by order date, never by arrival date</h2>



<p class="wp-block-paragraph">Two layers in S3. Keep them separate and keep them honest about what they are.</p>



<ol class="wp-block-list">
<li><strong>Raw.</strong> Every payload exactly as received, partitioned by ingestion date. Append-only, never edited. This is your audit trail and your rebuild source. Lifecycle it to a colder storage class after a few months, don&#8217;t delete it.</li>

<li><strong>Curated.</strong> One row per order representing current state, in Parquet, partitioned by <em>order date</em>. This is what the dashboard queries. It is derived, disposable and rewritable.</li>
</ol>



<p class="wp-block-paragraph">The partitioning choice on the curated layer is the load-bearing decision in the whole design. If you partition by arrival date, which is what Amazon Data Firehose does by default because it buckets on the moment it writes the file, then a refund that arrives three weeks late lands in today&#8217;s partition. Correcting Monday now means finding and rewriting fragments scattered across twenty other partitions. Partitioned by order date, correcting Monday means overwriting exactly one prefix.</p>



<p class="wp-block-paragraph">Firehose can do this with dynamic partitioning, which routes records by keys inside the payload rather than by write time. If you&#8217;re not using Firehose, extract the order date in your Lambda and write the prefix yourself.</p>



<h3 class="wp-block-heading">Use partition projection so Athena stops guessing</h3>



<p class="wp-block-paragraph">The default Glue Data Catalog approach means running a crawler or issuing <code>MSCK REPAIR TABLE</code> to register new partitions. Forget one and you get a query that silently returns nothing for recent days. Nobody notices until Monday.</p>



<p class="wp-block-paragraph">Partition projection removes the metastore lookup entirely. You tell Athena the shape of the partition keys and it calculates the prefixes at query time:</p>



<pre class="wp-block-code"><code>CREATE EXTERNAL TABLE shop_orders (
  order_id             bigint,
  order_number         string,
  created_at           timestamp,
  financial_status     string,
  cancelled_at         timestamp,
  is_test              boolean,
  total_shop           decimal(12,2),
  shop_currency        string,
  total_presentment    decimal(12,2),
  presentment_currency string
)
PARTITIONED BY (order_date string)
STORED AS PARQUET
LOCATION 's3://your-bucket/curated/orders/'
TBLPROPERTIES (
  'projection.enabled' = 'true',
  'projection.order_date.type' = 'date',
  'projection.order_date.format' = 'yyyy-MM-dd',
  'projection.order_date.range' = '2019-01-01,NOW',
  'projection.order_date.interval' = '1',
  'projection.order_date.interval.unit' = 'DAYS',
  'storage.location.template' =
    's3://your-bucket/curated/orders/order_date=${order_date}/'
);</code></pre>



<p class="wp-block-paragraph">Set the range start to your store&#8217;s actual first order month. Projection generates every prefix in the range, so a range starting a decade too early makes wide scans slower for no benefit.</p>



<p class="wp-block-paragraph">Parquet matters here for the same reason. Athena bills on bytes scanned, so a columnar format with good compression cuts the bill directly, and a dashboard that only ever selects six columns from a forty-column table never touches the rest.</p>



<h3 class="wp-block-heading">Net sales in one query</h3>



<p class="wp-block-paragraph">With refunds in their own table keyed by order and carrying their own date, attributing them back to the original order day is a left join and a subtraction:</p>



<pre class="wp-block-code"><code>SELECT
    o.order_date,
    SUM(o.total_shop)                                AS gross_shop,
    SUM(COALESCE(r.refunded_shop, 0))                AS refunded_shop,
    SUM(o.total_shop - COALESCE(r.refunded_shop, 0)) AS net_shop
FROM shop_orders o
LEFT JOIN (
    SELECT order_id, SUM(amount_shop) AS refunded_shop
    FROM shop_refunds
    GROUP BY order_id
) r ON r.order_id = o.order_id
WHERE o.order_date BETWEEN '2025-01-01' AND '2025-01-31'
  AND o.is_test = false
  AND o.cancelled_at IS NULL
GROUP BY o.order_date
ORDER BY o.order_date;</code></pre>



<p class="wp-block-paragraph">Note the two filters doing quiet work at the bottom. Those two lines are the difference between a number finance accepts and a number they don&#8217;t.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Choosing the dashboard layer</h2>



<p class="wp-block-paragraph">Once the data is correct, this part is genuinely a preference. All of these work.</p>



<ul class="wp-block-list">
<li><strong>Amazon QuickSight</strong>, now delivered as part of Amazon Quick Suite, is the least-friction option if you&#8217;re already in AWS. Its in-memory SPICE layer means viewers aren&#8217;t firing an Athena query per chart interaction, which controls both latency and scan cost. Per-viewer pricing tends to be the deciding factor either way, so model it for your actual audience size before committing.</li>

<li><strong>Grafana</strong> with the Athena data source is a good fit if you&#8217;re already running Grafana for infrastructure and want commercial and operational panels on one screen. Grafana Cloud removes the hosting question if you&#8217;d rather not run it.</li>

<li><strong>Power BI</strong> makes sense when the finance team already lives in Microsoft 365 and models in DAX. The cross-cloud hop is real but manageable.</li>

<li><strong>Metabase</strong> or a self-hosted alternative on a small VPS from a provider like InterServer or Hetzner is the pragmatic answer for a handful of internal viewers, where per-seat BI licensing costs more than the entire pipeline.</li>
</ul>



<p class="wp-block-paragraph">The honest trade-off: managed BI costs more per month and saves you from becoming the person who patches the reporting server. Self-hosting inverts that. Neither is wrong, but pick deliberately rather than by inertia.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting: symptom to cause</h2>



<h3 class="wp-block-heading">Totals are consistently higher than the Shopify admin</h3>



<p class="wp-block-paragraph">Almost always refunds, cancellations or test orders. Check in that order. If the gap grows with the age of the reporting window, it&#8217;s refunds. If it&#8217;s a fixed offset on specific days, look for test orders or a QA run.</p>



<h3 class="wp-block-heading">Totals are lower, and recent days are missing rows</h3>



<p class="wp-block-paragraph">Either partitions aren&#8217;t registered, which projection fixes permanently, or the webhook subscription has been dropped after repeated delivery failures. Check the subscription still exists before you go digging through Lambda logs. A nightly bulk pull would have masked this, which is another argument for having one.</p>



<h3 class="wp-block-heading">Orders appear twice</h3>



<p class="wp-block-paragraph">Shopify&#8217;s delivery model is at-least-once, not exactly-once, and you may also have more than one subscription on the same topic. Deduplicate on the delivery ID header before you touch anything else, and make the write itself idempotent so a duplicate is a no-op rather than a second row.</p>



<h3 class="wp-block-heading">Numbers are right on the daily view, wrong on the monthly</h3>



<p class="wp-block-paragraph">Timezone. Order timestamps carry an offset; your partition key is a date string. If you derive the date in UTC and the store reports in a local timezone, orders near midnight land on the wrong day. That averages out over a month, which is exactly why the discrepancy hides until month boundaries.</p>



<h3 class="wp-block-heading">Athena costs jumped without more data</h3>



<p class="wp-block-paragraph">Someone built a dashboard with a filter that doesn&#8217;t hit the partition column, so every panel refresh scans the full table. Look at bytes scanned per query and check whether the BI tool is caching results or re-querying on every interaction.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating orders as immutable events and never revisiting a past day.</li>

<li>Partitioning on arrival time because that&#8217;s the default, then discovering corrections are expensive.</li>

<li>Verifying the HMAC against a re-serialised body instead of the raw bytes.</li>

<li>Doing real work inside the webhook handler, generating duplicates under load.</li>

<li>Summing presentment amounts across currencies.</li>

<li>Registering the event bus ARN with Shopify instead of the partner event source ARN.</li>

<li>Relying on webhooks alone with no scheduled reconciliation.</li>

<li>Shipping a &#8220;revenue&#8221; number without ever defining what it includes.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Keep raw and curated layers separate. Raw is append-only; curated is rewritable.</li>

<li>Make every partition idempotently rebuildable from raw. Test that path deliberately, before you need it.</li>

<li>Buffer through SQS with a dead-letter queue. Free replay, free isolation of bad payloads.</li>

<li>Run a nightly bulk pull over a rolling window and overwrite those partitions.</li>

<li>Use partition projection. It removes an entire category of silent failure.</li>

<li>Alarm on the absence of events, not just on errors. A CloudWatch alarm on zero orders processed in an hour during business hours catches broken subscriptions the same day.</li>

<li>Store the API secret in Secrets Manager and scope the Lambda role to the exact prefixes it writes.</li>

<li>Publish a reconciliation panel comparing your total to the admin&#8217;s for the same window. Surfacing the gap builds more trust than hiding it.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need a data warehouse, or is S3 and Athena enough?</h3>



<p class="wp-block-paragraph">For a single store&#8217;s order data, S3 with Athena is almost certainly enough, and it&#8217;s cheaper because you pay per query rather than for a running cluster. Redshift starts to earn its place when you&#8217;re joining Shopify data against several other large sources, or when concurrent query load makes Athena&#8217;s queue times noticeable.</p>



<h3 class="wp-block-heading">How near-real-time can this be?</h3>



<p class="wp-block-paragraph">Events land within seconds. The practical floor is your buffering window, since writing one tiny file per order gives you a small-files problem that ruins query performance. A few minutes of buffering is the usual compromise. If you genuinely need sub-minute order counts, put a live counter in DynamoDB alongside the analytical pipeline rather than trying to make the data lake do both jobs.</p>



<h3 class="wp-block-heading">Which webhook topics should I subscribe to?</h3>



<p class="wp-block-paragraph">At minimum, order creation, order update, order cancellation and refund creation. Update and refund topics are the ones people skip, and they&#8217;re exactly the ones carrying the corrections. Subscribe to fewer topics than you think you need and add rather than subscribing to everything, since every extra topic is volume you pay to store and process.</p>



<h3 class="wp-block-heading">Can I skip AWS and use a connector tool?</h3>



<p class="wp-block-paragraph">Yes, and for many stores that&#8217;s the right answer. A managed connector into a hosted warehouse gets you a working dashboard in an afternoon. You&#8217;re paying a monthly fee to avoid owning any of this, and trading away control over the data model. Building it on AWS wins when you need Shopify data joined to systems the connector doesn&#8217;t cover, or when row-based connector pricing outgrows the infrastructure cost.</p>



<h3 class="wp-block-heading">How do I backfill historical orders?</h3>



<p class="wp-block-paragraph">Use a GraphQL bulk operation rather than paginating the API. Submit the query, poll for completion, then stream the JSONL result into your raw bucket and run the same transformation your live pipeline uses. If backfill and live processing use different code paths, they will diverge, and you&#8217;ll spend an afternoon working out which one is lying.</p>



<h3 class="wp-block-heading">What does a setup like this cost to run?</h3>



<p class="wp-block-paragraph">For a typical single store, the pipeline itself is small money: Lambda invocations, a few gigabytes in S3, and Athena billed on bytes scanned, which partitioning and Parquet keep low. The BI seats are usually the largest line item, which is why the dashboard layer decision deserves more thought than the ingestion one. Model it against current published rates rather than trusting any figure you read in a blog post, including this one.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">A Shopify sales dashboard with AWS doesn&#8217;t fail because the pipeline breaks. It fails because the pipeline keeps working perfectly on data that has since changed underneath it.</p>



<p class="wp-block-paragraph">Design for correction from the first commit. Partition by order date, keep the raw layer so you can always rebuild, run a scheduled pull to catch what the stream missed, and agree on what revenue means before anyone builds a chart. Do that and the Monday morning screenshot comparison becomes a non-event, which is the highest praise a reporting pipeline ever gets.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with your Shopify data pipeline on AWS?</h2>



<p class="wp-block-paragraph">I design and build ecommerce data pipelines and reporting stacks on AWS. Typical engagements look like:</p>



<ul class="wp-block-list">
<li>Working out why an existing Shopify dashboard disagrees with the admin, and fixing the root cause rather than patching the query</li>

<li>Building the ingestion layer end to end: EventBridge or API Gateway, Lambda, SQS with dead-letter handling, and a scheduled GraphQL bulk reconciliation job</li>

<li>Designing the S3 layout, Glue schema and Athena tables so past days can be recomputed cheaply and partitions never go missing</li>

<li>Migrating REST Admin API integrations to GraphQL before the deadline forces the issue</li>

<li>Building the dashboard itself in QuickSight, Grafana or Metabase, including the metric definitions finance will actually sign off on</li>

<li>Cutting Athena scan costs and BI licensing on a reporting stack that has grown more expensive than anyone planned</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;re in the middle of one of these, send me the actual thing: the Athena query, the S3 prefix layout, the two totals that don&#8217;t match. It&#8217;s a much faster conversation than describing it in the abstract.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/shopify-sales-dashboard-aws/">Shopify Sales Dashboard with AWS: Build One That Actually Reconciles</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
