<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>OAuth | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/oauth/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/oauth/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sun, 20 Sep 2026 13:07:08 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>OAuth | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/oauth/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Six Signs GoHighLevel Needs a Developer, Not Another Workflow</title>
		<link>https://john-nessime.com/blog/workflow-automation/gohighlevel-custom-development-vs-workflows/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 17 Sep 2026 06:00:00 +0000</pubDate>
				<category><![CDATA[Enterprise Integration]]></category>
		<category><![CDATA[SaaS Engineering]]></category>
		<category><![CDATA[Workflow Automation]]></category>
		<category><![CDATA[Architecture Debt]]></category>
		<category><![CDATA[CRM Audit]]></category>
		<category><![CDATA[Error Handling]]></category>
		<category><![CDATA[Execution Logs]]></category>
		<category><![CDATA[Exponential Backoff]]></category>
		<category><![CDATA[GoHighLevel]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[iPaaS]]></category>
		<category><![CDATA[Marketing Automation]]></category>
		<category><![CDATA[n8n]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Over-Engineering]]></category>
		<category><![CDATA[Private Integration]]></category>
		<category><![CDATA[Project Scoping]]></category>
		<category><![CDATA[Race Conditions]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[Reconciliation]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Sub-Account Management]]></category>
		<category><![CDATA[Webhooks]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=647</guid>

					<description><![CDATA[<p>Adding another workflow is usually the cheap answer, until it isn't. Six signals that a GoHighLevel account has outgrown the builder, the honest case for staying in it, and the three levels of code to try before committing to a marketplace app.</p>
<p>The post <a href="https://john-nessime.com/blog/workflow-automation/gohighlevel-custom-development-vs-workflows/">Six Signs GoHighLevel Needs a Developer, Not Another Workflow</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The account had forty-one workflows. Two were named &#8220;Lead Follow-Up v3&#8221; and &#8220;Lead Follow-Up v3 (USE THIS ONE)&#8221;. When someone asked what happens to a contact who submits the form twice in one afternoon, nobody could answer. Not the agency owner, not the person who built it.</p>



<p class="wp-block-paragraph">That is the moment. Not a crash, not an error banner. A plain question about the system that nobody can answer from the canvas.</p>



<p class="wp-block-paragraph">GoHighLevel is good at what it was built for and most accounts never need anything else. But there is a point where the next workflow makes the account worse instead of better, and past it, <strong>GoHighLevel custom development</strong> costs less than another six months of patching. Below: the failure mode hiding under all the others, six signals you have crossed that line, the honest case for staying in the builder, and three levels of code to try before committing to a full app.</p>



<h2 class="wp-block-heading">The failure hiding under all the others</h2>



<p class="wp-block-paragraph">The workflow builder has no failure semantics. Everything below is a consequence of that.</p>



<p class="wp-block-paragraph">When a step works, the contact moves on. When a step fails, the contact usually also moves on. No transaction, no rollback, no retry budget you control, no way to say &#8220;this contact is in a bad state, stop and tell someone&#8221;. A Custom Webhook that gets a 500 back does not park the contact for review. It becomes a line in an execution log nobody reads, and the contact continues as though the other system had answered.</p>



<p class="wp-block-paragraph">Fine when the step sends a text message. Not fine when the step is the only thing keeping two systems in agreement. So the useful test is not &#8220;is this hard to build in the builder&#8221;, because plenty of hard things build fine. The test is: <em>what happens when this step fails, and would I find out?</em></p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Signal one: you are storing state in tags</h2>



<p class="wp-block-paragraph">Tags start as labels. Then someone needs to know whether a lead was called, so there is a <code>called</code> tag. Then whether they were called twice, so <code>called-2</code>. Then a workflow that removes <code>called</code> when a new form arrives, and another that puts it back if an appointment was booked first.</p>



<p class="wp-block-paragraph">Those are no longer labels. That is a state machine with no diagram, no validation, and no guarantee that only one workflow writes to it at a time. Two workflows fired by the same event will both read the old value and both write, and whichever finishes last wins. That is a race condition, and it hides during testing because you test one contact at a time. The tell is a tag name with a number in it, or a tag whose meaning needs the word &#8220;unless&#8221; to explain.</p>



<h2 class="wp-block-heading">Signal two: the data has to be right, not eventually right</h2>



<p class="wp-block-paragraph">Marketing automation tolerates drift. One extra nurture email hurts nobody. The moment GoHighLevel holds something that must reconcile with another system, that tolerance is gone. Order status from a store, invoice state from an accounting package, availability shared with a clinic system.</p>



<p class="wp-block-paragraph">Those need idempotency, meaning a repeated message produces the same result as a single one, and reconciliation, meaning something periodically compares both sides and reports the difference. The builder gives you neither: no natural home for an idempotency key, no scheduled job that reads both systems and complains. You can fake reconciliation with a workflow on a timer, but that is a batch job written in a UI designed for follow-up sequences, and the first time it half-runs you will not hear about it.</p>



<h2 class="wp-block-heading">Signal three: per-execution billing became a per-lead tax</h2>



<p class="wp-block-paragraph">The actions that make workflows powerful are premium actions, billed per execution against the agency wallet. Custom Webhook, Inbound Webhook, Custom Code and the workflow AI actions all sit in that category. HighLevel gives each sub-account a small free allowance when premium actions are first switched on, publishes volume tiers above that, and lets you rebill sub-accounts individually instead of absorbing the charge. Take the current rate and tiers from HighLevel&#8217;s billing documentation rather than any blog post, including this one.</p>



<p class="wp-block-paragraph">The mechanism is the point. You pay per execution, not per useful outcome. A workflow calling out three times to enrich, score and push a lead is three executions per lead. Multiply by volume and sub-accounts, and the plumbing line can pass the cost of a small service doing all three in one request.</p>



<p class="wp-block-paragraph">Two caveats. Consolidating three calls into one endpoint is usually cheaper than moving the whole thing to code. And whatever you move still costs something to run, whether that is a VPS from a provider like Contabo or InterServer, a hosted platform such as Make or n8n, or a serverless function. Compare both sides.</p>



<h2 class="wp-block-heading">Signal four: the same logic lives in twenty sub-accounts</h2>



<p class="wp-block-paragraph">Snapshots are the platform&#8217;s answer and better than people expect. You can refresh a snapshot and push selected assets to sub-accounts that previously imported it, choosing both the assets and the recipients, and the push is selective rather than destructive.</p>



<p class="wp-block-paragraph">Two limits. Push Update only reaches sub-accounts inside your own agency, so anything shared externally has to be re-imported through a new link. And HighLevel documents that when a snapshot refresh deletes steps, contacts waiting on those steps in workflows created from that snapshot are removed. Fine for a copy change, destructive for a structural one.</p>



<p class="wp-block-paragraph">Snapshots distribute structure. They do not give you versioning, a rollback path, a diff between intended and running state, or a test that fails before the change reaches a client. Once a change has to land correctly in twenty accounts on one day, you want those, and they come from putting the logic behind an API you control and calling it from a thin step in each account.</p>



<h2 class="wp-block-heading">Signal five: the integration must survive the other side being down</h2>



<p class="wp-block-paragraph">A Custom Webhook fires the moment the contact reaches the step. If your endpoint is slow, restarting or rate limited, that single firing is what you get. No queue in front, no backoff behind.</p>



<p class="wp-block-paragraph">The API side has numbers worth knowing. HighLevel documents a burst limit of 100 requests per 10 seconds and a daily limit of 200,000 requests, counted per marketplace app per resource, where a resource is a location or the company. Every response carries headers telling you where you stand, so backoff can follow what the server says instead of a guessed sleep.</p>



<pre class="wp-block-code"><code>curl -sS -D - -o /dev/null 
  -H "Authorization: Bearer $GHL_TOKEN" 
  -H "Version: $GHL_API_VERSION" 
  "$GHL_API_BASE/contacts/" 
  | grep -i '^x-ratelimit'</code></pre>



<p class="wp-block-paragraph"><code>-D -</code> writes response headers to stdout while <code>-o /dev/null</code> discards the body, so you read the rate limit state without dumping contact data into your shell history. Set the base URL and the required <code>Version</code> value from the current API reference. Watch <code>X-RateLimit-Max</code> and <code>X-RateLimit-Remaining</code> for the burst window, <code>X-RateLimit-Interval-Milliseconds</code> for its length, and <code>X-RateLimit-Limit-Daily</code> with <code>X-RateLimit-Daily-Remaining</code> for the day.</p>



<p class="wp-block-paragraph">The daily budget sounds enormous until a migration. Reading a contact, upserting it, then writing custom fields is three calls each, so forty thousand contacts is 120,000 calls before live traffic touches the account. Batch the reads, skip writes where nothing changed, run backfills overnight. Watch for loops too: writes made through the API can themselves fire workflow triggers, so a receiver that writes back has built a cycle.</p>



<h2 class="wp-block-heading">Signal six: someone needs a screen, not a form</h2>



<p class="wp-block-paragraph">Forms collect. They do not display. Once a client asks to see a filtered view of their own data, approve something in two clicks, or work a queue that is not a pipeline, you are looking at an interface.</p>



<p class="wp-block-paragraph">The platform supports this. Marketplace apps can add Custom Pages, which render inside HighLevel in an embedded iframe loading a URL you host, appearing on the app details page or in the left navigation. HighLevel publishes an app template to start from, plus an SSO mechanism so your page can identify the user instead of asking them to log in again. That is still a real application with hosting, auth and deployment: the biggest step on this list, the one I would defer longest, and the only honest answer when the requirement is a screen.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Where the workflow builder genuinely wins</h2>



<p class="wp-block-paragraph">A one-sided argument is not worth reading, so here is the case against writing code.</p>



<ul class="wp-block-list">
<li><strong>The person who owns the outcome can change it.</strong> An account manager adjusts send timing on a Friday with no deploy and no developer awake.</li>



<li><strong>Messaging and scheduling are solved.</strong> SMS, email, calendars, reminders and the delivery infrastructure behind them are the platform&#8217;s core. Rebuilding any of it is a bad trade.</li>



<li><strong>Timezone handling is built in.</strong> Time-based steps run in the account timezone or the contact&#8217;s own, falling back to the account when a contact has none.</li>



<li><strong>No pipeline to maintain.</strong> No repository, no secrets rotation, no dependency upgrades, no server needing patched at an awkward hour.</li>



<li><strong>Execution history is right there.</strong> For a linear sequence, reading what happened to a contact beats correlating logs across two systems.</li>
</ul>



<p class="wp-block-paragraph">If the automation is a follow-up sequence with branching, the builder is correct and code is the expensive mistake.</p>



<h2 class="wp-block-heading">Three levels before you build an app</h2>



<p class="wp-block-paragraph">&#8220;Needs a developer&#8221; is not one decision. It is three, and most accounts only need the first.</p>



<h3 class="wp-block-heading">Level one: the Custom Code action</h3>



<p class="wp-block-paragraph">The builder has a Custom Code premium action that runs JavaScript inside the workflow. Values from earlier steps are mapped in as named input properties and arrive in an <code>InputData</code> object. The action returns a JavaScript object or an array of objects for later steps to reference, and <code>console.log</code> output is captured so you can see what ran.</p>



<pre class="wp-block-code"><code>// Normalise a phone number before it reaches a branch step.
// rawPhone and country are input properties mapped from earlier steps.
const digits = String(InputData.rawPhone || "").replace(/D/g, "");
const cc = InputData.country === "US" ? "1" : "";

if (!digits) {
  console.log("no phone on contact, routing to manual branch");
  return { e164: "", usable: false };
}

return {
  e164: "+" + (digits.startsWith(cc) ? digits : cc + digits),
  usable: digits.length &gt;= 10
};</code></pre>



<p class="wp-block-paragraph">This kills a lot of branch sprawl. Ten If/Else steps existing only to reshape a value collapse into one action returning a clean value and a boolean. Still inside the platform, still billed per execution, still no retries. But logic smeared across a canvas now reads in one place.</p>



<h3 class="wp-block-heading">Level two: a private integration and a script you own</h3>



<p class="wp-block-paragraph">For one account you do not need a marketplace app. Create a Private Integration inside the sub-account, select only the scopes the job needs, and use the token as a bearer credential. No OAuth handshake, no refresh cycle, no listing.</p>



<p class="wp-block-paragraph">This is where reconciliation jobs, nightly audits, migrations and bulk cleanups belong. Run it as a cron job on a small VPS, behind a Cloudflare Tunnel if it needs an inbound endpoint, with the token in a secrets store rather than a config file. Treat that token as a credential with a rotation plan, not a string you paste once and forget.</p>



<h3 class="wp-block-heading">Level three: a marketplace app</h3>



<p class="wp-block-paragraph">You need this when the thing spans many accounts, has a user interface, or is a product you intend to sell. It means OAuth 2.0 on the authorization code grant, token storage and refresh per installed location, webhook handling, and somewhere to run it. Note the version boundary: the v1 API is end of life and unmaintained, and new builds target the OAuth-based v2. If a tutorial hands you a permanent location API key, it is describing the old world.</p>



<p class="wp-block-paragraph">Marketplace apps can also register their own workflow triggers and actions, which appear in the builder for sub-accounts that installed the app. That is the pattern I reach for first: the hard logic lives in your service, and the person who owns the campaign still drags a step onto a canvas. It requires the workflows read scope, and those custom actions are themselves premium actions billed per execution.</p>



<h2 class="wp-block-heading">Arguments that do not survive contact</h2>



<ul class="wp-block-list">
<li><strong>&#8220;We will just add one more workflow.&#8221;</strong> The cost is not building it. It is that every future workflow now has to be reasoned about alongside it. Ten workflows listening for one tag is not ten problems, it is the interactions between them.</li>



<li><strong>&#8220;No-code means no developer.&#8221;</strong> It means no deployment. Someone still holds the whole model in their head, and when they leave, an undocumented canvas is harder to inherit than a repository with a README.</li>



<li><strong>&#8220;Zapier or Make will handle it.&#8221;</strong> Often true and worth trying first. What they do not give you is one place where business logic lives. Split rules across three tools and debugging means three log formats.</li>



<li><strong>&#8220;It works, so it is fine.&#8221;</strong> Every silent failure worked right up until someone counted. Absence of errors in a system with no error reporting is not evidence.</li>



<li><strong>&#8220;Custom code is lock-in.&#8221;</strong> Backwards. Logic on a canvas is bound to one vendor&#8217;s UI and exports only as a snapshot. Logic in a repository moves.</li>
</ul>



<h2 class="wp-block-heading">How I would decide</h2>



<ol class="wp-block-list">
<li><strong>Write down what happens when each external call fails.</strong> If the answer anywhere is &#8220;the contact continues as if it succeeded&#8221; and that matters, you have found your requirement.</li>



<li><strong>Count the workflows writing to the same tag or field.</strong> More than two means you have a state machine. Draw it before touching anything.</li>



<li><strong>Count premium executions per lead, then multiply by monthly volume.</strong> Compare against a few hours a month maintaining a small service. Sometimes the builder still wins. Do the arithmetic instead of assuming.</li>



<li><strong>Ask how a change reaches every account.</strong> If it involves opening sub-accounts one at a time, that is the real cost, and it grows with every client.</li>



<li><strong>Try the smallest level that fits, and leave the trigger in the builder.</strong> Custom Code action, then private integration script, then marketplace app. Whatever moves out, keep the entry point as a workflow step so the marketing side can still change timing and audience without you.</li>
</ol>



<p class="wp-block-paragraph">Whatever you build, monitor it. An uptime check on your receiver and an alert when the reconciliation job reports a non-zero difference will catch more real problems than any amount of careful workflow naming.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need a marketplace app to use the GoHighLevel API?</h3>



<p class="wp-block-paragraph">No. For a single sub-account, create a Private Integration inside that account, grant only the scopes you need, and use the token as a bearer credential. Marketplace apps and OAuth are for anything spanning accounts or distributed to other people.</p>



<h3 class="wp-block-heading">How much automation should stay in workflows?</h3>



<p class="wp-block-paragraph">The parts a non-developer should be able to change: timing, audience, message copy, branching on obvious contact attributes. Move out the parts where being wrong is expensive and being wrong is silent.</p>



<h3 class="wp-block-heading">Why does my workflow show as successful when the integration clearly failed?</h3>



<p class="wp-block-paragraph">A step firing and a step achieving something are different events, and the builder records the first. A webhook returning an error still counts as having run. If the outcome matters, verify it explicitly rather than trusting the log.</p>



<h3 class="wp-block-heading">Should I avoid premium workflow actions on cost grounds?</h3>



<p class="wp-block-paragraph">Not on principle. They cost per execution because they are the expensive verbs. The problem is fragmentation: three separate calls where one would do triples the bill for the same outcome. Consolidate before considering a move off the platform.</p>



<h3 class="wp-block-heading">What does a GoHighLevel custom development project usually start with?</h3>



<p class="wp-block-paragraph">An audit, not a build. Map which workflows write to which fields, where external calls happen, and what the failure path is for each. Often that document alone removes the need for the project.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Ask what happens when a step fails. That question separates automation belonging in the builder from automation needing code, more reliably than counting steps, counting workflows, or judging how clever the branching has become.</p>



<p class="wp-block-paragraph">If failure is visible and cheap, stay in the builder and enjoy that your client can edit it themselves. If failure is silent and expensive, GoHighLevel custom development stops being an upgrade and becomes maintenance you already deferred. Then start small: a Custom Code action beats a marketplace app you will not want to maintain.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Working on this with me</h2>



<p class="wp-block-paragraph">I work on the engineering side of GoHighLevel accounts, usually the parts failing quietly rather than the parts that look broken.</p>



<ul class="wp-block-list">
<li>Auditing an account for silent failures: which workflows write to which tags and fields, and what happens on every external call that returns an error.</li>



<li>Replacing branch sprawl with Custom Code actions, so logic reads as one block instead of fifteen steps on a canvas.</li>



<li>Webhook receivers and sync services that retry, deduplicate and reconcile, with the rate limit headers driving backoff instead of a fixed sleep.</li>



<li>Migrations and backfills planned against the daily API budget, batched and run in the account&#8217;s timezone so live traffic keeps working.</li>



<li>Private integration setup with least-privilege scopes, token storage and a rotation plan that is written down.</li>



<li>Marketplace app work when genuinely warranted: OAuth handling, custom workflow actions, and Custom Pages hosted somewhere you control.</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion, send something concrete: a screenshot of the workflow you are least sure about, an execution log entry that makes no sense, or the response body your webhook endpoint is returning. That is a faster conversation than a scoping call.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/workflow-automation/gohighlevel-custom-development-vs-workflows/">Six Signs GoHighLevel Needs a Developer, Not Another Workflow</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Build a CRM Knowledge Assistant on Amazon Bedrock Without Leaking Salesforce Records</title>
		<link>https://john-nessime.com/blog/devops/salesforce-crm-knowledge-assistant-bedrock/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 23 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[AgentCore]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon Kendra]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Bedrock Knowledge Bases]]></category>
		<category><![CDATA[Connected App]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Embeddings]]></category>
		<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Metadata Filtering]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[OpenSearch Serverless]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[Row-Level Security]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Vector Database]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=243</guid>

					<description><![CDATA[<p>Connecting Salesforce to Amazon Bedrock Knowledge Bases takes an afternoon. Making sure the assistant does not read people records they were never allowed to see takes longer. A practical walkthrough of the connector, the permission model it quietly drops, and the two architectures that actually hold up.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-crm-knowledge-assistant-bedrock/">Build a CRM Knowledge Assistant on Amazon Bedrock Without Leaking Salesforce Records</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A support lead types a question into the assistant you demoed last week: &#8220;What&#8217;s the renewal value on the Northwind account?&#8221; It answers in about two seconds, correctly, with a citation back to the source record. Everyone in the room is pleased.</p>



<p class="wp-block-paragraph">Then someone points out that this particular support lead has never had access to Opportunity records. Not through her profile, not through the role hierarchy, not through a sharing rule. And the assistant just read one out to her.</p>



<p class="wp-block-paragraph">That is the failure that kills most CRM assistant projects, and it almost never shows up in testing, because you test as an admin. Everything you ask works. Everything you ask is supposed to work. The gap only opens when a real user with a real, narrow profile starts asking questions.</p>



<p class="wp-block-paragraph">This post walks through building a CRM knowledge assistant on Salesforce and Amazon Bedrock Knowledge Bases: which of the two knowledge base paths you are actually allowed to use, how the Salesforce connector authenticates, what it does and does not carry across from your CRM, and the two architectures that survive a real permission model. The wiring is the easy part. The scoping is the work.</p>



<h2 class="wp-block-heading">The three moving parts</h2>



<p class="wp-block-paragraph">Strip away the marketing and a CRM knowledge assistant is three things bolted together:</p>



<ul class="wp-block-list">
<li><strong>Ingestion</strong> — something pulls Cases, Knowledge articles, Opportunities and Accounts out of Salesforce on a schedule, chunks them, embeds them, and writes the vectors somewhere.</li>

<li><strong>Retrieval</strong> — a query comes in, gets embedded, and the closest chunks come back. This is where filtering lives, and it is the security surface.</li>

<li><strong>Generation</strong> — a foundation model reads those chunks and writes an answer with citations.</li>
</ul>



<p class="wp-block-paragraph">Generation is the part everyone stares at and the part you should worry about least. It is a model choice and a prompt. Retrieval is where the assistant becomes either trustworthy or a liability.</p>



<h2 class="wp-block-heading">Decision one: which knowledge base you are allowed to use</h2>



<p class="wp-block-paragraph">Bedrock now has two shapes of knowledge base, and this decision gets made for you the moment you say &#8220;Salesforce&#8221;.</p>



<p class="wp-block-paragraph"><strong>Bedrock Managed Knowledge Base</strong> is the newer, opinionated one. AWS manages the vector store, picks the embedding and reranking models by default, handles parsing, and exposes agentic multi-hop retrieval. It ships with a small set of first-party connectors: Amazon S3, SharePoint, Confluence, Google Drive, OneDrive, Web Crawler, and a custom ingestion path. Salesforce is not on that list.</p>



<p class="wp-block-paragraph"><strong>Customer-managed knowledge bases</strong> are the older shape. You bring the vector store, you pick the embedding model, you own more of the pipeline. This is where the Salesforce connector lives, and AWS still flags that connector as a preview release subject to change. It also has a hard constraint worth reading twice: with a Salesforce data source, Amazon OpenSearch Serverless is currently the only supported vector store. If your organisation has standardised on Aurora pgvector or something else, that standard does not apply here.</p>



<p class="wp-block-paragraph">So the fork is real. Either you use the preview Salesforce connector on the customer-managed path, or you land Salesforce data in S3 yourself and use the managed path. I will come back to why the second option is often the right call, because it is not just a workaround.</p>



<p class="wp-block-paragraph">One piece of context that matters if you are inheriting an older design: Amazon Kendra entered maintenance mode on 30 June 2026 and stopped accepting new customers on 30 July 2026. Kendra was the standard answer to &#8220;search my Salesforce org&#8221; for years, and AWS now points people at Bedrock Knowledge Bases instead. Existing indexes keep running and no end-of-service date has been set, but Kendra will not gain features. If a proposal on your desk starts with a Kendra index, that is worth raising before anyone writes code.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The permission model is the whole problem</h2>



<p class="wp-block-paragraph">Salesforce spends enormous effort on who can see what. Profiles, permission sets, the role hierarchy, org-wide defaults, criteria-based sharing rules, field-level security, restriction rules. A given Opportunity is visible to a specific, computed set of users, and that computation happens on every query.</p>



<p class="wp-block-paragraph">None of that survives ingestion.</p>



<p class="wp-block-paragraph">The AWS documentation is blunt about it, and this is the single most important sentence in the whole feature: everything you sync from your data source becomes available to anyone holding <code>bedrock:Retrieve</code> on that knowledge base, including content that had controlled permissions at the source. Three layers of Salesforce scoping collapse into one IAM action.</p>



<p class="wp-block-paragraph">Worse, the connector authenticates as a single identity. You configure a Connected App using the OAuth 2.0 client credentials flow, and that flow runs as one named user. Whatever that user can see, the crawler ingests. Pick a System Administrator to &#8220;make sure the sync works&#8221; and you have just copied the entire org into a flat index that your whole application tier can read.</p>



<h3 class="wp-block-heading">Why this is invisible until it isn&#8217;t</h3>



<p class="wp-block-paragraph">Nothing errors. There is no warning in the console, no failed sync, no denied API call. Retrieval returns chunks with good similarity scores and the model writes a fluent, cited answer. The only signal that something is wrong is a human recognising a record they should not be looking at, and that human is usually a customer-facing employee, not you.</p>



<p class="wp-block-paragraph">Treat the knowledge base as a published dataset, not as a view over Salesforce. Ask the question you would ask about an S3 bucket: if every row of this landed in front of every user of the app, who gets hurt?</p>



<h3 class="wp-block-heading">The two architectures that actually hold up</h3>



<p class="wp-block-paragraph"><strong>Option A: ingest only what everyone is allowed to see.</strong> Run the Connected App as a deliberately low-privilege integration user whose profile grants read access to Knowledge articles, published solutions, product data and price books, and nothing else. Exclude Contact, Lead, Case and Opportunity entirely. You get a genuinely useful support and enablement assistant, the permission question disappears, and you can ship it without a security review turning into a project.</p>



<p class="wp-block-paragraph">This is the one I reach for first. Most of the value people want from a CRM knowledge assistant is &#8220;what do we know about this product, this policy, this recurring problem&#8221;, and none of that needs record-level data.</p>



<p class="wp-block-paragraph"><strong>Option B: export to S3 with your own access metadata.</strong> If you genuinely need per-record answers, stop using the connector. Pull the objects you need through the Salesforce API or a scheduled extract, write each record to S3 as a document, and write a sidecar metadata file next to it carrying the access attribute you will filter on. Then point a knowledge base at the bucket and apply an explicit metadata filter on every retrieval, derived from the caller&#8217;s identity rather than from the query text.</p>



<p class="wp-block-paragraph">The cost is honest: you now own an extraction pipeline, a metadata schema, and the job of keeping the access attribute correct when a record changes owner. That is real work. The benefit is that you can point at exactly where the boundary is enforced and test it.</p>



<p class="wp-block-paragraph">What you cannot do is enforce the boundary in the prompt. &#8220;Only answer about accounts the user owns&#8221; is a suggestion to a language model, not an access control. Retrieval already happened by then.</p>



<h2 class="wp-block-heading">Wiring the Salesforce connector</h2>



<p class="wp-block-paragraph">Assuming you have chosen Option A and want the connector, here is the shape of it.</p>



<h3 class="wp-block-heading">On the Salesforce side</h3>



<ol class="wp-block-list">
<li>Create a Connected App in your org and enable the OAuth 2.0 client credentials flow.</li>

<li>Under the client credentials flow settings, set the <strong>Run As</strong> field to your integration user. This is the step people skip, and the sync fails without it. It is also the step that decides your entire blast radius, so pick that user before you pick anything else.</li>

<li>Copy the consumer key and consumer secret from the app&#8217;s OAuth settings.</li>

<li>Note your instance URL.</li>
</ol>



<h3 class="wp-block-heading">On the AWS side</h3>



<p class="wp-block-paragraph">Store the credentials in AWS Secrets Manager. The connector looks for three specific keys, and it will not fall back to alternatives if you name them differently:</p>



<pre class="wp-block-code"><code>{
  "consumerKey": "your Connected App client ID",
  "consumerSecret": "your Connected App client secret",
  "authenticationUrl": "https://yourcompany.my.salesforce.com/"
}</code></pre>



<p class="wp-block-paragraph">The secret has to live in the same AWS region as the knowledge base. Cross-region reads are not supported here, and the resulting error is generic enough to waste an hour. Do not reuse this secret for any other data source; rotate it on the same schedule as your other integration credentials.</p>



<h3 class="wp-block-heading">Creating the data source</h3>



<p class="wp-block-paragraph">The data source configuration is a JSON document. The interesting part is <code>patternObjectFilter</code>, which is how you keep the crawler away from objects you never wanted:</p>



<pre class="wp-block-code"><code>{
  "salesforceConfiguration": {
    "sourceConfiguration": {
      "hostUrl": "https://yourcompany.my.salesforce.com/",
      "authType": "OAUTH2_CLIENT_CREDENTIALS",
      "credentialsSecretArn": "arn:aws:secretsmanager:REGION:ACCOUNT:secret:NAME"
    },
    "crawlerConfiguration": {
      "filterConfiguration": {
        "type": "PATTERN",
        "patternObjectFilter": {
          "filters": [
            {
              "objectType": "Knowledge__kav",
              "inclusionFilters": [".*"]
            },
            {
              "objectType": "Case",
              "exclusionFilters": [".*"]
            }
          ]
        }
      }
    }
  },
  "type": "SALESFORCE"
}</code></pre>



<p class="wp-block-paragraph">Two things to internalise about these filters. First, they are regular expressions matched against a per-object identifier, not against record contents: for <code>Knowledge__kav</code> that is the article title, for <code>Case</code> the case number, for <code>Attachment</code> the file name. Second, when an inclusion and an exclusion filter both match the same document, exclusion wins. That precedence is what makes an allow-list pattern safe to write.</p>



<p class="wp-block-paragraph">Then create the data source and attach it to an existing knowledge base:</p>



<pre class="wp-block-code"><code>aws bedrock create-data-source 
  --name "salesforce-knowledge" 
  --knowledge-base-id "YOUR_KB_ID" 
  --data-source-configuration file://salesforce-connector-config.json 
  --data-deletion-policy "DELETE" 
  --vector-ingestion-configuration '{"chunkingConfiguration":[{"chunkingStrategy":"FIXED_SIZE","fixedSizeChunkingConfiguration":[{"maxTokens":"300","overlapPercentage":"10"}]}]}'</code></pre>



<p class="wp-block-paragraph">A note on the command namespace: the agent-plane operations have historically lived under a separate CLI command group from the model-inference ones, and this has shifted between CLI versions. Run <code>aws bedrock help</code> against your installed version before assuming. The JSON payload is the stable part.</p>



<p class="wp-block-paragraph"><code>--data-deletion-policy DELETE</code> means the vectors get removed from the store when you delete the data source. The alternative retains them. Choose <code>DELETE</code> unless you have a specific reason not to, because orphaned vectors in an OpenSearch Serverless collection are both a cost line and a surprise waiting for whoever inherits this.</p>



<p class="wp-block-paragraph">After that, kick off a sync with <code>StartIngestionJob</code>, or hit <strong>Sync</strong> in the console. The connector does incremental syncs after the first run: added, updated and deleted content only. The first run crawls everything.</p>



<h2 class="wp-block-heading">Chunking is a one-way door</h2>



<p class="wp-block-paragraph">You cannot change the chunking strategy after connecting the data source. Not tune it, not swap it. You delete the data source and start over, which means re-crawling and re-embedding everything.</p>



<p class="wp-block-paragraph">So spend twenty minutes here rather than five. CRM content is not uniform. Knowledge articles are structured prose and behave like documentation. Case comment threads are short, conversational, and lose their meaning when split mid-exchange. Opportunity descriptions are often two sentences. A fixed-size chunk that suits one of these will mangle another.</p>



<p class="wp-block-paragraph">The practical move is to stop trying to find one strategy that fits everything. Create separate data sources for materially different content shapes, each with its own chunking configuration, all pointing at the same knowledge base. It costs you a bit of configuration and saves you a full re-index later.</p>



<p class="wp-block-paragraph">One more constraint: the Salesforce connector does not handle multimodal content. Tables, charts and diagrams embedded in your Knowledge articles are not going to be understood. If your best troubleshooting articles are mostly screenshots, this connector will quietly ingest the surrounding text and nothing else, and your retrieval quality will be worse than the article count suggests.</p>



<h2 class="wp-block-heading">Retrieval, filtering, and one trap</h2>



<p class="wp-block-paragraph">Bedrock gives you two kinds of metadata filter at query time, and confusing them is a security bug.</p>



<p class="wp-block-paragraph"><strong>Explicit filters</strong> are the ones you construct in code and pass in the retrieval configuration. Your application decides them from the authenticated caller. They are deterministic.</p>



<pre class="wp-block-code"><code>{
  "retrievalQuery": { "text": "renewal value on the Northwind account" },
  "retrievalConfiguration": {
    "vectorSearchConfiguration": {
      "numberOfResults": 8,
      "filter": {
        "andAll": [
          { "equals": { "key": "visibility_group", "value": "emea-enterprise" } },
          { "equals": { "key": "record_class", "value": "internal" } }
        ]
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph"><strong>Implicit filters</strong> are the other kind. You declare which metadata attributes exist and Bedrock uses a model to infer filters from the natural-language query. Someone asks about last quarter, it infers a date filter. This is a genuinely nice relevance feature and it is not an access control. A model inferring a filter from user-supplied text is a model that can be talked out of it. Use implicit filters to improve answers; use explicit filters to enforce boundaries.</p>



<p class="wp-block-paragraph">The trap: filters on the Salesforce connector operate on auto-detected fields from your CRM records. You cannot inject your own <code>visibility_group</code> attribute at ingestion. That capability comes from sidecar metadata files, which is an S3 data source feature. This is the concrete reason Option B routes through a bucket. If per-user scoping is a requirement, the connector cannot give it to you no matter how carefully you configure it.</p>



<h2 class="wp-block-heading">Where the money goes</h2>



<p class="wp-block-paragraph">Rates change, so learn the meters instead of memorising numbers.</p>



<ul class="wp-block-list">
<li><strong>Embedding at ingestion</strong> — billed per token processed. A full re-crawl re-embeds everything, which is exactly why the chunking decision is expensive to reverse.</li>

<li><strong>Vector storage</strong> — on the customer-managed path this is your OpenSearch Serverless collection, billed on provisioned compute units and storage, with a floor whether or not anyone queries it. This is the line item that surprises people on a proof of concept that nobody uses. Managed Knowledge Base bills on indexed data size and retrieval count instead, with no idle collection sitting underneath.</li>

<li><strong>Generation</strong> — input and output tokens on whichever foundation model you chose. Retrieved chunks are input tokens, so <code>numberOfResults</code> is a direct cost lever and worth tuning down until quality actually degrades.</li>

<li><strong>Everything around it</strong> — the API layer, the chat frontend, and the observability stack. A small VPS from a provider like InterServer or a Lambda behind API Gateway both work; the choice barely matters next to the Bedrock line. Push retrieval latency and citation counts into Grafana Cloud or Datadog early, because &#8220;the assistant feels worse this week&#8221; is otherwise unfalsifiable.</li>
</ul>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Sync fails immediately with an authentication error.</strong> Check the Run As user on the Connected App&#8217;s client credentials flow first. An empty Run As field is the most common cause. Then check that the secret keys are spelled exactly <code>consumerKey</code>, <code>consumerSecret</code>, <code>authenticationUrl</code>, and that the secret is in the knowledge base&#8217;s region.</li>

<li><strong>Sync succeeds but the document count is far lower than expected.</strong> Usually the integration user&#8217;s profile, not the filters. Query the object as that user in Salesforce and count. Also check the per-knowledge-base file count and file size quotas, which are separate limits and are silently enforced.</li>

<li><strong>Retrieval returns nothing for queries you know are covered.</strong> Test with no filter at all. If results appear, your filter keys do not match the auto-detected field names. If they do not, the content probably never made it past chunking; check whether those articles are mostly images.</li>

<li><strong>Deleted Salesforce records still turn up in answers.</strong> Incremental sync handles deletions, but only on the next sync. If you deleted something sensitive, run a sync immediately rather than waiting for the schedule.</li>

<li><strong>Answers are confidently wrong about numbers.</strong> Fixed-size chunking split a record away from its label. Retrieve the raw chunks for that query and read them. Nine times out of ten the problem is visible in the chunk, not in the model.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Running the Connected App as a System Administrator &#8220;just to get the sync working&#8221;. That decision is almost never revisited.</li>

<li>Assuming sharing rules and field-level security follow the data into the index. They do not.</li>

<li>Enforcing access in the system prompt. Retrieval has already happened by the time the model reads your instructions.</li>

<li>Ingesting Contact and Lead because they were there. Both carry personal data and both are usually irrelevant to the questions people actually ask.</li>

<li>Accepting default chunking, then discovering it is immutable.</li>

<li>Building on a preview connector without telling stakeholders it is preview. The configuration surface can change under you.</li>

<li>Testing only as an admin. Every acceptance test should run as the narrowest real profile you have.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Decide the integration user&#8217;s profile before you touch AWS. It defines the security boundary more than any IAM policy will.</li>

<li>Start with Knowledge articles only. Ship that, get real queries, then argue about whether Opportunity data is worth the architecture it requires.</li>

<li>Write filters as allow-lists. Exclusion beats inclusion, so an allow-list fails closed.</li>

<li>Split content shapes across separate data sources so chunking suits each one.</li>

<li>Keep a fixed evaluation set of about fifty real questions with known-good answers, and re-run it after every configuration change.</li>

<li>Log the retrieved chunk IDs alongside every generated answer. When someone reports a bad answer months from now, that log is the only thing that makes it debuggable.</li>

<li>Treat <code>bedrock:Retrieve</code> as a privileged permission and scope it to the specific knowledge base ARN, not to a wildcard.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does the Bedrock Salesforce connector respect sharing rules?</h3>



<p class="wp-block-paragraph">No. The crawler authenticates as one user and everything it can read becomes retrievable by anyone with <code>bedrock:Retrieve</code> on the knowledge base. Per-user scoping has to be built at the retrieval layer with explicit metadata filters, which in practice means routing through S3 rather than using the connector.</p>



<h3 class="wp-block-heading">Can I use Aurora or pgvector instead of OpenSearch Serverless?</h3>



<p class="wp-block-paragraph">Not with the Salesforce data source. AWS currently restricts that connector to Amazon OpenSearch Serverless. If your vector store is non-negotiable, extract to S3 and use a data source that supports your chosen store.</p>



<h3 class="wp-block-heading">Should I use Agentforce instead of building this on Bedrock?</h3>



<p class="wp-block-paragraph">If your questions are answerable entirely from Salesforce data and your users already live in Salesforce, Agentforce is the shorter path and it operates inside the platform&#8217;s existing permission model, which removes the hardest problem in this post. Bedrock earns its complexity when the assistant has to reason across CRM data plus sources that are not in Salesforce, or when you need a specific foundation model, or when the data must stay inside your AWS account. Cost profiles differ substantially and are worth modelling on your real query volume before committing.</p>



<h3 class="wp-block-heading">How current will the assistant&#8217;s answers be?</h3>



<p class="wp-block-paragraph">As current as your last sync. Ingestion is scheduled, not live. For rapidly changing fields such as case status or pipeline stage, do not answer from the index at all: have the assistant call the Salesforce API for the live value and use the knowledge base only for the surrounding context.</p>



<h3 class="wp-block-heading">What happens to the connector when it leaves preview?</h3>



<p class="wp-block-paragraph">Unknown, which is the point of the label. Preview means the configuration surface, supported objects and behaviour can change without the compatibility guarantees you would expect from a GA service. Keep your data source configuration in version control so a re-create is a rerun rather than an archaeology exercise.</p>



<h3 class="wp-block-heading">Is a CRM knowledge assistant worth building if we already have Salesforce search?</h3>



<p class="wp-block-paragraph">It depends on the question shape. Search is better when people know what record they want. A CRM knowledge assistant wins on synthesis: &#8220;what usually causes this error for enterprise customers&#8221; pulls from twenty articles and cases at once, which no search result page does well. If your users mostly navigate to known records, you will not get much return.</p>



<h3 class="wp-block-heading">Do I need Bedrock AgentCore for this?</h3>



<p class="wp-block-paragraph">Not for retrieval and generation. AgentCore matters when you want the assistant to take actions, chain tools, or be exposed to agent frameworks over MCP. A read-only question-answering assistant works fine against the knowledge base APIs directly, and skipping AgentCore removes a layer you would otherwise have to operate.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">Connecting Salesforce to Amazon Bedrock takes an afternoon. The connector works, the syncs run, the answers are good. That speed is exactly what makes this dangerous, because nothing in the setup path forces you to think about who the answers are for.</p>



<p class="wp-block-paragraph">A CRM knowledge assistant is a publishing decision wearing an engineering costume. Every record you ingest, you are publishing to everyone who can call retrieve. Decide what that audience is allowed to see, encode it in the integration user&#8217;s profile and in your inclusion filters, and test every question as your narrowest real user. Do that first and the rest of the build is routine.</p>



<h2 class="wp-block-heading">Need help getting this into production?</h2>



<p class="wp-block-paragraph">I work with teams building retrieval systems on AWS, usually at the point where a working prototype has to survive a security review. Things I can help with on a CRM knowledge assistant:</p>



<ul class="wp-block-list">
<li>Reviewing an existing Bedrock knowledge base for records that should never have been ingested, and scoping the integration user&#8217;s Salesforce profile to match.</li>

<li>Designing the S3 extraction path with sidecar access metadata when per-user filtering is a hard requirement.</li>

<li>Choosing between the managed and customer-managed knowledge base paths, with the OpenSearch Serverless cost floor modelled against your actual query volume.</li>

<li>Building the chunking and data source split before it becomes an immutable mistake, plus a repeatable evaluation set to measure changes against.</li>

<li>Terraform or OpenTofu for the whole stack: knowledge base, data sources, Secrets Manager, IAM, and the vector store.</li>

<li>Retrieval observability so you can tell whether answer quality moved, and why.</li>

<li>Migration planning for teams currently on a Kendra index.</li>
</ul>



<p class="wp-block-paragraph">If you have a data source configuration, a failing sync, or a retrieval result that looks wrong, send it over and we can work out what it is doing.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-crm-knowledge-assistant-bedrock/">Build a CRM Knowledge Assistant on Amazon Bedrock Without Leaking Salesforce Records</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Salesforce AWS Integration Architecture: The Four Paths and What Breaks Silently</title>
		<link>https://john-nessime.com/blog/devops/salesforce-aws-integration-architecture/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 19 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AppFlow]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Change Data Capture]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Event Relay]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[Named Credentials]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Platform Events]]></category>
		<category><![CDATA[PrivateLink]]></category>
		<category><![CDATA[Pub/Sub API]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[Serverless]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=209</guid>

					<description><![CDATA[<p>Most Salesforce AWS integrations don't fail loudly, they drift. This guide covers the four integration paths between Salesforce and AWS, when to use each, the silent failure mode hiding in every one of them, and the reconciliation job you have to build yourself.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-aws-integration-architecture/">Salesforce AWS Integration Architecture: The Four Paths and What Breaks Silently</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone in the ops channel asks why the opportunity count on the QuickSight dashboard doesn&#8217;t match the count in the Salesforce report. It&#8217;s off by a few hundred. Nobody can answer that, and worse, nobody can say when the two numbers stopped agreeing. Every pipeline is green. The scheduled flow reports successful runs. The EventBridge rule shows invocations. Everything is healthy except the data.</p>



<p class="wp-block-paragraph">That&#8217;s the shape of most Salesforce AWS integration problems. They&#8217;re rarely outages. They&#8217;re divergence, accumulating quietly while every status page stays green, and you find out about it when someone in finance runs a number past a customer.</p>



<p class="wp-block-paragraph">This guide covers the four integration paths between Salesforce and AWS, which one to reach for and when, the specific failure each one hides, and the reconciliation layer you&#8217;ll have to build yourself because neither vendor ships it.</p>



<h2 class="wp-block-heading">Why a Salesforce AWS integration fails silently</h2>



<p class="wp-block-paragraph">Start here, because it changes how you design everything downstream. There are three mechanisms that lose data without raising an error anywhere you&#8217;re looking.</p>



<p class="wp-block-paragraph"><strong>Events sent to an unassociated partner event bus are dropped on the floor.</strong> When you create an Event Relay, Salesforce creates a partner event source in your AWS account. Until you associate that source with an event bus, EventBridge discards anything the partner publishes to it. AWS is explicit that those events are not persisted at rest. There&#8217;s no queue holding them for you, no backlog to drain once you finish the setup. If the relay is running and the bus isn&#8217;t associated, you are streaming into a void.</p>



<p class="wp-block-paragraph"><strong>The Salesforce event bus retains high-volume events for 72 hours.</strong> Platform events and change data capture events sit on the event bus for three days, and each one carries a replay ID so a durable subscriber can resume from where it left off. Past that window, they&#8217;re purged. Salesforce doesn&#8217;t guarantee availability beyond it. So a relay that goes into an error state on a Friday afternoon is a data loss event by Monday morning, and the only symptom during those three days is an absence of messages, which looks identical to a quiet weekend.</p>



<p class="wp-block-paragraph"><strong>Deletes don&#8217;t show up in query-based syncs.</strong> A scheduled pull selects records that exist. Records that stopped existing between runs are simply not in the result set, so your S3 prefix or Redshift table keeps a row that Salesforce no longer has. Nothing errors. The row count on the AWS side just creeps upward relative to the CRM, forever, until someone notices.</p>



<p class="wp-block-paragraph">Every architecture decision below is really a decision about which of these three you&#8217;re accepting and how you&#8217;re going to detect it.</p>



<h2 class="wp-block-heading">A Salesforce AWS integration starts with a direction, not a service</h2>



<p class="wp-block-paragraph">The most common mistake I see is picking a service first, usually whichever one appeared in a conference demo, and then bending the requirement to fit it. Two questions decide the path, and they take about a minute to answer.</p>



<ol class="wp-block-list">
<li><strong>Which way does the data move?</strong> Salesforce to AWS, or AWS to Salesforce. If the answer is &#8220;both&#8221;, you have two integrations, not one, and they will have different failure modes and different owners.</li>

<li><strong>What shape is the workload?</strong> Batch, where you want a consistent set of records on a schedule, or event-driven, where you want a reaction within seconds of a change.</li>
</ol>



<p class="wp-block-paragraph">That gives you four cells, and each cell has a default answer.</p>



<h3 class="wp-block-heading">Salesforce to AWS, batch: Amazon AppFlow</h3>



<p class="wp-block-paragraph">Amazon AppFlow is the managed answer for pulling Salesforce objects into S3, Redshift, or the Glue Data Catalog on a schedule or on demand. It handles the OAuth connection, field mapping, and incremental pulls without you writing or hosting anything. For analytics loads it&#8217;s usually the right default, and it&#8217;s the path of least resistance if the destination is a data lake you&#8217;re already cataloguing with Glue and querying with Athena.</p>



<p class="wp-block-paragraph">The trade-off is that it&#8217;s a polling client, and it spends your Salesforce API allocation to do it. AWS documents this plainly with an example: an hourly flow pulling five pages of data makes 120 API calls a day, and all of them count against the org&#8217;s 24-hour API request limit. Multiply that by every object you&#8217;re syncing and every environment you&#8217;re syncing from, and AppFlow can become the largest single consumer of an allocation that Apex callouts, integration users, and third-party tools are also drawing on.</p>



<p class="wp-block-paragraph">The failure mode that catches people: if a field referenced in the flow&#8217;s mapping is deleted in Salesforce, the flow run fails. Someone cleans up an unused custom field on a Tuesday and your nightly load breaks on Wednesday, in a service that nobody on the Salesforce side knows exists. AppFlow will automatically pick up newly created Salesforce fields when the destination is S3, but the reverse case, a removed field, needs a human to edit the mapping. Put the flow names somewhere the Salesforce admins will see them before they run a field cleanup.</p>



<h3 class="wp-block-heading">Salesforce to AWS, event-driven: Event Relay and Amazon EventBridge</h3>



<p class="wp-block-paragraph">Event Relay subscribes to a channel on the Salesforce event bus and streams platform events and change data capture events into Amazon EventBridge as a partner event source. No middleware, no subscriber you have to keep alive, no polling and therefore no API allocation burn on the read path. From EventBridge you fan out to Lambda, Step Functions, SQS, Firehose, whatever the downstream is.</p>



<p class="wp-block-paragraph">Setup runs across both consoles, and the order matters. On the Salesforce side you create a named credential that points at your AWS account and region. It uses no authentication protocol, because trust is established by the partner event source relationship rather than by a credential in the request. The region goes in uppercase, which is the single most common reason a first attempt doesn&#8217;t work.</p>



<pre class="wp-block-code"><code>POST /services/data/vXX.X/tooling/sobjects/NamedCredential/

{
  "FullName": "AwsRelayCredential",
  "Metadata": {
    "label": "AwsRelayCredential",
    "endpoint": "arn:aws:US-EAST-1:111122223333",
    "principalType": "NamedUser",
    "protocol": "NoAuthentication",
    "generateAuthorizationHeader": true
  }
}</code></pre>



<p class="wp-block-paragraph">The endpoint is not a URL. It&#8217;s an ARN-shaped string carrying the AWS region and the twelve-digit account ID, and that&#8217;s the whole addressing scheme. Get the account wrong and the partner event source appears in a stranger&#8217;s console, not yours.</p>



<p class="wp-block-paragraph">You then create an event channel, add the platform events or change events you want as channel members, and create the relay configuration against that channel. A relay is created in a stopped state deliberately, so you get a chance to verify the AWS account and region before anything flows. That pause is the useful part of the design, not an inconvenience to click through.</p>



<p class="wp-block-paragraph">Now switch to AWS and associate the source. This is the step that, skipped, produces the silent drop described earlier.</p>



<pre class="wp-block-code"><code># Find the partner event source Salesforce created in your account
aws events list-event-sources --region us-east-1

# Create the matching bus. Both values must be the source name, character for character.
aws events create-event-bus 
  --name "aws.partner/salesforce.com/&lt;org-and-channel-suffix&gt;" 
  --event-source-name "aws.partner/salesforce.com/&lt;org-and-channel-suffix&gt;" 
  --region us-east-1

# Confirm the source moved from PENDING to ACTIVE
aws events describe-event-source 
  --name "aws.partner/salesforce.com/&lt;org-and-channel-suffix&gt;" 
  --region us-east-1</code></pre>



<p class="wp-block-paragraph"><code>describe-event-source</code> returning <code>ACTIVE</code> is your proof that a matching bus exists and is enabled. <code>PENDING</code> means either you haven&#8217;t created the bus or it&#8217;s deactivated, and every event published in that state is gone. This is the single check worth wiring into a synthetic monitor, because it&#8217;s cheap to run and it catches the most expensive mistake.</p>



<p class="wp-block-paragraph">Only once the source is active do you set the relay state to run. From the Salesforce side, the feedback object is where relay errors surface rather than in any log you&#8217;d normally tail:</p>



<pre class="wp-block-code"><code>SELECT Id, RemoteResource, Status, ErrorMessage, ErrorTime, ErrorIdentifier
FROM EventRelayFeedback
WHERE EventRelayConfigId = '7k2XXXXXXXXXXXXXXX'</code></pre>



<p class="wp-block-paragraph">If you&#8217;d rather not use Event Relay at all, the Pub/Sub API is the direct route: a gRPC interface to the same event bus, delivering Avro-encoded payloads, which you subscribe to from your own consumer. That&#8217;s more code and a process to keep running, on ECS or a VPS from a provider like InterServer or Hetzner if you&#8217;re keeping it outside AWS, but you get explicit control over replay ID checkpointing. Teams already running Kafka often go this way and land the stream through a managed connector from Confluent instead. Event Relay is the lower-operational-cost option; the Pub/Sub API is the one to reach for when you need the checkpoint under your own control.</p>



<h3 class="wp-block-heading">AWS to Salesforce, event-driven: EventBridge API Destinations</h3>



<p class="wp-block-paragraph">Going the other way, an API destination lets EventBridge call the Salesforce REST API directly, with no Lambda in the middle. You create a connection holding OAuth client credentials from a Salesforce connected app, point the destination at your org&#8217;s My Domain endpoint, and route matched events to it. Publishing back into an inbound platform event is the common pattern, so the write lands on the Salesforce event bus and Apex or Flow picks it up from there.</p>



<p class="wp-block-paragraph">Two constraints shape the design, and both are documented:</p>



<ul class="wp-block-list">
<li><strong>Five second client execution timeout.</strong> If Salesforce takes longer than that to respond, EventBridge times out the request and treats it as a retryable failure. Anything you make synchronous behind that endpoint, a trigger doing a callout, a validation chain, a busy org at peak, will eat into it.</li>

<li><strong>The default retry policy is generous, which is the problem.</strong> For event buses, EventBridge will retry a failed delivery for up to 24 hours and up to 185 times by default. If the endpoint is genuinely broken, that&#8217;s a day of retries against your org, consuming API calls, before the event is discarded with nowhere to go.</li>
</ul>



<p class="wp-block-paragraph">Tighten the policy and always attach a dead letter queue. Failed events land somewhere you can inspect and replay rather than evaporating at the end of the retry window:</p>



<pre class="wp-block-code"><code>{
  "Id": "salesforce-writeback",
  "Arn": "arn:aws:events:us-east-1:111122223333:api-destination/sf-platform-event/&lt;id&gt;",
  "RoleArn": "arn:aws:iam::111122223333:role/EventBridgeApiDestinationRole",
  "RetryPolicy": {
    "MaximumRetryAttempts": 20,
    "MaximumEventAgeInSeconds": 3600
  },
  "DeadLetterConfig": {
    "Arn": "arn:aws:sqs:us-east-1:111122223333:salesforce-writeback-dlq"
  }
}</code></pre>



<p class="wp-block-paragraph">Set the invocation rate limit on the destination to something your org can absorb. It exists specifically so you can align delivery with a third-party API&#8217;s limits, and Salesforce is very much a third-party API with limits.</p>



<p class="wp-block-paragraph">One thing worth knowing before you design around it: API destinations don&#8217;t support mutual TLS. If your security review requires mTLS on outbound calls, you&#8217;re putting a Lambda or a proxy in the path regardless.</p>



<h3 class="wp-block-heading">AWS to Salesforce, batch: Lambda and the Bulk API</h3>



<p class="wp-block-paragraph">For loading volume back into the org, nothing managed does this well and you write it yourself. A Lambda or a container job authenticates with the JWT bearer flow against a connected app, submits a Bulk API job, and polls for results. The REST API is fine for small writes; past a few thousand records the per-call overhead and the API allocation make the Bulk API the only sensible choice.</p>



<p class="wp-block-paragraph">This is the cell where you own everything: retries, idempotency, partial-failure handling, and backoff when the org throttles you. Use external ID fields and upsert semantics rather than insert, so a retried batch converges instead of duplicating. Store the job ID before you submit, not after, so a Lambda timeout mid-flight leaves you something to reconcile against. And keep the Salesforce credential in Secrets Manager with rotation wired up, because a hardcoded connected app secret in an environment variable is the finding that shows up in every audit.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The network layer: public internet, PrivateLink, or Private Connect</h2>



<p class="wp-block-paragraph">All four paths above work over the public internet with TLS, and for most workloads that&#8217;s genuinely fine. The question is whether your compliance posture or your risk appetite says otherwise.</p>



<p class="wp-block-paragraph">Salesforce Private Connect is the managed private path. Salesforce operates a transit VPC in each supported AWS region, and you create a PrivateLink connection between that transit VPC and yours. Traffic between the two never traverses the public internet. The terminology is worth getting right because it trips people up in design reviews:</p>



<ul class="wp-block-list">
<li><strong>Inbound</strong> means traffic flowing <em>into</em> Salesforce. AWS to Salesforce, or on-premises to Salesforce.</li>

<li><strong>Outbound</strong> means traffic flowing <em>out of</em> Salesforce. Salesforce calling a service in your VPC.</li>
</ul>



<p class="wp-block-paragraph">The direction is named from Salesforce&#8217;s point of view, not yours, and getting it backwards means provisioning the wrong connection.</p>



<p class="wp-block-paragraph">AppFlow can ride on this too. Choosing the PrivateLink option when you create the Salesforce connection has AppFlow route through Private Connect and manage the endpoint lifecycle for you, which is a meaningful simplification over building it by hand.</p>



<p class="wp-block-paragraph">Where I&#8217;d push back: Private Connect is a licensed add-on, and each VPC needs its own connection, so a multi-account AWS estate with separate VPCs per environment multiplies the cost. If the driver is a checkbox on a questionnaire rather than a real threat model, TLS with strict IP allowlisting on the Salesforce side and a well-scoped connected app gets you most of the way for nothing. If the driver is a regulator who wants to see that customer data never touches a public network path, Private Connect is the clean answer and the audit conversation ends quickly. Decide which one you&#8217;re in before you start pricing it.</p>



<h2 class="wp-block-heading">The reconciliation job nobody builds</h2>



<p class="wp-block-paragraph">This is the part that separates an integration that survives from one that quietly rots, and it&#8217;s about thirty lines of code plus a schedule.</p>



<p class="wp-block-paragraph">Run a count comparison on a window that both sides can express. Nightly is plenty:</p>



<pre class="wp-block-code"><code>-- Salesforce side, via the REST query endpoint
SELECT COUNT(Id) FROM Opportunity
WHERE LastModifiedDate = LAST_N_DAYS:1

-- AWS side, same window, in Athena
SELECT COUNT(DISTINCT id) FROM crm.opportunity
WHERE last_modified_date &gt;= date_add('day', -1, current_date)</code></pre>



<p class="wp-block-paragraph">If the two numbers differ by more than a small tolerance for clock skew at the window edges, alert. That single check catches a stopped relay, a failed flow, a mapping that silently dropped a field, and a partner event source sitting in <code>PENDING</code>. It catches all of them without you having to instrument each path separately, which is exactly why it&#8217;s worth building first.</p>



<p class="wp-block-paragraph">For the delete problem, Salesforce exposes a dedicated endpoint. Ask it for records removed in a window and apply those deletions on the AWS side:</p>



<pre class="wp-block-code"><code>GET /services/data/vXX.X/sobjects/Opportunity/deleted/
    ?start=&lt;ISO-8601 timestamp, URL encoded&gt;
    &amp;end=&lt;ISO-8601 timestamp, URL encoded&gt;</code></pre>



<p class="wp-block-paragraph">The window you can ask about is bounded by how long deleted records stay recoverable, so run it more often than that bound, not less. Change data capture delivers delete events too, which is the better answer when it&#8217;s available, but the endpoint is what you use to backfill after a gap, and after a 72-hour gap it&#8217;s the <em>only</em> thing that will tell you what you missed.</p>



<p class="wp-block-paragraph">Put both checks on one dashboard alongside your AppFlow run status and EventBridge failed-invocation metrics. CloudWatch will do it, and if you&#8217;re already running Grafana Cloud or Datadog for the rest of the estate, pull the Salesforce-side numbers in as a custom metric so one panel answers &#8220;are the two systems agreeing right now&#8221;.</p>



<h2 class="wp-block-heading">Troubleshooting a Salesforce AWS integration</h2>



<ul class="wp-block-list">
<li><strong>Relay is running, nothing arrives in EventBridge.</strong> Run <code>describe-event-source</code>. If the state is <code>PENDING</code>, the bus was never created or has been deactivated, and everything published so far is gone. Create the bus, then treat the gap as a backfill job.</li>

<li><strong>Relay won&#8217;t start, or flips to an error state.</strong> Query <code>EventRelayFeedback</code> for the config. The usual causes are a wrong AWS account ID in the named credential, a region in lowercase, or the channel having no members.</li>

<li><strong>Events arrive but no rule matches.</strong> Check the rule is attached to the partner event bus, not the default bus. This one costs people an afternoon. Match on the source prefix and the detail type, and test with an event pattern before wiring a real target.</li>

<li><strong>An AppFlow flow that ran for months suddenly fails.</strong> Look for a field in the mapping that no longer exists in the source object. Also check for a bad gateway error, which points at a connection configured with the wrong instance URL and is fixed by deleting the connection and recreating it rather than editing it.</li>

<li><strong>API destination returns 401 intermittently.</strong> The connection refreshes its OAuth token when it sees a 401 or 407. Persistent 401s usually mean the connected app has no usable scopes for the client credentials flow, or the run-as user was changed. Check the connected app&#8217;s scopes before you touch anything in AWS.</li>

<li><strong>Everything times out at almost exactly five seconds.</strong> That&#8217;s the API destination timeout, not Salesforce being slow. Move the work behind an inbound platform event so the HTTP call returns immediately and the processing happens asynchronously in the org.</li>

<li><strong>Unexplained API limit exhaustion.</strong> Sum the scheduled AppFlow runs against page counts before blaming a rogue integration user. A frequent flow across several objects adds up faster than most people expect.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Starting the relay before associating the partner event bus, and assuming the events queued somewhere.</li>

<li>Treating the 72-hour retention as a comfortable buffer. It&#8217;s three days including a weekend, and an unmonitored relay burns through it without a single alert.</li>

<li>Using change data capture as a replacement for an initial load. It streams changes from the moment you enable it; it doesn&#8217;t backfill history.</li>

<li>Building the whole integration bidirectionally through one mechanism, then discovering you&#8217;ve created an echo loop where a write from AWS fires a change event that triggers another write.</li>

<li>Sizing the AppFlow schedule for freshness without checking what it does to the org&#8217;s API allocation.</li>

<li>Leaving the default 24-hour EventBridge retry policy on an API destination with no dead letter queue.</li>

<li>Provisioning Private Connect for every VPC because it&#8217;s the secure-sounding option, without modelling the per-connection cost across environments.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Build the reconciliation check before you build the second integration path. It&#8217;s the only thing that tells you the whole system is working.</li>

<li>Monitor the partner event source state, not just rule invocations. Invocations go to zero on a quiet day and on a broken day, and the state field distinguishes them.</li>

<li>Give every integration its own connected app and its own integration user. Shared credentials make it impossible to attribute API consumption or revoke one path without breaking three.</li>

<li>Make writes idempotent with external IDs and upsert. Every path here retries, and some retry aggressively.</li>

<li>Version your event payloads. Platform events are immutable once published, and adding a required field later means a new event definition, not an edit.</li>

<li>Keep the AWS-side resource names discoverable from the Salesforce side. Put the flow name or event bus name in the named credential label so a Salesforce admin can see what&#8217;s attached before they change something.</li>

<li>Define the whole thing in Terraform or CloudFormation on the AWS side and in metadata on the Salesforce side, so a sandbox refresh doesn&#8217;t leave you clicking through two consoles from memory.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need MuleSoft to integrate Salesforce with AWS?</h3>



<p class="wp-block-paragraph">No. For the four patterns above, the native services cover it without an integration platform. MuleSoft, Boomi, and similar tools earn their place when you&#8217;re orchestrating across many systems, need transformation logic that lives outside both platforms, or have a governance requirement for a single integration layer. For a two-system integration, they&#8217;re usually more machinery than the problem needs.</p>



<h3 class="wp-block-heading">Does Event Relay use my Salesforce API allocation?</h3>



<p class="wp-block-paragraph">Event delivery counts against your platform event delivery allocation, which is a separate budget from the 24-hour API request limit that REST calls and AppFlow polling draw on. That separation is one of the strongest arguments for the event-driven path when API consumption is already tight in the org.</p>



<h3 class="wp-block-heading">Can I replay Salesforce events that AWS missed?</h3>



<p class="wp-block-paragraph">Within the 72-hour retention window, yes, using replay IDs from a durable subscriber. Beyond it, no. The events are purged and there&#8217;s no recovery path through the event bus. Backfilling from that point means querying the objects directly and using the deleted-records endpoint to catch removals, which is exactly why the reconciliation job matters.</p>



<h3 class="wp-block-heading">Should I use AppFlow or write my own Lambda extractor?</h3>



<p class="wp-block-paragraph">Start with AppFlow. It removes the auth handling, the pagination, and the scheduling, and for standard object pulls into S3 or Redshift it&#8217;s hard to justify writing that yourself. Move to a custom extractor when you need transformation logic AppFlow can&#8217;t express, when you&#8217;re hitting connector limitations on a specific object, or when you want tighter control over API consumption than a fixed schedule allows.</p>



<h3 class="wp-block-heading">How do I send data from AWS back into Salesforce without a Lambda?</h3>



<p class="wp-block-paragraph">EventBridge API destinations call the Salesforce REST API directly using OAuth client credentials from a connected app. Point them at an inbound platform event endpoint so the write returns fast and processing happens asynchronously in the org. You still need a Lambda if you require mTLS or payload transformation beyond what the input transformer handles.</p>



<h3 class="wp-block-heading">Is Private Connect required for a secure Salesforce AWS integration?</h3>



<p class="wp-block-paragraph">Not required. Everything runs over TLS by default. Private Connect removes the public internet from the path entirely, which matters when a regulator or a security review asks you to demonstrate it. It&#8217;s a licensed add-on with a per-connection cost, so model it against your actual VPC count before committing.</p>



<h3 class="wp-block-heading">Why does my EventBridge rule never fire even though events are arriving?</h3>



<p class="wp-block-paragraph">Almost always because the rule is on the default event bus rather than the partner event bus. Partner events land on their own bus and rules are bus-scoped, so a correct pattern on the wrong bus matches nothing and reports nothing.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A Salesforce AWS integration doesn&#8217;t usually break loudly. It drifts. A relay stops, a mapping goes stale, a delete never propagates, and the two systems keep answering questions confidently with different numbers. The services themselves are well built and the setup is genuinely not hard once you know the region goes in uppercase and the bus has to be associated first.</p>



<p class="wp-block-paragraph">So pick the path by direction and shape, wire the failure modes you&#8217;ve accepted into alerts, and build the nightly count comparison before you build anything clever. That one job is worth more than any amount of pipeline polish, because it&#8217;s the only thing in the architecture that can tell you the whole thing is still true.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your Salesforce and AWS setup?</h2>



<p class="wp-block-paragraph">I work with teams on the infrastructure side of CRM integrations, usually where the data is disagreeing and nobody can prove why. Things I can help with:</p>



<ul class="wp-block-list">
<li>Reviewing an existing Salesforce to AWS pipeline and finding where records are being lost between the two.</li>

<li>Standing up Event Relay into EventBridge end to end, including the partner bus association and the monitoring that proves it&#8217;s live.</li>

<li>Designing the AppFlow schedule and object set so it doesn&#8217;t eat the org&#8217;s API allocation.</li>

<li>Building the reconciliation and delete-propagation jobs, with alerting in CloudWatch, Grafana, or whatever you already run.</li>

<li>Writing the write-back path properly: Bulk API jobs, idempotent upserts, dead letter queues, and credential rotation through Secrets Manager.</li>

<li>Working out whether Private Connect is worth the licence in your specific environment, or whether TLS and tighter scoping gets you there.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;ve got a flow that&#8217;s failing, a relay stuck in an error state, or two record counts that won&#8217;t agree, send me the config or the error and I&#8217;ll tell you what I&#8217;d check first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-aws-integration-architecture/">Salesforce AWS Integration Architecture: The Four Paths and What Breaks Silently</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a Salesforce AI Assistant on Amazon Bedrock Without Leaking Your CRM</title>
		<link>https://john-nessime.com/blog/devops/salesforce-ai-assistant-amazon-bedrock/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 18 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[AgentCore]]></category>
		<category><![CDATA[Agentforce]]></category>
		<category><![CDATA[AI Agents]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[API Gateway]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Bedrock Guardrails]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Cognito]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[External Client Apps]]></category>
		<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[Identity Propagation]]></category>
		<category><![CDATA[MCP]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Salesforce]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=211</guid>

					<description><![CDATA[<p>The demo works, then someone sees records they shouldn't. Nothing errors. Here's how to build a Salesforce AI assistant with Amazon Bedrock that respects your sharing model: which doors AWS and Salesforce just closed, why identity propagation is the failure mode that bites, and how to shape the tool surface so the agent can't wander.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-ai-assistant-amazon-bedrock/">Building a Salesforce AI Assistant on Amazon Bedrock Without Leaking Your CRM</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The demo went fine. Someone from the support team asked the assistant which accounts looked at risk this quarter, and it answered. Good answer, too. Right opportunities, right open cases, sensible summary.</p>



<p class="wp-block-paragraph">Then somebody noticed that the person who asked couldn&#8217;t see half those opportunities in the Salesforce UI. Different role, different branch of the hierarchy. They saw them through the assistant.</p>



<p class="wp-block-paragraph">Nothing errored. No 403, no exception in the logs, no alert. The integration user had access, so the API call succeeded, so the model got the records, so it summarised them. Every layer did exactly what it was configured to do. That is the failure mode that should worry you most when you build a Salesforce AI assistant with Amazon Bedrock: it is silent, it is correct-looking, and you will not find it in a load test.</p>



<p class="wp-block-paragraph">This post covers what actually goes wrong when you wire a large language model into a CRM: which door AWS closed on the entry path, whose permissions your agent is really running under, how to shape the tool surface so the model can&#8217;t wander, and what happens to your org&#8217;s API budget when an agent starts looping. There&#8217;s a build walkthrough, but the build is the easy part.</p>



<h2 class="wp-block-heading">Two doors closed while you were reading the old tutorial</h2>



<p class="wp-block-paragraph">Most of what ranks for this topic was written against an architecture you can no longer build in a fresh account. Both vendors changed the front door within a few months of each other, and the tutorials haven&#8217;t caught up. Check this before you follow anything, including this post.</p>



<h3 class="wp-block-heading">On the AWS side: Bedrock Agents is now Bedrock Agents Classic</h3>



<p class="wp-block-paragraph">AWS renamed the original Bedrock Agents service to Bedrock Agents Classic and put it into maintenance mode. It stopped accepting new customers on 30 July 2026. This one is worth stating with the date attached, because the date is the whole point.</p>



<p class="wp-block-paragraph">The mechanics matter more than the branding:</p>



<ul class="wp-block-list">
<li>Only <code>CreateAgent</code> and <code>InvokeInlineAgent</code> are restricted. Every other API stays available to everyone.</li>

<li>Allowlisting is per AWS account, based on whether that account used Bedrock Agents in the previous twelve months. Multi-account estates get a split result: your old sandbox works, your new production account doesn&#8217;t.</li>

<li>A non-allowlisted account calling <code>CreateAgent</code> gets an <code>AccessDeniedException</code> with HTTP 403, not a quota error. If you&#8217;re debugging IAM and seeing 403s, this is a plausible cause that no policy change will fix.</li>

<li>There is no exception process. AWS documents this plainly.</li>

<li>The model catalogue inside Classic is frozen. Bedrock itself keeps getting new models; the Classic orchestration layer does not.</li>
</ul>



<p class="wp-block-paragraph">The replacement is Amazon Bedrock AgentCore. It splits into a managed harness (config-driven, closest to the old experience) and code-defined agents on the AgentCore runtime (bring Strands, LangGraph, CrewAI, or your own loop). For a Salesforce assistant, the harness is usually enough, and you can move to code later without re-plumbing the tools.</p>



<p class="wp-block-paragraph">Existing Classic agents keep running. There&#8217;s no announced end-of-life. But a frozen model catalogue sets a ceiling on the thing, and if your team can&#8217;t recreate the agent in a new account, your disaster recovery story has a hole in it.</p>



<h3 class="wp-block-heading">On the Salesforce side: Connected Apps gave way to External Client Apps</h3>



<p class="wp-block-paragraph">Salesforce blocked creation of new Connected Apps through both UI and API, with package installation as the exception. External Client Apps are the replacement. Existing Connected Apps keep working and keep their OAuth flows, so this is a wall on new creation rather than a sunset.</p>



<p class="wp-block-paragraph">The practical symptom is that the <em>New Connected App</em> button is greyed out with no explanation. Every AWS-to-Salesforce tutorial that opens with &#8220;first, create a Connected App&#8221; stops working right there.</p>



<p class="wp-block-paragraph">One difference matters for agent work specifically: External Client Apps do not support the username-password OAuth flow at all. If your integration design assumed you&#8217;d stash a service account username and password somewhere and call it a day, that design is gone. Which is a good thing, and it forces the question the next section is about.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: whose permissions is the assistant using?</h2>



<p class="wp-block-paragraph">This is the one from the opening, and it&#8217;s the reason most of these projects should slow down before they ship.</p>



<p class="wp-block-paragraph">Salesforce&#8217;s security model is genuinely good. An OAuth token runs in a user context, and object permissions, field-level security, and sharing rules all still apply to API calls. A token with the <code>api</code> scope does not bypass CRUD, FLS, or sharing. People hear that and relax.</p>



<p class="wp-block-paragraph">The catch is the phrase &#8220;a user context.&#8221; Which user?</p>



<p class="wp-block-paragraph">If you authenticate the gateway to Salesforce with client credentials, every request from every person runs as one integration user. Salesforce enforces that user&#8217;s permissions perfectly. It has no idea a different human asked the question. You&#8217;ve built what&#8217;s sometimes called a multiplexor: one licence fronting many people, with all the real access control now living in your application layer instead of the platform&#8217;s.</p>



<p class="wp-block-paragraph">At that point the sharing model is decoration. A prompt that talks the model into calling a query tool with a wider filter returns data the asker was never entitled to, and Salesforce logs it as a legitimate read by a legitimate user. There&#8217;s nothing to alert on, because nothing went wrong.</p>



<h3 class="wp-block-heading">Two-legged versus three-legged, and when each is honest</h3>



<p class="wp-block-paragraph">AgentCore Gateway supports both client credentials (two-legged) and authorization code (three-legged) flows for reaching downstream targets, and handles ingress and egress authorisation separately. That distinction is the control you need.</p>



<ul class="wp-block-list">
<li><strong>Two-legged is honest when the data is genuinely org-wide.</strong> Product catalogues, published knowledge articles, case deflection content. Nobody&#8217;s entitlement varies, so a single service identity represents the truth.</li>

<li><strong>Three-legged is the only correct answer for record data.</strong> Opportunities, accounts, contacts, custom objects with sharing rules on them. The token is minted for the actual human, Salesforce applies their sharing, and the answer is bounded by what they could have looked up themselves.</li>
</ul>



<p class="wp-block-paragraph">Three-legged costs you something real, and I&#8217;d rather say so than pretend it&#8217;s free. You need per-user consent, token storage, refresh handling, and a way to deal with a user whose token has expired mid-conversation. AgentCore Identity provides a token vault and handles the consent flow, including binding the authorisation URL to the session that started it so a shared link can&#8217;t be completed by someone else. It&#8217;s still more moving parts than a single secret in Secrets Manager.</p>



<p class="wp-block-paragraph">Pay the cost anyway for record data. The alternative is a system whose blast radius is &#8220;everything the integration user can see,&#8221; discovered by a customer or an auditor rather than by you.</p>



<p class="wp-block-paragraph">If you already run Okta, Auth0, or Ping as your identity provider, Gateway can sit in front of them for inbound auth rather than you standing up a parallel Cognito user pool. Reusing the IdP you already audit is worth more here than saving a config file.</p>



<h2 class="wp-block-heading">Failure family two: the shape of the tool surface</h2>



<p class="wp-block-paragraph">The second mistake is giving the model a tool called <code>run_soql</code> that accepts an arbitrary query string. It demos beautifully. Ask anything, get anything.</p>



<p class="wp-block-paragraph">You&#8217;ve now made the model your query planner and your authorisation layer at once. Every safety property of the system depends on prompt text. Guardrails help with content, not with whether a generated <code>WHERE</code> clause happened to be too generous.</p>



<p class="wp-block-paragraph">Narrow tools are the fix, and they&#8217;re boring on purpose. <code>get_open_cases_for_account(account_id)</code> beats <code>run_soql(query)</code> because the query shape is fixed in your code, the model only supplies an identifier, and you can log and rate-limit each tool independently. When something goes wrong you&#8217;re reading a parameter, not parsing SOQL to work out what the model asked for.</p>



<p class="wp-block-paragraph">AgentCore Gateway takes OpenAPI specs, Smithy models, and Lambda functions as tool input types, and exposes them to the agent as MCP tools. It also has one-click integrations for several SaaS products including Salesforce. Practically that gives you three routes:</p>



<ol class="wp-block-list">
<li><strong>The built-in Salesforce integration</strong> for the fastest path to something working. Good for a proof of concept. Look closely at which operations it exposes before you take it further, because breadth is exactly what you don&#8217;t want.</li>

<li><strong>An OpenAPI target</strong> describing a small purpose-built API in front of Salesforce. This is the one I reach for first. You control the surface completely, and the spec doubles as the tool description the model reads.</li>

<li><strong>Lambda targets</strong> when a tool needs to combine several Salesforce calls, or talk to something else at the same time.</li>
</ol>



<p class="wp-block-paragraph">MCP server targets are also supported, with the gateway aggregating tools from multiple targets into a single <code>tools/list</code> response. If you&#8217;re running your own MCP server as the Salesforce facade, it needs a stable HTTPS endpoint the gateway can reach; a small VPS from a provider like InterServer or Hetzner is plenty for the development and staging copies, with the production one behind whatever your standard ingress is.</p>



<h2 class="wp-block-heading">Failure family three: the write path</h2>



<p class="wp-block-paragraph">Reads leak. Writes destroy. They deserve different treatment and usually get the same treatment, which is how you end up with three hundred opportunities silently restaged.</p>



<p class="wp-block-paragraph">Start read-only and stay there longer than feels necessary. When you do add writes, the constraints that actually help:</p>



<ul class="wp-block-list">
<li><strong>One record per call.</strong> No bulk update tool. If the model wants to change twenty records it makes twenty calls, each individually logged and individually rate-limited. Slow is a feature here.</li>

<li><strong>Separate the write identity.</strong> A distinct External Client App and permission set for writes means you can revoke the write path at 2am without taking the assistant offline.</li>

<li><strong>Human confirmation on anything irreversible.</strong> The AgentCore harness supports inline function tools that pause the agent and hand control back to your client code, equivalent to return-of-control. That pause is where a person says yes.</li>

<li><strong>Stamp the source.</strong> A custom field recording that a change came from the assistant, with the session and actor ID, is the difference between a ten-minute rollback and an afternoon of forensics.</li>
</ul>



<p class="wp-block-paragraph">Salesforce field history tracking is on a limited number of fields per object. If the field your assistant writes to isn&#8217;t tracked, you have no before-value. Check that before go-live, not after.</p>



<h2 class="wp-block-heading">Failure family four: what happens under load</h2>



<p class="wp-block-paragraph">Salesforce meters API requests against a daily org-wide allocation that depends on your edition and licence count. Your assistant does not consume that budget the way a nightly sync does.</p>



<p class="wp-block-paragraph">A single conversational turn can be several tool calls. A model that doesn&#8217;t get a clean answer will often retry with a different filter. Ten support agents having ten conversations each becomes a request pattern nobody modelled, and the org-wide limit means the integration that breaks first might be your ETL job, your marketing platform, or your billing sync. Not the assistant. That&#8217;s what makes it hard to diagnose.</p>



<p class="wp-block-paragraph">Three things that help, in order of how much they buy you:</p>



<ul class="wp-block-list">
<li>Cap tool calls per session in your agent config, and return a clear message when the cap is hit rather than letting the loop run.</li>

<li>Cache anything org-wide and slow-moving. Picklist values, record type metadata, product catalogue. These are a surprising share of calls and almost never change mid-day.</li>

<li>Watch API usage as a leading indicator, not a post-mortem. Salesforce exposes remaining daily requests via the REST limits resource. Scrape it on a schedule into CloudWatch, Grafana Cloud, or whatever you already page from, and alert on the trend rather than the wall.</li>
</ul>



<h2 class="wp-block-heading">Wiring it up</h2>



<p class="wp-block-paragraph">With the design decisions settled, the build is short. The AgentCore CLI scaffolds and deploys the harness.</p>



<ol class="wp-block-list">
<li>Create an External Client App in Salesforce with the narrowest OAuth scopes the tools need. Assign it via permission set, not profile.</li>

<li>Build the tool layer: an OpenAPI spec for your purpose-built endpoints, or Lambda functions for composite operations.</li>

<li>Create an AgentCore Gateway, attach the target, and configure the credential provider in AgentCore Identity for the egress side.</li>

<li>Create the harness, point it at the gateway, and deploy.</li>
</ol>



<p class="wp-block-paragraph">Scaffolding a harness and attaching a gateway looks like this. The <code>--gateway-arn</code> value is the gateway you created in step three, and the <code>--type</code> tells the CLI this tool is a gateway rather than one of the built-in tools such as the browser or code interpreter:</p>



<pre class="wp-block-code"><code>agentcore create --name sf-assistant

agentcore add tool --harness sf-assistant 
  --type agentcore_gateway --name sf-gateway 
  --gateway-arn arn:aws:bedrock-agentcore:REGION:ACCOUNT_ID:gateway/GATEWAY_ID</code></pre>



<p class="wp-block-paragraph">Then declare the model and the system prompt, and deploy. Check the current model identifiers in the Bedrock console rather than copying one from any blog post, this one included, because they change:</p>



<pre class="wp-block-code"><code>agentcore add harness 
  --name sf-assistant 
  --model-id YOUR_MODEL_ID 
  --system-prompt "You answer questions about CRM records using only the provided tools. Never guess a record value." 
  --tools sf-gateway

agentcore deploy</code></pre>



<p class="wp-block-paragraph">Invoking it takes a session ID, and an actor ID if you want memory scoped per user. Skipping <code>--actor-id</code> is the memory equivalent of the integration user problem: one shared memory store, everyone&#8217;s context in it.</p>



<pre class="wp-block-code"><code>agentcore invoke --harness sf-assistant 
  --session-id "$(uuidgen)" --actor-id alice 
  "Which of my open opportunities have had no activity in three weeks?"</code></pre>



<p class="wp-block-paragraph">Note &#8220;my&#8221; in that question. With three-legged auth it means something. With an integration user it means nothing, and the model will cheerfully invent a definition.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<p class="wp-block-paragraph"><strong><code>AccessDeniedException</code> when creating an agent.</strong> If you&#8217;re calling <code>CreateAgent</code> or <code>InvokeInlineAgent</code> on the Classic API, check whether the account has prior Bedrock Agents usage before you touch IAM. No policy fixes this one. Build on AgentCore instead.</p>



<p class="wp-block-paragraph"><strong>The New Connected App button is greyed out.</strong> Expected. Create an External Client App. If you have a documented technical requirement that only a Connected App satisfies, that&#8217;s a Salesforce Support request, not a setting you can toggle.</p>



<p class="wp-block-paragraph"><strong>Tools appear in <code>tools/list</code> but calls fail.</strong> Gateway can cache tool schemas from a target without holding a working credential for calls. Discovery succeeding proves the schema synced, not that egress auth works. Look at the credential provider.</p>



<p class="wp-block-paragraph"><strong>The assistant returns fewer records than the user expects.</strong> Under three-legged auth this is often correct behaviour that looks like a bug. Have the user run the same query in the Salesforce UI. If the UI shows more, check FLS on the specific fields; if it shows the same, the assistant is right and the user&#8217;s mental model is wrong.</p>



<p class="wp-block-paragraph"><strong>Answers are confidently wrong about record values.</strong> Usually the model answering from training data instead of calling a tool. Tighten the system prompt to forbid answering record questions without a tool result, and check the traces to confirm which turns actually called out.</p>



<p class="wp-block-paragraph"><strong>Everything works, then breaks at the same time daily.</strong> Look at org-wide API consumption before you look at your own code. Something else in the org may be eating the allocation on a schedule.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Using a System Administrator profile for the integration identity because it makes the errors go away.</li>

<li>Exposing a generic SOQL tool and calling the prompt a security control.</li>

<li>Testing only in a sandbox with one user, where the sharing model is invisible because there&#8217;s nothing to share.</li>

<li>Treating Guardrails as an access control layer. They filter content; they don&#8217;t know who&#8217;s asking.</li>

<li>Shipping writes in the first release because the read-only version felt underwhelming in the demo.</li>

<li>Building on Bedrock Agents Classic in a new account without checking allowlist status first.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Use three-legged OAuth for anything governed by sharing rules. Reserve client credentials for genuinely org-wide data.</li>

<li>Give each tool one job and a fixed query shape. The model supplies parameters, never structure.</li>

<li>Separate read and write identities so you can kill one without the other.</li>

<li>Scope memory per actor, not per deployment.</li>

<li>Test with at least three users at different levels of the role hierarchy, and diff the answers. Identical answers to &#8220;show me my pipeline&#8221; is the signal that identity isn&#8217;t propagating.</li>

<li>Keep end-to-end traces and ship them somewhere you already look, so agent behaviour lands in the same place as the rest of your telemetry.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I still use Bedrock Agents for a new Salesforce project?</h3>



<p class="wp-block-paragraph">Only if the AWS account already used Bedrock Agents in the previous twelve months. Accounts without that history get an <code>AccessDeniedException</code> on <code>CreateAgent</code>, and AWS has said there&#8217;s no exception process. For anything new, build on AgentCore.</p>



<h3 class="wp-block-heading">Do I need Agentforce to build a Salesforce AI assistant with Amazon Bedrock?</h3>



<p class="wp-block-paragraph">No. Agentforce is Salesforce&#8217;s own agent layer, and it can call out to AWS, which is useful when you want the assistant to live inside the Salesforce UI and inherit the Einstein Trust Layer. But an AgentCore agent calling Salesforce APIs through a gateway is a complete architecture on its own, and it&#8217;s the better fit when the assistant also needs to reach systems outside Salesforce.</p>



<h3 class="wp-block-heading">Does the model see all my CRM data?</h3>



<p class="wp-block-paragraph">It sees whatever your tools return. That&#8217;s determined by the OAuth identity and the tool definitions, not by the model. This is why the identity decision matters more than the model choice.</p>



<h3 class="wp-block-heading">Will Bedrock Guardrails stop the assistant leaking records?</h3>



<p class="wp-block-paragraph">No, and this is worth being blunt about. Guardrails filter content categories and can mask sensitive data patterns. They have no concept of which user is asking or which records that user is entitled to. A record correctly retrieved under an over-permissioned token passes every guardrail check, because there&#8217;s nothing objectionable about it. Guardrails are a good layer. They are not this layer.</p>



<h3 class="wp-block-heading">How do I keep the assistant from burning the org&#8217;s API limit?</h3>



<p class="wp-block-paragraph">Cap tool calls per session, cache org-wide reference data, and monitor remaining daily requests via the REST limits resource on a schedule. The allocation is org-wide, so the symptom of overuse usually shows up in some other integration first.</p>



<h3 class="wp-block-heading">Should I migrate an existing Bedrock Agents Classic setup?</h3>



<p class="wp-block-paragraph">There&#8217;s no deadline and no announced end-of-life, so there&#8217;s no emergency. The reasons to move are the frozen model catalogue and the fact that new accounts can&#8217;t recreate your agent. If you&#8217;re still actively developing the thing, move. If it&#8217;s stable and finished, it can wait. AWS publishes a capability mapping and a CLI import path for existing configurations.</p>



<h3 class="wp-block-heading">Is an External Client App harder to set up than a Connected App?</h3>



<p class="wp-block-paragraph">Slightly different rather than harder. The main structural change is that developer settings and admin policies are separated, so the app definition and the per-org access controls are configured independently. The one real removal is the username-password flow, which no longer exists in the ECA model.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A Salesforce AI assistant with Amazon Bedrock is not really a model problem. Model quality is the part that will take care of itself, and the part you can change with one config line later.</p>



<p class="wp-block-paragraph">The part that won&#8217;t take care of itself is identity. If the assistant answers as one integration user, you have quietly moved your entire access control model from a platform that enforces it well into a prompt that enforces it not at all, and nothing in your monitoring will tell you. Get that right first. Then argue about models.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help building or auditing this?</h2>



<p class="wp-block-paragraph">Most of the work on these projects is unglamorous: deciding what the agent is allowed to touch, and proving it can&#8217;t touch anything else. That&#8217;s the part I&#8217;m useful for.</p>



<ul class="wp-block-list">
<li>Auditing an existing Salesforce AI integration for identity propagation gaps, including testing across role hierarchy levels to find silent over-permissioning.</li>

<li>Migrating a Bedrock Agents Classic deployment to AgentCore, including mapping action groups to gateway targets.</li>

<li>Designing the tool surface: replacing generic query tools with narrow, logged, individually rate-limited operations.</li>

<li>Setting up three-legged OAuth between AgentCore Gateway and Salesforce, including token vault and consent handling.</li>

<li>Moving integrations from Connected Apps to External Client Apps without breaking existing OAuth flows.</li>

<li>Instrumenting API consumption and agent traces so you see problems before an unrelated integration starts failing.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;ve got something concrete, send it over: a gateway target config, an agent trace, a tool schema, or just the error you&#8217;re stuck on. Easier to say something useful about a real setup than a hypothetical one.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-ai-assistant-amazon-bedrock/">Building a Salesforce AI Assistant on Amazon Bedrock Without Leaking Your CRM</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>SAP Data Integration with AWS Glue: The Failure Modes Nobody Demos</title>
		<link>https://john-nessime.com/blog/devops/sap-data-integration-aws-glue/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 18 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Change Data Capture]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Delta Token]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[ODP]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[SAP]]></category>
		<category><![CDATA[SAP OData]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[VPC]]></category>
		<category><![CDATA[Zero-ETL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=213</guid>

					<description><![CDATA[<p>The AWS Glue SAP OData connector demos beautifully and then drifts. A practical look at delta token invalidation, deletes that never delete, partitioning limits on ODP entities, and the licensing line between ODP-RFC and ODP-OData.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/sap-data-integration-aws-glue/">SAP Data Integration with AWS Glue: The Failure Modes Nobody Demos</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The Glue job had been green for weeks. Every run finished, every run wrote Parquet, nobody had a reason to look at it. Then someone from finance mentioned that a report was short by a few hundred rows, and the reconciliation that followed turned up the real problem: the job had not actually been reading changes for most of that time. It had been reading nothing, successfully.</p>



<p class="wp-block-paragraph">That is the shape of most trouble with SAP data integration with AWS Glue. The connector does not usually crash. It returns an empty or partial result set, the job exits zero, and your data lake quietly diverges from the source system. By the time anyone notices, you have no idea which run was the last correct one.</p>



<p class="wp-block-paragraph">This post covers the parts that bite after the proof of concept: how delta tokens actually behave, why deleted records keep showing up in your reports, where partitioning stops working, and the licensing distinction between two things that both get called &#8220;ODP&#8221; and are not the same thing at all. There is a troubleshooting section at the end for when a job is already broken.</p>



<h2 class="wp-block-heading">First, get the ODP terminology right, because licensing depends on it</h2>



<p class="wp-block-paragraph">Operational Data Provisioning is SAP&#8217;s framework for exposing extraction interfaces in a delta-aware way. A provider (a BW extractor, an ABAP CDS view, SLT, a HANA information view) writes changes into an Operational Delta Queue, and subscribers pull from that queue. This part is uncontroversial.</p>



<p class="wp-block-paragraph">What matters is that there are two ways to talk to ODP, and SAP treats them very differently. SAP Note 3255746 restricts third-party and customer applications from using the RFC modules of the ODP Data Replication API against ABAP sources. That note has been sharpened over several revisions, from &#8220;unsupported&#8221; to &#8220;not permitted&#8221;, and SAP has stated it reserves the right to introduce technical measures and audit unpermitted use. A number of established extraction tools were built directly on ODP-RFC and have had to move.</p>



<p class="wp-block-paragraph">The AWS Glue SAP OData connector takes the other path. It talks OData over HTTPS to an activated SAP Gateway service, and uses the ODP framework through that OData layer rather than through RFC. That is the route SAP points customers at. If you are being asked in an architecture review whether your pipeline is on the wrong side of Note 3255746, this distinction is the whole answer, and it is worth having the note number written down before the meeting.</p>



<p class="wp-block-paragraph">Two honest caveats. First, notes get revised, so read the current version rather than a vendor blog summarising it, including this one. Second, &#8220;permitted&#8221; is not the same as &#8220;free&#8221;: you still need the OData service activated, a service user with the right authorisations, and a Basis team willing to own the queue growth on their side.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: the delta token is shared state you do not own</h2>



<p class="wp-block-paragraph">This is the one that produces silent drift, and it is the reason the finance report was short.</p>



<p class="wp-block-paragraph">When you read an ODP-enabled entity with change data capture switched on, the connector asks SAP to initialise a delta queue subscription. SAP hands back a delta token. That token is your bookmark. On the next run you pass it back, and you get only what changed since.</p>



<pre class="wp-block-code"><code>sapodata_df = glueContext.create_dynamic_frame.from_options(
    connection_type="SAPOData",
    connection_options={
        "connectionName": "sap-prod-odata",
        "ENTITY_NAME": "entityName",
        "ENABLE_CDC": "true"
    },
    transformation_ctx="initial_load"
)</code></pre>



<p class="wp-block-paragraph"><code>ENABLE_CDC</code> defaults to false. Leave it off and you get a full load every single run, which is expensive but at least obvious. Turn it on and the token appears in the <code>DELTA_TOKEN</code> column, populated only on the last row of the frame. That last row is not a business record. It is a carrier for the token, and if you write your frame straight to S3 without stripping it you have just added a junk row to your table.</p>



<p class="wp-block-paragraph">The subsequent run passes the token back:</p>



<pre class="wp-block-code"><code>sapodata_df = glueContext.create_dynamic_frame.from_options(
    connection_type="SAPOData",
    connection_options={
        "connectionName": "sap-prod-odata",
        "ENTITY_NAME": "entityName",
        "DELTA_TOKEN": last_known_token
    },
    transformation_ctx="incremental_load"
)</code></pre>



<h3 class="wp-block-heading">Why this goes wrong</h3>



<p class="wp-block-paragraph">A delta token is bound to a service collection <em>and a user</em>. If a new initialisation runs with <code>ENABLE_CDC</code> set true for that same collection and user, SAP invalidates every token previously issued from the earlier initialisation. Not &#8220;expires eventually&#8221;. Invalidates, immediately.</p>



<p class="wp-block-paragraph">So picture two Glue jobs sharing one service account, both reading the same extractor. Someone redeploys job B with the CDC flag still set from testing. Job B re-initialises. Job A&#8217;s stored token is now dead, and the next time job A runs you get:</p>



<pre class="wp-block-code"><code>Could not open data access via extraction API RODPS_REPL_ODP_OPEN</code></pre>



<p class="wp-block-paragraph">An error is the good outcome. The bad outcome is code that catches the exception, falls back to a full load, and writes it over the top of an incremental table with no dedupe. Or code that re-initialises on failure, which fixes job A and breaks job B, and now the two jobs take turns invalidating each other forever while both keep exiting zero.</p>



<p class="wp-block-paragraph">Practical rules that come out of this:</p>



<ul class="wp-block-list">
<li>One SAP service user per consuming pipeline. Sharing a user is what couples independent jobs together.</li>

<li>Persist the token somewhere durable and versioned. DynamoDB, Parameter Store, S3 with versioning enabled, all fine. Job bookmarks are not a substitute, because the token is source state, not read position.</li>

<li>Never auto re-initialise on error. Fail loudly, page someone, and make the reinitialisation a deliberate human action with a documented backfill.</li>

<li>Alert on zero-change runs. A pipeline that returns no changes for several consecutive runs against a busy table is telling you something, and it is rarely &#8220;business is quiet&#8221;.</li>
</ul>



<h3 class="wp-block-heading">Retention on the SAP side is a hard deadline</h3>



<p class="wp-block-paragraph">The delta queue is not infinite. In transaction <code>ODQMON</code> the Basis team configures retention, and a reorganisation job cleans up data past that window. If your pipeline is down longer than the retention period, the changes you missed are gone from the queue and no token will bring them back. You are into a full reload and a reconciliation.</p>



<p class="wp-block-paragraph">Two things to find out before you go live: what the retention setting is, and whether the cleanup job is actually scheduled. Then make sure your pipeline&#8217;s alerting escalates well inside that window. A weekend outage on a queue with a short retention is a very expensive Monday.</p>



<p class="wp-block-paragraph">Related: if anyone has been testing extraction with <code>RODPS_REPL_TEST</code>, that program registers as a real additional subscriber rather than simulating one. Because the queue is shared and data is only cleaned once every subscriber has collected it, forgotten test subscriptions hold delta data indefinitely and the underlying queue tables grow. Ask whoever ran the tests to reset those subscriptions in ODQMON.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family two: deletes arrive as a column, not as a deletion</h2>



<p class="wp-block-paragraph">CDC extraction from ODP entities returns a couple of extra fields alongside your business columns. <code>DML_STATUS</code> carries <code>UPDATED</code> for inserts and updates, and <code>DELETED</code> for records removed at source. <code>GLUE_FETCH_SQ</code> is a per-record sequence derived from an epoch timestamp, in receipt order, and it is your tie-breaker when the same key changes twice inside one delta.</p>



<p class="wp-block-paragraph">Nothing acts on those fields for you. If your job appends the frame to S3 and calls it done, a deleted sales order is now a row in your lake with <code>DML_STATUS = DELETED</code> and every other column populated exactly as before. Any query that does not filter on that column counts it. Cancelled orders keep appearing in revenue. Closed accounts stay open. The pipeline is green, the numbers are wrong, and the discrepancy grows slowly enough that nobody catches it early.</p>



<p class="wp-block-paragraph">You have to apply the changes rather than accumulate them. Broadly two options:</p>



<ol class="wp-block-list">
<li><strong>Merge into a table format that supports it.</strong> Apache Iceberg, which Glue supports natively, gives you a real merge with delete handling. Land the raw delta, then merge on the business key with <code>GLUE_FETCH_SQ</code> deciding the winner within a batch, deleting where the status says deleted.</li>

<li><strong>Keep an append-only history and build a current-state view.</strong> Cheaper to write, and it preserves an audit trail. It also means every downstream consumer has to use the view, and someone eventually will not.</li>
</ol>



<p class="wp-block-paragraph">I reach for Iceberg first on anything that feeds reporting, because the correctness lives in the table rather than in the discipline of whoever writes the next query. Append-only is the right call when you genuinely need history, or when the consumer is a downstream warehouse like Amazon Redshift that will do its own merge anyway.</p>



<p class="wp-block-paragraph">Worth knowing: not every ODP provider emits deletes reliably. Some older extractors express them as reversal records rather than a delete flag. Confirm the behaviour for your specific entity with the functional owner before you build merge logic on an assumption.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family three: pagination and partitioning do not compose</h2>



<p class="wp-block-paragraph">The connector&#8217;s default page size is 50,000 records, and you can raise it up to 500,000. Anything above that is ignored and the maximum is used instead. Raising the page size cuts the number of round trips, which is usually the dominant cost on a wide table across a slow link.</p>



<p class="wp-block-paragraph">The catch is what your SAP Gateway will tolerate. A large page means a large response the ABAP stack has to assemble and hold, and Gateway or an intervening load balancer may cut it off before it completes. Symptoms are timeouts, connection resets, or a job that appears to paginate without ever finishing. Tune the page size with your Basis team, not in isolation, and treat it as a negotiation between round trips and response size rather than a value you set once.</p>



<p class="wp-block-paragraph">On partitioning, there is a specific documented limit that catches people out: ODP entities are not compatible with record-based partitioning, because pagination is already handled through skip and delta tokens. For record-based partitioning, the default concurrency is forced to null regardless of what you asked for. So the obvious lever for making a slow ODP extraction faster is not available.</p>



<p class="wp-block-paragraph">What you can still do on non-ODP OData entities is field-based partitioning, splitting a read across a bounded range:</p>



<ul class="wp-block-list">
<li><code>PARTITION_FIELD</code> is the field to split on, typically a date.</li>

<li><code>LOWER_BOUND</code> is inclusive, <code>UPPER_BOUND</code> is exclusive. Getting that backwards is how you duplicate or drop a boundary day.</li>

<li><code>NUM_PARTITIONS</code> defaults to 1, so leaving it out means you did not partition at all.</li>

<li>Where a limit and a partition are both applied, the limit wins. That is the usual cause of &#8220;I partitioned it and got fewer rows&#8221;.</li>
</ul>



<p class="wp-block-paragraph">For non-ODP entities you can also do timestamp-based incremental reads with <code>filteringExpression</code>, pointing at whatever last-changed field the entity exposes. Note the casing: that option is camelCase while most of the others are upper snake case, which trips people up when a filter appears to be ignored.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family four: connection, auth and network</h2>



<p class="wp-block-paragraph">These fail early and loudly, which makes them the easy ones. The trap is that they fail during setup, when you have three people on a call and someone suggests loosening something to move on.</p>



<p class="wp-block-paragraph">The connector supports two authentication types: CUSTOM, which is SAP basic authentication, and OAUTH2. The custom secret in AWS Secrets Manager takes this shape:</p>



<pre class="wp-block-code"><code>{
  "basicAuthUsername": "&lt;your SAP username&gt;",
  "basicAuthPassword": "&lt;your SAP password&gt;",
  "basicAuthDisableSSO": "&lt;True/False&gt;",
  "customAuthenticationType": "CustomBasicAuth"
}</code></pre>



<p class="wp-block-paragraph">For OAuth 2.0 the client secret goes into Secrets Manager under <code>USER_MANAGED_CLIENT_APPLICATION_CLIENT_SECRET</code>, and the client ID, authorisation code URL, token URL and scopes are supplied on the connection itself.</p>



<p class="wp-block-paragraph">Basic auth is genuinely easier for a scheduled pipeline: no interactive redirect, no refresh token to keep alive, and rotation is a Secrets Manager operation. OAuth gets you short-lived tokens and revocation you can actually reason about, which is what a security review will want. Both are defensible. What is not defensible is one shared service user across every pipeline, for the delta token reasons above as much as the audit ones.</p>



<p class="wp-block-paragraph">A few things that reliably waste a morning:</p>



<ul class="wp-block-list">
<li>The service path on the connection is the <em>catalog</em> service path, not the path to your object. The entity path goes in <code>ENTITY_NAME</code> on the job.</li>

<li>The OData service has to be activated in the SAP system. Until it is, nothing you do on the AWS side matters.</li>

<li>For a non-VPC connection, the application host URL must be reachable from the public internet. If your SAP system sits in a private network, configure VPC connectivity properly rather than opening it up.</li>

<li>The Glue role needs Secrets Manager read plus the EC2 network interface permissions when a VPC is involved. A missing <code>ec2:CreateNetworkInterface</code> shows up as a connection failure that looks like a network problem.</li>

<li>Private certificate authorities need the chain in a form Glue accepts. If you are wiring this up yourself, expect it to take a pass or two.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Choosing your path: Glue jobs, zero-ETL, or something else</h2>



<p class="wp-block-paragraph">The same connector is reachable through more than one product surface, and the choice is really about how much state you want to own.</p>



<ul class="wp-block-list">
<li><strong>Glue Studio or script jobs.</strong> You control page size, filters, output format, transformation and scheduling, and you own delta token persistence. Pick this when you need real transformation in flight, or when the merge logic has to live next to the extraction.</li>

<li><strong>Glue zero-ETL integrations.</strong> AWS manages token state internally, runs the initial full load and then the incremental cycle, and preserves the token on failure so retries do not lose data. Less code, less control. Worth noting that once an incremental update field is chosen for a connection and entity, a later integration on the same pair cannot pick a different one.</li>

<li><strong>Amazon AppFlow.</strong> Simpler, flow-shaped, well suited to modest volumes and quick prototypes. Less appropriate once you need Spark-scale processing.</li>

<li><strong>Commercial replication tools.</strong> Fivetran, Airbyte and SAP-specialist vendors all play here. They cost money and add a vendor, and in exchange they absorb exactly the state management this post is about. If your team has no SAP depth, that trade is often correct.</li>
</ul>



<p class="wp-block-paragraph">On cost, Glue bills for the resources a job consumes for as long as it runs, so the levers are the same as any Spark workload: fewer round trips, less data pulled, fewer idle executors, and not re-reading history you already have. Rates change and vary by region, so model it against current pricing rather than a figure from a blog post. If your incremental run is pulling the same volume as your initial load, the delta mechanism is not working and the bill is the symptom.</p>



<p class="wp-block-paragraph">Also worth writing back. The connector supports write operations to SAP OData, using <code>INSERT</code> and <code>UPDATE</code>, with <code>ID_FIELD_NAMES</code> required on update to identify the external ID field. Useful for enrichment loops, and something to treat with a good deal more caution than a read.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">&#8220;Could not open data access via extraction API RODPS_REPL_ODP_OPEN&#8221;</h3>



<p class="wp-block-paragraph">Your token is invalid or expired. Find out why before you re-initialise, because re-initialising destroys the evidence. Check ODQMON for a recent initialisation on that collection and user, check whether another job or a test program shares the credentials, and check whether the request you wanted has aged out of retention. Then re-initialise deliberately and reconcile the gap.</p>



<h3 class="wp-block-heading">The job succeeds but returns no rows</h3>



<p class="wp-block-paragraph">Most often a stale or wrong token, a filter predicate that matches nothing, or CDC never having been initialised so the connector has nothing to hand back. Run once without <code>ENABLE_CDC</code> against a small filtered slice and confirm you get data at all. That separates &#8220;no changes&#8221; from &#8220;not reading&#8221;.</p>



<h3 class="wp-block-heading">Pagination never terminates</h3>



<p class="wp-block-paragraph">Usually the SAP side enforcing its own page limit under your requested size, so the connector keeps requesting more. Reduce the page size, verify the filter is actually being applied server side rather than after retrieval, and have Basis confirm what Gateway is capping.</p>



<h3 class="wp-block-heading">Row counts drift over time</h3>



<p class="wp-block-paragraph">Almost always unhandled <code>DML_STATUS = DELETED</code> records, or a token invalidation that silently turned an incremental into a partial. Count distinct business keys in your target against a direct count in SAP for a bounded date range. Make that comparison a scheduled check, not something you run when someone complains.</p>



<h3 class="wp-block-heading">Duplicates after a rerun</h3>



<p class="wp-block-paragraph">A delta was consumed, the write failed, and the token advanced anyway. Store the token only after the write commits, never before. If you are appending rather than merging, add a dedupe on business key ordered by <code>GLUE_FETCH_SQ</code>.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Sharing one SAP service user across multiple pipelines, which couples their delta tokens together.</li>

<li>Automatic re-initialisation in an exception handler. It converts a loud failure into permanent silent drift.</li>

<li>Writing the token-carrier row into the target table as if it were data.</li>

<li>Appending CDC output and ignoring <code>DML_STATUS</code>.</li>

<li>Persisting the token before the write succeeds.</li>

<li>Assuming an entity is ODP-enabled because it looks analytical. Confirm it, because the incremental mechanism differs.</li>

<li>Leaving test subscriptions from <code>RODPS_REPL_TEST</code> registered against the queue.</li>

<li>Building the pipeline without ever asking Basis what the ODQ retention window is.</li>
</ul>



<h2 class="wp-block-heading">Best practices for SAP data integration with AWS Glue</h2>



<ul class="wp-block-list">
<li>Treat the delta token as production state. Versioned storage, a documented recovery procedure, and a written-down answer to &#8220;how do we backfill&#8221;.</li>

<li>One service user per pipeline, one connection per environment. Isolation is the cheapest fix for the worst failure.</li>

<li>Land raw, merge second. Keep the untouched connector output so you can replay a merge without going back to SAP.</li>

<li>Alert on absence, not just errors. Zero rows for several consecutive runs against a busy entity should page someone. Push Glue metrics into CloudWatch and forward them wherever you already look, whether that is Grafana Cloud, Datadog or your own Prometheus.</li>

<li>Reconcile on a schedule against SAP for a bounded window, and record the result. Drift you can see is a bug; drift you cannot see is an incident.</li>

<li>Start with a filtered slice and a small page size while you are building. Get it correct, then tune throughput.</li>

<li>Write down what each field means while the functional owner is still on the call. SAP field names do not explain themselves later.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does the AWS Glue SAP OData connector fall under SAP Note 3255746?</h3>



<p class="wp-block-paragraph">The note restricts the RFC modules of the ODP Data Replication API. The Glue connector uses OData over HTTPS against an activated Gateway service, which is the interface SAP directs customers toward. Read the current revision of the note yourself and get it confirmed with your SAP account team before you rely on any summary, including this one.</p>



<h3 class="wp-block-heading">Where should I store the delta token?</h3>



<p class="wp-block-paragraph">Anywhere durable and versioned that your job can write to transactionally: DynamoDB, Parameter Store, or S3 with versioning. What matters more than the store is the ordering. Write it only after the target write commits, and keep the previous value so you can roll back after a bad run.</p>



<h3 class="wp-block-heading">Can I run two Glue jobs against the same SAP entity?</h3>



<p class="wp-block-paragraph">Yes, but give them separate SAP service users. A delta token is scoped to the service collection and the user, so two jobs on one user will invalidate each other&#8217;s tokens whenever either re-initialises.</p>



<h3 class="wp-block-heading">Why do deleted SAP records still appear in my data lake?</h3>



<p class="wp-block-paragraph">Because the connector reports them rather than applying them. Deletes come back as rows with <code>DML_STATUS</code> set to <code>DELETED</code>. If your pipeline appends without a merge, those rows sit in the target looking like live data. Merge into Iceberg, or filter through a current-state view that every consumer actually uses.</p>



<h3 class="wp-block-heading">What page size should I use?</h3>



<p class="wp-block-paragraph">The default is 50,000 and the ceiling is 500,000. Larger pages mean fewer round trips but bigger responses for SAP to assemble, and Gateway or a load balancer may cut them short. Start at the default, raise it while watching for timeouts, and agree the ceiling with your Basis team.</p>



<h3 class="wp-block-heading">Can I partition an ODP extraction for speed?</h3>



<p class="wp-block-paragraph">Not with record-based partitioning. ODP entities handle pagination through skip and delta tokens, so concurrency is forced to null there regardless of your input. Field-based partitioning on a bounded range is available for non-ODP OData entities. For ODP, tune page size and narrow the entity instead.</p>



<h3 class="wp-block-heading">Can Glue write data back into SAP?</h3>



<p class="wp-block-paragraph">Yes. The SAP OData connector supports insert and update operations, with the external ID field named through <code>ID_FIELD_NAMES</code> on updates. Test that against a sandbox client with a functional owner watching, because a bad write into an ERP system is a very different class of incident to a bad read.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">SAP data integration with AWS Glue is not hard because the connector is difficult to configure. Configuration takes an afternoon. It is hard because the correctness of your pipeline depends on a token held in a queue on a system your team does not administer, subject to a retention policy you may not have been told about, and shared with every other subscriber using the same credentials.</p>



<p class="wp-block-paragraph">Design for that. Isolate the users, persist the token deliberately, apply deletes rather than collecting them, and alert on silence as loudly as you alert on errors. Do that and the connector is a good piece of engineering that will run for a long time without your attention. Skip it and you get a job that stays green while the numbers quietly stop being true.</p>



<h2 class="wp-block-heading">Need a hand with an SAP to AWS pipeline?</h2>



<p class="wp-block-paragraph">This is the kind of work I take on regularly. If you are standing one up, or you have one that has started drifting, I can help with:</p>



<ul class="wp-block-list">
<li>Designing delta token state management that survives failed runs, redeploys and concurrent pipelines.</li>

<li>Building merge logic that applies <code>DML_STATUS</code> deletes correctly into Iceberg or Redshift instead of accumulating ghost rows.</li>

<li>Diagnosing <code>RODPS_REPL_ODP_OPEN</code> failures, silent zero-row runs and row count drift, and working out how far back the damage goes.</li>

<li>Setting up SAP OData connections properly: Secrets Manager, least-privilege IAM, VPC connectivity and private certificate authorities.</li>

<li>Tuning page size, filters and job sizing so incremental runs cost what an incremental run should cost.</li>

<li>Adding reconciliation and absence alerting so drift shows up in a dashboard rather than in a finance report.</li>
</ul>



<p class="wp-block-paragraph">Send me a job log, a connection config, or the row counts that stopped matching, and I will tell you what I think is going on.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/sap-data-integration-aws-glue/">SAP Data Integration with AWS Glue: The Failure Modes Nobody Demos</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Zendesk Data Integration with AWS Glue Zero-ETL: The Delete Gap That Skews Your Numbers</title>
		<link>https://john-nessime.com/blog/technical-guides/zendesk-aws-glue-zero-etl/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 16 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Change Data Capture]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Customer Support Analytics]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Zendesk]]></category>
		<category><![CDATA[Zero-ETL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=223</guid>

					<description><![CDATA[<p>AWS Glue zero-ETL replicates seven Zendesk entities, but only three of them ever remove a row. Here is how that gap quietly skews CSAT and knowledge base counts, plus the three IAM layers to wire, the two settings you cannot change after creation, and the CloudWatch metrics that make drift visible before someone spots it in a meeting.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/zendesk-aws-glue-zero-etl/">Zendesk Data Integration with AWS Glue Zero-ETL: The Delete Gap That Skews Your Numbers</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The support lead pings you on a Tuesday. The CSAT figure on the exec dashboard is higher than the one in Zendesk Explore, and somebody noticed during the meeting. Nothing failed. The Glue integration is green. CloudWatch shows ingestion succeeding on every interval. And yet the warehouse and the source disagree, and have quietly disagreed for weeks.</p>



<p class="wp-block-paragraph">That divergence is not a bug. It is documented behaviour, sitting in one table in the AWS docs that almost everyone scrolls past. This post covers what Zendesk data integration with AWS Glue zero-ETL actually replicates, the entities where deletes silently never propagate, the configuration choices you cannot undo after creation, how to wire the three IAM roles involved, and how to monitor the thing so drift shows up as an alarm instead of a meeting.</p>



<h2 class="wp-block-heading">What Zendesk AWS Glue zero-ETL actually gives you</h2>



<p class="wp-block-paragraph">Zero-ETL is AWS&#8217;s name for managed replication. You create a Glue connection to Zendesk, pick the entities you want, pick a target, and Glue handles the initial snapshot, the schema mapping, and ongoing change data capture. No Spark job. No bookmark logic. No pagination code hammering the Zendesk API and no retry handling for it.</p>



<p class="wp-block-paragraph">Supported targets are a general purpose Amazon S3 bucket through the lakehouse architecture of Amazon SageMaker, S3 Tables through the same lakehouse, Redshift Managed Storage, or an Amazon Redshift data warehouse directly. Data lands as Apache Iceberg, which is what makes row-level updates and deletes viable on object storage at all.</p>



<p class="wp-block-paragraph">What you give up is control. No transform step, no server-side filter, no column pruning at ingest. You get the entity as the connector sees it and you shape it downstream. For most support analytics that trade is fine. It stops being fine the moment your compliance team asks why deleted records are still queryable.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The delete gap: four of seven entities never remove a row</h2>



<p class="wp-block-paragraph">AWS Glue zero-ETL supports seven Zendesk entities. Only three of them replicate deletes.</p>



<ul class="wp-block-list">
<li><strong>tickets</strong> &#8211; create, update and delete all propagate</li>



<li><strong>users</strong> &#8211; create, update and delete all propagate</li>



<li><strong>organizations</strong> &#8211; create, update and delete all propagate</li>



<li><strong>satisfaction-rating</strong> &#8211; create and update only, <strong>no delete</strong></li>



<li><strong>articles</strong> &#8211; create and update only, <strong>no delete</strong></li>



<li><strong>calls</strong> &#8211; create and update only, <strong>no delete</strong></li>



<li><strong>legs</strong> (call legs) &#8211; create and update only, <strong>no delete</strong></li>
</ul>



<p class="wp-block-paragraph">When a satisfaction rating disappears in Zendesk, the row stays in your lakehouse. Permanently. Nothing errors, <code>IngestionSucceeded</code> fires as usual, and the row count in that table only ever goes up.</p>



<p class="wp-block-paragraph">Now think about what your CSAT query does. It averages a score column across a set of rows. If removed ratings never leave the target, your average drifts further from the source every cycle, and the skew is rarely random because retracted or disputed ratings are exactly the ones most likely to be removed. The same mechanism inflates knowledge base article counts, call volume, and any call leg analysis you build. It stays invisible until two dashboards get compared side by side.</p>



<h3 class="wp-block-heading">How to design around it</h3>



<p class="wp-block-paragraph">You cannot make the connector track deletes it does not track. So stop trying, and split the problem instead.</p>



<ol class="wp-block-list">
<li><strong>Run two integrations, not one.</strong> Put tickets, users and organizations in a continuously synced integration. Put the four append-only entities in a second integration you can tear down and rebuild on a cadence that matches how much drift you can tolerate.</li>



<li><strong>Treat the append-only tables as an event log, not as state.</strong> Query them through a view that reconciles against a periodic authoritative count, rather than trusting row presence to mean the record still exists.</li>



<li><strong>Prefer status fields over row existence.</strong> Where an entity exposes a field describing its state, filter on that field downstream. Check which fields your entities actually return before you build on this, because coverage varies by entity.</li>



<li><strong>Reconcile on a schedule.</strong> Pull counts from the Zendesk API for the four entities and compare against the target. Alert on divergence, not on an absolute number.</li>
</ol>



<p class="wp-block-paragraph">Splitting the integration costs you a second set of IAM wiring and a second thing to monitor. It buys you the ability to rebuild the drifting half without touching the half that is correct. That is a good trade.</p>



<h2 class="wp-block-heading">Entity coverage is narrower than the Zendesk API</h2>



<p class="wp-block-paragraph">Seven entities is not the Zendesk API. Ticket comments, ticket metrics, ticket audits, groups, views, macros and SLA policy definitions are not part of the zero-ETL entity set. If your reporting needs first response time, the audit trail behind a status change, or custom field definitions, zero-ETL alone will not get you there.</p>



<p class="wp-block-paragraph">The useful part is that the Glue connection you create is reusable. The same connection backs a normal Glue ETL job, so you can run zero-ETL for the bulk entities and a scheduled Spark job for the ones it does not cover:</p>



<pre class="wp-block-code"><code># Read a Zendesk entity through the same connection
# used by the zero-ETL integration. ENTITY_NAME takes
# the entity name, not the label, e.g. "tickets".
zendesk_read = glueContext.create_dynamic_frame.from_options(
    connection_type="Zendesk",
    connection_options={
        "connectionName": "my-zendesk-connection",
        "ENTITY_NAME": "tickets",
        "API_VERSION": "v2"
    }
)</code></pre>



<p class="wp-block-paragraph">One schema detail worth knowing before you write a single downstream query: the connector converts struct and list types to strings. Zendesk custom fields arrive as arrays of objects, so they land as serialized text, not as a nested column you can address with dot notation. Partitioning is also not supported on the Zendesk source. Plan for a parse step in your silver layer rather than discovering it in a broken dashboard.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Two decisions you cannot change after creation</h2>



<p class="wp-block-paragraph">Most Glue settings are editable. These two are not, and getting them wrong means deleting the integration and starting over.</p>



<h3 class="wp-block-heading">Continuous sync versus on-demand snapshot</h3>



<p class="wp-block-paragraph">By default the integration syncs continuously. Enable the on-demand snapshot setting and it does a single one-time replication with no ongoing CDC. That setting is locked once the integration exists. If you enabled it for a proof of concept and then decided you wanted CDC, you are recreating the integration and re-running the full load.</p>



<h3 class="wp-block-heading">Refresh interval, and why your target choice locks it</h3>



<p class="wp-block-paragraph">The refresh interval controls how often CDC pulls run. It accepts anything from 15 minutes up to 8640 minutes, which is six days. Here is the part that catches people:</p>



<ul class="wp-block-list">
<li>If the target is <strong>Amazon Redshift</strong>, the refresh interval cannot be modified after creation.</li>



<li>For <strong>other targets</strong>, including the SageMaker lakehouse on S3, you can change it later.</li>
</ul>



<p class="wp-block-paragraph">So the target you pick is not only an architecture decision, it is a flexibility decision. If you are not certain what freshness the business actually needs, land in the lakehouse first and query Redshift over it. You keep the ability to tune the interval once you have real usage data instead of a guess made in week one.</p>



<p class="wp-block-paragraph">On cost: AWS does not charge separately for the integration itself. You pay for everything around it, target storage, Redshift or Athena query compute, S3 requests, Data Catalog usage and CloudWatch logs. A 15 minute interval on seven entities produces far more small-file churn and log volume than a 6 hour one. Set the interval from the freshness a human will actually act on, not from the smallest number the field accepts.</p>



<h2 class="wp-block-heading">Wiring the three roles</h2>



<p class="wp-block-paragraph">There are three separate permission layers here and they fail in different ways, which is why &#8220;check IAM&#8221; is useless advice on its own.</p>



<ul class="wp-block-list">
<li><strong>The Zendesk OAuth credential.</strong> The connector uses the authorization code grant, so you get redirected to Zendesk to log in and approve. You can rely on the Glue-managed client application and supply only your instance URL, or register your own OAuth app in the Zendesk admin center and provide your own client ID and secret. The resulting access token does not expire, which is convenient and also means a revoked app in Zendesk is something you will only discover through failed ingestions.</li>



<li><strong>The source role.</strong> This is what lets the integration read through the connection. It is a prerequisite for SaaS sources and it is the step most people miss, because creating the connection successfully does not mean the integration can use it.</li>



<li><strong>The target role and catalog policy.</strong> This is what lets the integration write. A misconfigured catalog resource policy puts the integration into <code>NEEDS_ATTENTION</code>, not into a clear error at creation time.</li>
</ul>



<p class="wp-block-paragraph">The source role policy looks like this. Note <code>glue:RefreshOAuth2Tokens</code>, which is the one people leave out and then spend an afternoon debugging:</p>



<pre class="wp-block-code"><code>{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "GlueConnections",
            "Effect": "Allow",
            "Action": [
                "glue:GetConnections",
                "glue:GetConnection"
            ],
            "Resource": [
                "arn:aws:glue:*:111122223333:catalog",
                "arn:aws:glue:us-east-1:111122223333:connection/*"
            ]
        },
        {
            "Sid": "GlueActionBasedPermissions",
            "Effect": "Allow",
            "Action": [
                "glue:ListEntities",
                "glue:RefreshOAuth2Tokens"
            ],
            "Resource": ["*"]
        },
        {
            "Sid": "CloudWatchLogging",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": ["*"]
        }
    ]
}</code></pre>



<p class="wp-block-paragraph">The trust policy allows <code>glue.amazonaws.com</code> to assume it. Then you attach the role to the connection, which is a CLI-only step and does not appear in the console flow:</p>



<pre class="wp-block-code"><code>aws glue create-integration-resource-property 
  --resource-arn arn:aws:glue:us-east-1:123456789012:connection/my-zendesk-connection 
  --source-processing-properties "{"RoleArn" : "arn:aws:iam::123456789012:role/zendesk-zetl-source"}" 
  --region us-east-1</code></pre>



<p class="wp-block-paragraph">If you skip this, the connection tests fine and the integration still cannot read. That mismatch is the single most common reason a first attempt sits in <code>NEEDS_ATTENTION</code> with no obvious cause.</p>



<h2 class="wp-block-heading">Setting up the integration</h2>



<ol class="wp-block-list">
<li>If you are using your own OAuth app, register it in the Zendesk admin center under the API settings and note the client ID and secret. Store the secret in AWS Secrets Manager, one secret per Glue connection.</li>



<li>In Glue Studio, create a connection under Data Connections. Choose Zendesk as the connection type, supply your instance URL and environment, and complete the OAuth redirect.</li>



<li>Create the source role with the policy above and attach it to the connection using <code>create-integration-resource-property</code>.</li>



<li>Prepare the target: the Glue database or S3 Table bucket, the target role, the catalog resource policy, and any Lake Formation grants your account&#8217;s permission model requires.</li>



<li>Create the integration. Select entities, set the refresh interval, and decide continuous sync versus on-demand snapshot. Both of those are locked from here.</li>



<li>Watch the first full load complete before you point anything at the tables. Full load and CDC are separate load types in the metrics, and the numbers look very different.</li>
</ol>



<p class="wp-block-paragraph">Once it works, move it into code. Glue zero-ETL integrations are supported by CloudFormation and the AWS CDK, which is the difference between a thing one person built in the console and a thing your team can redeploy into another account.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Monitoring: the metrics that expose drift</h2>



<p class="wp-block-paragraph">Zero-ETL publishes to the <code>AWS/Glue/ZeroETL</code> CloudWatch namespace, with dimensions for <code>integrationArn</code>, <code>loadType</code> and <code>tableName</code>. The metrics are <code>InsertCount</code>, <code>UpdateCount</code>, <code>DeleteCount</code>, <code>IngestionSucceeded</code>, <code>IngestionFailed</code> and <code>LastSyncTimestamp</code>.</p>



<p class="wp-block-paragraph">The obvious alarm is on failure:</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-alarm 
  --alarm-name zendesk-zetl-ingestion-failed 
  --namespace AWS/Glue/ZeroETL 
  --metric-name IngestionFailed 
  --dimensions Name=integrationArn,Value=&lt;your-integration-arn&gt; 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 1 
  --threshold 1 
  --comparison-operator GreaterThanOrEqualToThreshold 
  --treat-missing-data notBreaching</code></pre>



<p class="wp-block-paragraph">That one catches loud failures. It does not catch the quiet one, which is an integration that stops running entirely. For that, alarm on <code>IngestionSucceeded</code> with <code>--treat-missing-data breaching</code> over a window longer than your refresh interval. Absence of success is the signal, not presence of failure.</p>



<p class="wp-block-paragraph">And here is the one specific to this post. Because <code>DeleteCount</code> carries a <code>tableName</code> dimension, you can see the delete gap directly in a graph. Plot <code>DeleteCount</code> per table. Tickets, users and organizations will show a nonzero line. Satisfaction ratings, articles, calls and legs will sit flat at zero forever. That flat line is not a broken metric, it is the behaviour, and having it on a dashboard is the cheapest way to keep the whole team aware of it.</p>



<p class="wp-block-paragraph">Glue also writes a system table into the target database recording the outcome of each full load and CDC run, with per-run record, insert and delete counts. Query that when CloudWatch is not granular enough. If you already run Grafana Cloud or Datadog, pulling the <code>AWS/Glue/ZeroETL</code> namespace in alongside your existing infrastructure dashboards puts pipeline health next to everything else on call already watches.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<p class="wp-block-paragraph">Integrations move through Creating, Active, Modifying, Syncing, Needs attention, Failed and Deleting. What each one means in practice:</p>



<ul class="wp-block-list">
<li><strong>Syncing</strong> means it hit a recoverable error and is re-seeding data. Not an emergency. Let it finish before you touch anything.</li>



<li><strong>Needs attention</strong> means you have to fix something, usually connection credentials, the source role, the target role, or the catalog resource policy. Once you fix it, there is no manual recovery action. Glue retries automatically on an exponential backoff schedule, so the gap between attempts grows over time. If you fixed a policy and nothing happened after two minutes, that is expected. Wait it out.</li>



<li><strong>Failed</strong> is terminal. Delete the integration and recreate it. There is no repair path.</li>
</ul>



<p class="wp-block-paragraph">For anything else, the CloudWatch logs emitted after each full load or CDC run carry the actual root cause. The console status is a summary, and a fairly lossy one. Read the logs first.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Building CSAT, article count or call volume metrics on raw row counts from the four entities that never delete.</li>



<li>Creating the connection and assuming the integration can use it. The source role attachment is a separate CLI step.</li>



<li>Choosing Redshift as the direct target during a proof of concept and locking a refresh interval you picked arbitrarily.</li>



<li>Setting a 15 minute interval because it was available, then being surprised by small-file churn and CloudWatch log volume.</li>



<li>Writing downstream queries that address Zendesk custom fields as nested columns. They arrive as strings.</li>



<li>Alarming only on <code>IngestionFailed</code>, so a stalled integration produces silence and silence looks like health.</li>



<li>Reaching for zero-ETL when the requirement is ticket comments or ticket metrics, which are not in the entity set at all.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Split delete-tracking and append-only entities into separate integrations so you can rebuild one without disturbing the other.</li>



<li>Land in the SageMaker lakehouse rather than straight into Redshift unless you are certain about freshness. It keeps the refresh interval editable.</li>



<li>Put <code>DeleteCount</code> per table on a dashboard. It makes an invisible behaviour visible to everyone, not just whoever read the docs.</li>



<li>Define the integration in CloudFormation or CDK once it works, so the next environment is a deploy rather than a repeat of the console clicking.</li>



<li>Keep a silver layer between the replicated tables and your BI tool. That is where you parse the stringified structs and apply the reconciliation logic, and it means schema surprises break one view instead of every dashboard.</li>



<li>Document which entities do not track deletes somewhere your analysts will actually read, next to the tables, not in a wiki page nobody opens.</li>
</ul>



<h3 class="wp-block-heading">When to use something else</h3>



<p class="wp-block-paragraph">Zero-ETL is a good fit when your target is already AWS, your entities are in the supported set, and you want AWS to own the pipeline. It is the wrong fit if you need entities outside that list, need transformation at ingest, or need the same Zendesk data landing somewhere that is not AWS. Managed connector platforms like Fivetran or Airbyte cover a wider slice of the Zendesk API and will replicate to non-AWS destinations, at the cost of another vendor relationship, another bill, and data leaving your account boundary on the way through. If the seven entities cover your reporting, zero-ETL is cheaper and simpler. If they do not, forcing it is worse than paying for a tool that fits.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Which Zendesk entities does AWS Glue zero-ETL support?</h3>



<p class="wp-block-paragraph">Tickets, users, organizations, satisfaction ratings, articles, calls and call legs. Tickets, users and organizations replicate creates, updates and deletes. The other four replicate creates and updates only.</p>



<h3 class="wp-block-heading">Can I get ticket comments or ticket metrics through zero-ETL?</h3>



<p class="wp-block-paragraph">No. They are not in the entity set. Use a Glue ETL job against the same Zendesk connection, or a connector platform with broader API coverage, and join the result to your replicated tables downstream.</p>



<h3 class="wp-block-heading">How fresh can the data be?</h3>



<p class="wp-block-paragraph">The refresh interval goes down to 15 minutes and up to six days. This is near real time replication, not streaming. If you need sub-minute latency on Zendesk events, zero-ETL is the wrong mechanism and you want webhooks into an event pipeline instead.</p>



<h3 class="wp-block-heading">Can I change the refresh interval later?</h3>



<p class="wp-block-paragraph">Only if the target is not Redshift. With Redshift as the target the interval is fixed at creation. With the SageMaker lakehouse and other targets you can modify it afterwards.</p>



<h3 class="wp-block-heading">Why is my integration stuck in NEEDS_ATTENTION after I fixed the permissions?</h3>



<p class="wp-block-paragraph">Because recovery is automatic but not immediate. Glue retries on exponential backoff, so if the integration has been unhealthy for a while the next retry may be some way off. There is no manual recovery command. Confirm the fix is correct, then wait.</p>



<h3 class="wp-block-heading">Does zero-ETL cost extra on top of Glue?</h3>



<p class="wp-block-paragraph">AWS does not bill a separate charge for the integration itself. You pay for the surrounding services: target storage, Redshift or Athena query compute, S3 requests, Data Catalog usage and CloudWatch logs. Refresh interval is the main lever, since it drives how many small commits and log entries you generate.</p>



<h3 class="wp-block-heading">Why do my Zendesk custom fields look like text?</h3>



<p class="wp-block-paragraph">The connector converts struct and list types to strings. Custom fields come through as serialized values rather than nested columns, so parse them in a downstream layer instead of querying them directly.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">Zendesk AWS Glue zero-ETL removes a genuinely tedious pipeline from your plate. The setup is fiddly in the IAM layer and then it mostly runs itself, which is exactly the point of a managed integration.</p>



<p class="wp-block-paragraph">The one thing worth carrying out of this post: four of the seven supported entities never replicate a delete. Satisfaction ratings, articles, calls and call legs only grow. Nothing warns you, every metric stays green, and your numbers drift a little further from Zendesk with every cycle. Design for that on day one, put <code>DeleteCount</code> per table on a dashboard, and you will not be the person explaining the discrepancy in a meeting six months from now.</p>



<h2 class="wp-block-heading">Need help with your Zendesk to AWS data pipeline?</h2>



<p class="wp-block-paragraph">I work with teams building and fixing replication pipelines on AWS. On this kind of setup, that usually means:</p>



<ul class="wp-block-list">
<li>Auditing an existing Zendesk zero-ETL integration and quantifying how far the target has drifted from the source</li>



<li>Designing the split between continuously synced and append-only entities, including the rebuild cadence and reconciliation queries</li>



<li>Untangling the source role, target role and catalog policy layers when an integration sits in <code>NEEDS_ATTENTION</code> with no clear cause</li>



<li>Building the silver layer that parses stringified Zendesk custom fields into something your BI tool can actually use</li>



<li>Setting up CloudWatch alarms and dashboards that catch a stalled integration, not just a failed one</li>



<li>Moving a console-built integration into CloudFormation or CDK so it can be deployed across accounts</li>
</ul>



<p class="wp-block-paragraph">Send me the integration status, a CloudWatch log excerpt, or the query that is giving you the wrong number, and I will tell you what I think is going on.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/zendesk-aws-glue-zero-etl/">Zendesk Data Integration with AWS Glue Zero-ETL: The Delete Gap That Skews Your Numbers</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>It Works Isn&#8217;t Ready: A Production Checklist for Salesforce AWS Integrations</title>
		<link>https://john-nessime.com/blog/devops/salesforce-aws-integration-checklist/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 03 Aug 2026 21:53:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=107</guid>

					<description><![CDATA[<p>"It works" and "it's ready" are different claims. Integrations rarely fail on launch day; they fail in week six, when the token expires and someone adds a field. A readiness checklist for Salesforce AWS integrations, with a pass test on every item.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-aws-integration-checklist/">It Works Isn&#8217;t Ready: A Production Checklist for Salesforce AWS Integrations</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone asks whether the integration is ready to go live. The answer that comes back is &#8220;it works.&#8221; Those are different statements, and the gap between them is where the next three months of interruptions live.</p>



<p class="wp-block-paragraph">Integrations very rarely fail on launch day. They fail in week six. That is roughly when the OAuth token first expires, when an admin adds a field nobody mentioned, when a second tool starts consuming the same API allowance, and when enough records have been deleted for the row counts to visibly diverge. None of those conditions exist in staging, which is why staging said yes.</p>



<p class="wp-block-paragraph">So this is a <strong>Salesforce AWS integration checklist</strong> aimed at week six rather than at Tuesday. Every item has a pass test, because &#8220;we thought about that&#8221; is not the same as &#8220;we checked.&#8221;</p>



<h2 class="wp-block-heading">1. Credentials and identity</h2>



<ul class="wp-block-list">
<li><strong>The integration has its own Salesforce user.</strong> <em>Pass: the username is not a person&#8217;s, and nobody uses it interactively.</em></li>
<li><strong>That user is API-only and cannot log in through the UI.</strong> <em>Pass: someone tried and was refused.</em></li>
<li><strong>Its profile grants only the objects and fields you actually read.</strong> <em>Pass: you have the field-level permission list and can explain every entry.</em></li>
<li><strong>Credentials live in a secrets manager, not in environment variables or config files.</strong> <em>Pass: the secret is retrievable by ARN and nothing in the repo contains it.</em></li>
<li><strong>You have rotated the credential at least once, in production, and the pipeline survived.</strong> <em>Pass: there is a dated record of the rotation and no incident attached to it.</em></li>
<li><strong>You know what invalidates the refresh token.</strong> <em>Pass: you can name the events that revoke it, including password resets and session policy changes on the integration user.</em></li>
</ul>



<p class="wp-block-paragraph">That last item is the classic week-four failure. A refresh token feels permanent right up until an admin does something routine to the user account it belongs to.</p>



<h2 class="wp-block-heading">2. Network, versions and deadlines</h2>



<ul class="wp-block-list">
<li><strong>You know whether the compute needs to be in a VPC, and why.</strong> <em>Pass: if it is, you can name the private resource it reaches.</em></li>
<li><strong>If it is in a VPC, S3 traffic goes through a gateway endpoint.</strong> <em>Pass: the endpoint exists and the route table references it.</em></li>
<li><strong>The Salesforce API version in your endpoints is pinned and current.</strong> <em>Pass: it is not a version on a retirement list.</em></li>
<li><strong>IAM permissions are scoped to the specific resources involved.</strong> <em>Pass: no wildcard on S3 buckets or Secrets Manager paths.</em></li>
<li><strong>Login IP restrictions and connected app policies are configured deliberately.</strong> <em>Pass: you know whether IP relaxation is on and made that choice on purpose.</em></li>
</ul>



<p class="wp-block-paragraph">The API version item has real dates attached, so it is worth stating plainly rather than leaving as a vague &#8220;keep current&#8221;. Salesforce retired versions 7.0 through 20.0 in Summer &#8217;22 and versions 21.0 through 30.0 in Summer &#8217;25, and has announced that 31.0 through 40.0 retire in June 2028, with the SOAP <code>login()</code> call retiring separately in June 2027. Retired versions return HTTP 410 on REST rather than degrading gracefully.</p>



<p class="wp-block-paragraph">Usefully, Salesforce warns you first. Calls to legacy versions come back with a <code>Warning</code> header, so this is detectable today rather than on the morning it breaks:</p>



<pre class="wp-block-code"><code># If this matches anything, you have a deadline you did not know about.
grep -i "Warning:" integration.log | sort -u</code></pre>



<h2 class="wp-block-heading">3. Correctness</h2>



<ul class="wp-block-list">
<li><strong>Deletes are handled, not just creates and updates.</strong> <em>Pass: you can point at the code path and name the mechanism it uses.</em></li>
<li><strong>Merged records do not double-count.</strong> <em>Pass: you have checked what happens to the losing record of a merge.</em></li>
<li><strong>Formula and computed fields are either recalculated downstream or documented as snapshots.</strong> <em>Pass: there is a written list of which columns are derived.</em></li>
<li><strong>Re-running the same window produces the same result.</strong> <em>Pass: you ran it twice and diffed the output.</em></li>
<li><strong>One real aggregate reconciles against Salesforce for a closed period.</strong> <em>Pass: daily and monthly totals both match, not just daily.</em></li>
<li><strong>Timezone handling is explicit at the boundary.</strong> <em>Pass: someone can say which timezone the date column is in without guessing.</em></li>
</ul>



<p class="wp-block-paragraph">If daily figures reconcile and monthly ones do not, that is a timezone problem at period boundaries, not missing data. Find it now, because finding it inside somebody&#8217;s board pack is a worse day.</p>



<h2 class="wp-block-heading">4. Failure handling</h2>



<ul class="wp-block-list">
<li><strong>Retries are scoped to failures that deserve them.</strong> <em>Pass: transient errors retry, data errors do not, and you can show which is which.</em></li>
<li><strong>Backoff is exponential with jitter.</strong> <em>Pass: a fixed sleep does not appear anywhere in the retry path.</em></li>
<li><strong>A partial failure leaves recoverable state.</strong> <em>Pass: the watermark advances only after data lands, not when the API call returns.</em></li>
<li><strong>You can replay an arbitrary historical window on demand.</strong> <em>Pass: someone has done it, in production, with a command you could hand to a colleague.</em></li>
<li><strong>Rate limiting is treated as an expected condition.</strong> <em>Pass: a 429 or a limit error produces a backoff, not an alert and a failed run.</em></li>
</ul>



<h2 class="wp-block-heading">5. Observability</h2>



<ul class="wp-block-list">
<li><strong>The pipeline emits a metric on successful completion.</strong> <em>Pass: the metric exists and you can graph it.</em></li>
<li><strong>An alarm fires when that metric stops arriving.</strong> <em>Pass: you disabled the schedule on purpose and the alarm went red.</em></li>
<li><strong>Data freshness is measured on the destination side.</strong> <em>Pass: you can answer &#8220;how far behind are we right now&#8221; from a dashboard.</em></li>
<li><strong>Salesforce API allowance consumption is graphed.</strong> <em>Pass: there is a percentage on a chart with a threshold line.</em></li>
<li><strong>Alerts describe symptoms and link to a runbook.</strong> <em>Pass: the alarm description contains a URL somebody could follow at 3am.</em></li>
</ul>



<p class="wp-block-paragraph">The second item is the one people skip and it is the only one that matters. An alarm you have never seen fire is a belief, not a control.</p>



<h2 class="wp-block-heading">6. Cost</h2>



<ul class="wp-block-list">
<li><strong>Every resource carries a cost allocation tag.</strong> <em>Pass: Cost Explorer filtered to that tag returns a number.</em></li>
<li><strong>Log groups have a retention period.</strong> <em>Pass: none of them say &#8220;Never expire&#8221;.</em></li>
<li><strong>The schedule matches how the output is actually used.</strong> <em>Pass: somebody named the decision the data supports and the latency it tolerates.</em></li>
<li><strong>Average object size in the destination is sensible.</strong> <em>Pass: you have looked, and it is not thousands of tiny files per partition.</em></li>
<li><strong>A budget alert exists on the pipeline&#8217;s tag.</strong> <em>Pass: it has a threshold and a recipient who is not on holiday.</em></li>
</ul>



<h2 class="wp-block-heading">7. Operations and handover</h2>



<ul class="wp-block-list">
<li><strong>The integration has a named owner.</strong> <em>Pass: a person, not a team inbox, and they know.</em></li>
<li><strong>A runbook exists covering the three most likely failures.</strong> <em>Pass: someone who did not build it followed the runbook successfully.</em></li>
<li><strong>Infrastructure is defined as code.</strong> <em>Pass: you could rebuild it in a fresh account from the repository.</em></li>
<li><strong>Schema drift produces a notification.</strong> <em>Pass: adding a field in a sandbox generated an alert somewhere.</em></li>
<li><strong>Personal data has an access policy and a retention policy.</strong> <em>Pass: both are written down and somebody outside the team has read them.</em></li>
<li><strong>You know what a Salesforce sandbox refresh does to it.</strong> <em>Pass: you have been through one, or you know what will break when you do.</em></li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">If you only do five</h2>



<p class="wp-block-paragraph">Nobody clears a thirty-item list before a deadline. These five catch most of what actually goes wrong:</p>



<ol class="wp-block-list">
<li><strong>Rotate the credential once before go-live.</strong> It proves the whole credential path works and it is the failure with the longest fuse.</li>
<li><strong>Break it deliberately and confirm someone is told.</strong> Disable the schedule, watch the alarm, put it back.</li>
<li><strong>Reconcile one real number</strong> against Salesforce for a closed period, daily and monthly.</li>
<li><strong>Set log retention and one cost allocation tag.</strong> Five minutes, and it is the difference between a knowable bill and a mystery.</li>
<li><strong>Write the runbook and name the owner.</strong> If it is nobody&#8217;s, it is nobody&#8217;s at 3am too.</li>
</ol>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">What actually breaks first in production?</h3>



<p class="wp-block-paragraph">Credentials, usually around the first expiry or the first time an admin touches the integration user. After that, schema drift and API allowance contention with a tool somebody else added.</p>



<h3 class="wp-block-heading">How do I know if I&#8217;m on a retiring API version?</h3>



<p class="wp-block-paragraph">Look at the version string in your endpoint URLs, and search your logs for the <code>Warning</code> header Salesforce returns on legacy versions. Retired versions fail with an HTTP 410 rather than degrading, so there is no soft landing.</p>



<h3 class="wp-block-heading">Is a staging environment enough to sign off?</h3>



<p class="wp-block-paragraph">No, because the conditions that break integrations barely exist there: real volume, real credential lifecycles, real deletions, and a shared API budget. Staging proves the code runs. It cannot prove the thing survives contact with an organisation.</p>



<h3 class="wp-block-heading">How often should I revisit the list?</h3>



<p class="wp-block-paragraph">Quarterly for the alarm test and the reconciliation, annually for permissions, credentials and API versions. Put both in a calendar, because an integration that works is one nobody thinks about until it does not.</p>



<h3 class="wp-block-heading">Does this apply to managed connectors too?</h3>



<p class="wp-block-paragraph">Most of it. You inherit retries and pagination from the vendor, but credentials, permissions, reconciliation, cost, alerting and ownership are all still yours. A managed connector removes code, not accountability.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">&#8220;It works&#8221; is a statement about today. &#8220;It&#8217;s ready&#8221; is a claim about week six, and the only honest way to make it is to have tested the things that only happen later: rotate the credential, break the schedule, reconcile a number, follow the runbook.</p>



<p class="wp-block-paragraph">If you cannot point at evidence for an item, it is not done. It is intended, which is a different thing, and the gap between the two is where you will spend your next quarter.</p>



<h2 class="wp-block-heading">Want a second pair of eyes before go-live?</h2>



<p class="wp-block-paragraph">Readiness reviews are cheap compared with the incidents they prevent, and they go faster with someone who has seen the same six failures repeatedly. Work I take on:</p>



<ul class="wp-block-list">
<li>Pre-launch readiness review against this checklist, with evidence collected rather than assurances taken.</li>
<li>Credential and permission audit: integration users, profiles, field-level security, secret storage and rotation.</li>
<li>Reconciliation harness so row counts and aggregates are checked automatically rather than when someone complains.</li>
<li>Alerting and runbooks, including a failure drill that proves the alerts reach a human.</li>
<li>Legacy API version audits ahead of the announced retirement deadlines.</li>
<li>Cost tagging, log retention and budget alerts so the first month&#8217;s bill is not a surprise.</li>
</ul>



<p class="wp-block-paragraph">Tell me which items on this list you have evidence for, and I will tell you which of the rest matter most for your setup.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-aws-integration-checklist/">It Works Isn&#8217;t Ready: A Production Checklist for Salesforce AWS Integrations</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Run It Twice, Get Two Answers: Building an ETL Pipeline From Zoho CRM to Amazon S3</title>
		<link>https://john-nessime.com/blog/devops/zoho-crm-to-amazon-s3/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 02 Aug 2026 18:16:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[Zoho]]></category>
		<category><![CDATA[Zoho CRM]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=86</guid>

					<description><![CDATA[<p>You re-run the same extract for the same window and get a different set of rows. Nothing errored. You were paginating a result set that kept changing while you read it. Here's how to build a Zoho CRM to S3 pipeline whose runs are repeatable, from closed read windows to Bulk Read and deletions.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/zoho-crm-to-amazon-s3/">Run It Twice, Get Two Answers: Building an ETL Pipeline From Zoho CRM to Amazon S3</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The monthly deal count in your warehouse is short by about thirty rows. You re-run the extract for exactly the same window, expecting to confirm the bug, and this time you get a different thirty. Run it a third time and the number moves again.</p>



<p class="wp-block-paragraph">Nothing is broken in the way you are looking for. The records exist in Zoho. Your job did not error. What happened is that you asked for every deal modified since your last watermark, sorted, and then walked through it page by page while sales reps kept working. Records got modified during the read, changed their position in the sort order, and shifted from page 4 to page 2 after you had already read page 2. So you never saw them.</p>



<p class="wp-block-paragraph">That is the failure that defines building an <strong>ETL pipeline from Zoho CRM to Amazon S3</strong>, and it does not look like a bug. It looks like flaky data. It only shows up on busy modules, it never produces an error, and re-running the job produces a different wrong answer, which is the single most confusing symptom in data engineering.</p>



<p class="wp-block-paragraph">The fix is one rule, and everything else in this post follows from it. This covers that rule, which extraction API to use and when, authentication and the data centre trap that wastes an afternoon, handling deletions (Zoho is genuinely good here), shaping nested CRM JSON for S3, and staying inside your API credits.</p>



<h2 class="wp-block-heading">The rule: bound both ends of the window</h2>



<p class="wp-block-paragraph">Almost every incremental extract is written like this:</p>



<pre class="wp-block-code"><code>Modified_Time &gt; {last_run}</code></pre>



<p class="wp-block-paragraph">That query has no upper bound, which means the result set keeps growing while you read it. You are paginating a moving target. Add an upper bound and put it slightly in the past:</p>



<pre class="wp-block-code"><code>Modified_Time &gt; {last_run} AND Modified_Time &lt; {now_minus_lag}</code></pre>



<p class="wp-block-paragraph">Now the set is frozen. Records modified during your run land after the upper bound and get picked up next time. The extract becomes repeatable: run it twice, get the same rows twice. That property is worth more than any amount of retry logic, because it means a failed run costs you nothing and a suspicious number can be checked by re-running.</p>



<p class="wp-block-paragraph">Two details that matter. The lag needs to comfortably exceed how long your extract takes plus any clock skew between you and Zoho; ten minutes is a sensible starting point and costs you ten minutes of freshness. And <strong>only advance the watermark after the entire window has landed in S3</strong>, never after the API call succeeds. Those are different moments, and the gap between them is where data goes missing.</p>



<pre class="wp-block-code"><code>#!/usr/bin/env bash
set -euo pipefail

LAG_MINUTES=10
T1=$(cat state/deals.watermark)
T2=$(date -u -d "-${LAG_MINUTES} minutes" +%Y-%m-%dT%H:%M:%S+00:00)

extract_window "$T1" "$T2"

# Watermark advances only once the data is durably in S3.
echo "$T2" &gt; state/deals.watermark</code></pre>



<p class="wp-block-paragraph">Keep that state somewhere durable and versioned, not on the box running the job. DynamoDB, Parameter Store, or a small object in S3 all work. A watermark file on an ephemeral runner is a watermark you will lose.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Authentication, and the trap that costs an afternoon</h2>



<p class="wp-block-paragraph">Zoho uses OAuth 2.0. You register a client in the Zoho API Console, get a client ID and secret, exchange a grant token for a refresh token once, and then trade the refresh token for short-lived access tokens from then on. Access tokens last an hour, so cache them rather than requesting one per call.</p>



<p class="wp-block-paragraph">Here is the part that catches everyone: <strong>Zoho runs multiple data centres and they are separate worlds.</strong> An account in the EU data centre authenticates against a different accounts domain and calls a different API domain than a US account. A token issued in one region is meaningless in another, and the error you get back does not say &#8220;wrong data centre&#8221;, it says the token is invalid.</p>



<p class="wp-block-paragraph">So make the domain a configuration value from the first line of code, never a hardcoded string. The examples below use the US domain. If your org lives elsewhere, both the accounts domain and the API domain change together.</p>



<p class="wp-block-paragraph">Two more things worth getting right at the start. Request the narrowest OAuth scopes that work, read-only on the modules you actually extract, because scopes are easy to widen later and awkward to explain in an audit. And store the refresh token in Secrets Manager rather than an environment variable, since it does not expire and is effectively a permanent key to your CRM.</p>



<h2 class="wp-block-heading">Which API to extract with</h2>



<p class="wp-block-paragraph">Three options, and the choice is mostly about volume.</p>



<h3 class="wp-block-heading">Bulk Read, for anything large</h3>



<p class="wp-block-paragraph">This is the right default for a lake. You POST a job description, Zoho runs the export server-side, and you poll for status or supply a callback URL. When it finishes you get a download URL for a ZIP of CSV. A single job handles up to 200,000 records, with paging beyond that, and it does not consume your standard API limits the way record-by-record calls do.</p>



<p class="wp-block-paragraph">The important property for our purposes: the pagination problem disappears. Zoho assembles the export on its side, so there is no page-by-page read for records to slip between.</p>



<pre class="wp-block-code"><code>curl "https://www.zohoapis.com/crm/bulk/v8/read" 
  -X POST 
  -H "Authorization: Zoho-oauthtoken $ACCESS_TOKEN" 
  -H "Content-Type: application/json" 
  -d @job.json</code></pre>



<pre class="wp-block-code"><code>{
  "query": {
    "module": { "api_name": "Deals" },
    "fields": ["id", "Deal_Name", "Amount", "Stage", "Modified_Time"],
    "criteria": {
      "group_operator": "and",
      "group": [
        {
          "field": { "api_name": "Modified_Time" },
          "comparator": "greater_than",
          "value": "{T1}"
        },
        {
          "field": { "api_name": "Modified_Time" },
          "comparator": "less_than",
          "value": "{T2}"
        }
      ]
    },
    "page": 1
  }
}</code></pre>



<p class="wp-block-paragraph">Two limits to design around. The download URL is only valid for about a day, so fetch and land the file promptly rather than queueing it for a later step. And downloads are rate limited per minute, so if you fan out across many modules at once you will start collecting 429s.</p>



<h3 class="wp-block-heading">COQL, for moderate volumes and real filtering</h3>



<p class="wp-block-paragraph">Zoho&#8217;s SQL-like query API. Up to 2,000 records per call and up to 100,000 records total per unique criteria through pagination. Past that, Zoho&#8217;s own documentation tells you to use Bulk Read, which is good advice to take rather than work around.</p>



<p class="wp-block-paragraph">If you do paginate COQL, paginate by key rather than by offset. Sort by <code>id</code> and carry the last ID you saw into the next call, so shifting records cannot cause a skip:</p>



<pre class="wp-block-code"><code>{
  "select_query": "select id, Deal_Name, Amount, Stage, Modified_Time from Deals where (Modified_Time &gt; '{T1}' and Modified_Time &lt; '{T2}') and id &gt; {last_id} order by id asc limit 2000"
}</code></pre>



<p class="wp-block-paragraph">Note the shape: a closed time window that freezes the set, plus a keyset cursor that walks it deterministically. Offset pagination gives you neither. Zoho&#8217;s own documentation uses this pattern in its examples, which is a fair hint about what they expect.</p>



<h3 class="wp-block-heading">Get Records, for small modules and metadata</h3>



<p class="wp-block-paragraph">The plain module endpoint returns up to 200 records a page and supports an <code>If-Modified-Since</code> header for incremental reads. Fine for lookup tables and small custom modules. Not what you want pointed at a Deals module with six figures of rows.</p>



<p class="wp-block-paragraph">Worth saying plainly: if all of this sounds like a fortnight of work you would rather not own, a managed connector from Fivetran or Airbyte solves the extraction half and you spend your time on the modelling instead. Price it against your own hours honestly, because the build-it-yourself option is habitually costed at zero.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Deletions, which Zoho actually handles well</h2>



<p class="wp-block-paragraph">An incremental extract can never see a deletion, because a record that no longer exists cannot appear in a query for modified records. Most CRMs make you infer this. Zoho gives you a direct endpoint, and it is one of the nicer parts of the API.</p>



<pre class="wp-block-code"><code># type: all | recycle | permanent
curl "https://www.zohoapis.com/crm/v8/Deals/deleted?type=all&amp;per_page=200&amp;page=1" 
  -X GET 
  -H "Authorization: Zoho-oauthtoken $ACCESS_TOKEN" 
  -H "If-Modified-Since: {T1}"</code></pre>



<p class="wp-block-paragraph">The retention windows are generous: records in the recycle bin are retrievable for up to 60 days from deletion, and permanently deleted records for up to 120 days. Compared with warehouses where the detection window is a couple of weeks and not guaranteed, that is a lot of slack. It also means there is no excuse for missing a deletion, since any sane sync interval sits comfortably inside it.</p>



<p class="wp-block-paragraph">Run the deleted-records call as part of every extract cycle, using the same closed window, and mark the matching rows as deleted in your curated layer rather than removing them from raw. Soft-delete preserves the audit trail and lets you answer &#8220;when did this disappear&#8221; later.</p>



<p class="wp-block-paragraph">Even with a good endpoint, run a periodic full ID reconciliation as a backstop, monthly is plenty. Pull just the ID column for the whole module, diff it against the lake, and flag anything you still believe in that Zoho does not. It catches the cases the deletion endpoint does not describe cleanly, such as records merged or moved between modules.</p>



<h2 class="wp-block-heading">Shaping CRM data for S3</h2>



<p class="wp-block-paragraph">Land raw first, exactly as returned, in its own prefix. Then convert to Parquet in a second step. Skipping the raw layer feels efficient right up to the first time you need to reprocess a month with corrected logic.</p>



<pre class="wp-block-code"><code># Partition by the window's upper bound, not by a business date.
# One bad run is then one partition to replace.
s3://acme-lake/raw/zoho_crm/deals/window_end=YYYY-MM-DDTHH/
s3://acme-lake/curated/zoho_crm/deals/window_end=YYYY-MM-DDTHH/</code></pre>



<p class="wp-block-paragraph">Zoho returns nested structures that do not map onto a flat table. Lookup fields come back as objects with an ID and a display name. Multi-select fields and tags come back as arrays. Subforms come back as arrays of objects. You have three choices per field and should make each one deliberately:</p>



<ul class="wp-block-list">
<li><strong>Flatten into columns.</strong> A lookup becomes <code>account_id</code> and <code>account_name</code>. Correct for anything you filter or join on.</li>
<li><strong>Split into a child table.</strong> Subforms and line items become their own Parquet dataset keyed by parent ID. Correct when the nested rows are things people count.</li>
<li><strong>Keep as a JSON string column.</strong> Fine for rarely-queried arrays, and Athena can parse it on demand. Not fine for anything in a regular report.</li>
</ul>



<p class="wp-block-paragraph">Pin your types explicitly during conversion rather than letting the writer infer them from a batch. Inference is the reason a column is a string in January&#8217;s files and a double in February&#8217;s, and Athena will happily read both and quietly fail to reconcile them.</p>



<p class="wp-block-paragraph">Register the result in the Glue Data Catalog and query with Athena. If you expect to apply updates and deletes in place rather than rebuilding partitions, Apache Iceberg earns its extra setup: row-level operations and schema evolution are exactly what a CRM feed generates.</p>



<p class="wp-block-paragraph">On the AWS side, the orchestration is unremarkable and should stay that way: EventBridge on a schedule, Lambda or a small container for the extract, Step Functions if you need to poll a Bulk Read job and branch on the result. Resist Glue jobs for the extraction itself unless you are already deep in Glue; a Python container you can run locally is easier to debug at seven in the morning.</p>



<h2 class="wp-block-heading">API credits</h2>



<p class="wp-block-paragraph">Zoho meters API usage as credits, allocated by edition and user count, over a rolling 24-hour window, and shared with every other integration on the org. There are also concurrency limits, so hammering the API in parallel fails differently from exceeding your daily allowance.</p>



<ul class="wp-block-list">
<li><strong>Bulk Read for volume.</strong> It sidesteps the standard limits and is the whole reason the API exists.</li>
<li><strong>Ask for fewer fields.</strong> Selecting only the columns you actually land reduces payload and processing on both sides.</li>
<li><strong>Mind the COQL limit value.</strong> Credits scale with the page size you request, so grabbing 2,000 rows costs more than grabbing 200. Larger pages are still usually the better trade, but know you are making it.</li>
<li><strong>Give the pipeline its own connected app and user.</strong> Then consumption is attributable when someone asks who exhausted the credits.</li>
<li><strong>Back off properly on 429.</strong> Exponential backoff with jitter, not a fixed sleep, and treat rate limiting as an expected condition rather than an error.</li>
</ul>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Two runs of the same window return different rows</h3>



<p class="wp-block-paragraph">Open-ended window, offset pagination, or both. Add the upper bound, switch to keyset pagination or Bulk Read, and the symptom disappears. This is the one to check before anything else, because it makes every other investigation unreliable.</p>



<h3 class="wp-block-heading">Invalid token, but the credentials are definitely right</h3>



<p class="wp-block-paragraph">Data centre mismatch. Confirm which region the org lives in and that your accounts domain and API domain both match it. This is far more common than an actually bad token.</p>



<h3 class="wp-block-heading">Row counts drift upward over time</h3>



<p class="wp-block-paragraph">Deletions are not being applied. Run the deleted-records endpoint for the last 60 days and see how much comes back, then run a full ID reconciliation to catch the rest.</p>



<h3 class="wp-block-heading">Bulk Read job finished but the file is gone</h3>



<p class="wp-block-paragraph">The download URL expires after roughly a day. Land the file as soon as the job reports complete, rather than deferring it to a downstream step that might not run until tomorrow.</p>



<h3 class="wp-block-heading">Athena fails with a schema mismatch across partitions</h3>



<p class="wp-block-paragraph">Type inference changed between runs, usually because a nullable field was all-null in one batch and populated in the next. Define the schema explicitly at write time and reprocess the affected partitions from raw.</p>



<h3 class="wp-block-heading">A new custom field never appears</h3>



<p class="wp-block-paragraph">You are selecting fields explicitly, which is correct, and nobody told you a field was added. Poll the module&#8217;s field metadata on a schedule and alert on changes, so schema drift is a notification rather than a discovery six weeks later.</p>



<h3 class="wp-block-heading">Frequent 429s</h3>



<p class="wp-block-paragraph">Either concurrency or the download rate limit, depending on which call is failing. Serialise the module extracts rather than fanning them all out at once, and add jittered backoff.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>An open-ended <code>Modified_Time</code> filter with no upper bound.</li>
<li>Offset pagination over a dataset that is being modified while you read it.</li>
<li>Advancing the watermark when the API call succeeds rather than when the data lands.</li>
<li>Keeping watermark state on an ephemeral runner.</li>
<li>Hardcoding the Zoho API domain and discovering data centres the hard way.</li>
<li>Never calling the deleted-records endpoint, so counts only ever grow.</li>
<li>Deferring the Bulk Read download until after the URL has expired.</li>
<li>Landing only Parquet with no raw layer, so reprocessing means re-extracting.</li>
<li>Letting the Parquet writer infer types per batch.</li>
<li>Flattening subforms into a wide table and double-counting parent rows.</li>
<li>Fanning out every module in parallel and collecting rate limits.</li>
<li>Sharing the pipeline&#8217;s connected app with other integrations, so nobody can attribute credit usage.</li>
<li>Storing the refresh token in an environment variable and forgetting it never expires.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Closed read windows with a deliberate lag, always.</li>
<li>Keyset pagination when you paginate at all, Bulk Read when you can avoid it.</li>
<li>Watermarks in durable storage, advanced only on confirmed landing.</li>
<li>Idempotent runs, so re-running a window is always safe.</li>
<li>Deleted-records endpoint every cycle, full ID reconciliation monthly.</li>
<li>Raw layer untouched, curated layer derived, partitions keyed by window end.</li>
<li>Explicit schemas on write, explicit field lists on read.</li>
<li>Its own connected app, narrow scopes, refresh token in Secrets Manager.</li>
<li>Jittered exponential backoff and serialised module extracts.</li>
<li>Alerting on schema drift and on row-count delta against Zoho.</li>
<li>Encryption and a retention policy on CRM data in S3 from day one, since it is personal data.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">What is the best way to export Zoho CRM data to S3?</h3>



<p class="wp-block-paragraph">Bulk Read for anything of size, on a schedule, with closed time windows. COQL for moderate volumes where you want real filtering. The plain records endpoint only for small modules. Land raw output first, convert to Parquet second.</p>



<h3 class="wp-block-heading">Why does my extract return different results each run?</h3>



<p class="wp-block-paragraph">Your query has no upper time bound, so the result set changes while you paginate through it. Bound both ends of the window and put the upper bound a few minutes in the past. That single change makes runs repeatable.</p>



<h3 class="wp-block-heading">How do I capture deleted records?</h3>



<p class="wp-block-paragraph">Call the module&#8217;s deleted-records endpoint each cycle. Recycle-bin deletions stay retrievable for 60 days and permanent deletions for 120, so a daily or hourly sync has plenty of margin. Add a monthly full ID reconciliation as a backstop.</p>



<h3 class="wp-block-heading">Will this exhaust my Zoho API credits?</h3>



<p class="wp-block-paragraph">Not if you use Bulk Read for the heavy lifting, since it does not draw on standard API limits the way per-record calls do. The credit pool is shared across the org, so give the pipeline its own connected app and monitor consumption rather than finding out when someone else&#8217;s integration breaks.</p>



<h3 class="wp-block-heading">How do I handle subforms and multi-select fields?</h3>



<p class="wp-block-paragraph">Decide per field. Flatten lookups into ID and name columns, split subforms and line items into their own child datasets keyed by parent ID, and keep genuinely peripheral arrays as JSON strings. Flattening a subform into the parent row is how you end up double-counting deals.</p>



<h3 class="wp-block-heading">Build it or buy a connector?</h3>



<p class="wp-block-paragraph">Buy it if Zoho CRM is one source among several and you have no unusual requirements; managed connectors handle pagination, deletes and schema drift, which is most of the work described here. Build it when you need control over the shape of the output, want to avoid per-row pricing, or already run the surrounding infrastructure.</p>



<h3 class="wp-block-heading">How fresh can the data realistically be?</h3>



<p class="wp-block-paragraph">Hourly is comfortable and suits nearly all reporting. Every fifteen minutes is achievable with COQL on modest modules. Anything closer to real time means webhooks or notification subscriptions rather than polling, which is a considerably larger commitment for a benefit most dashboards do not use.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">An extract that returns a different answer each time it runs is not a data quality problem, it is a read consistency problem, and no amount of downstream validation will fix it. Bound both ends of the window, put the upper bound in the past, paginate by key rather than offset, and advance the watermark only when the data is safely in S3.</p>



<p class="wp-block-paragraph">Get that right and the rest of the pipeline becomes ordinary engineering: land it, convert it, catalogue it, query it. Get it wrong and you will spend months chasing numbers that move every time you look at them.</p>



<h2 class="wp-block-heading">Want this built or reviewed?</h2>



<p class="wp-block-paragraph">Most CRM pipelines I get handed work fine on a quiet module and lose rows on the busy one, which is the hardest version to notice. Work I take on:</p>



<ul class="wp-block-list">
<li>Building a Zoho CRM to S3 pipeline end to end: OAuth, Bulk Read extraction, Parquet conversion, Glue catalog, Athena query layer.</li>
<li>Auditing an existing pipeline for read consistency and telling you whether it is silently dropping records.</li>
<li>Reworking incremental logic into closed windows, keyset pagination and durable watermarks.</li>
<li>Deletion handling and reconciliation jobs so row counts stay equal to the CRM.</li>
<li>Modelling nested CRM data into a schema that does not double-count.</li>
<li>Orchestration and alerting on AWS: EventBridge, Lambda or containers, Step Functions, and monitoring that catches drift early.</li>
</ul>



<p class="wp-block-paragraph">Send me your extract query and how you paginate, and I will tell you whether it can lose rows.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/zoho-crm-to-amazon-s3/">Run It Twice, Get Two Answers: Building an ETL Pipeline From Zoho CRM to Amazon S3</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
