<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Amazon Redshift | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/amazon-redshift/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/amazon-redshift/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sun, 27 Sep 2026 06:01:58 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Amazon Redshift | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/amazon-redshift/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>SageMaker Lakehouse vs Glue + Athena: Is Migration Worth It?</title>
		<link>https://john-nessime.com/blog/cloud-computing/sagemaker-lakehouse-migration/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 27 Sep 2026 06:01:58 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Cost Allocation Tags]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Cross-Account Access]]></category>
		<category><![CDATA[Data Catalog]]></category>
		<category><![CDATA[Data Governance]]></category>
		<category><![CDATA[Federated Catalogs]]></category>
		<category><![CDATA[Hybrid Access Mode]]></category>
		<category><![CDATA[Iceberg REST Catalog]]></category>
		<category><![CDATA[Lake Formation]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[Migration Assessment]]></category>
		<category><![CDATA[Operational Complexity]]></category>
		<category><![CDATA[S3 Tables]]></category>
		<category><![CDATA[SageMaker Unified Studio]]></category>
		<category><![CDATA[Total Cost of Ownership]]></category>
		<category><![CDATA[Zero-ETL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=595</guid>

					<description><![CDATA[<p>SageMaker Lakehouse is built on the Glue Data Catalog and Lake Formation, so a migration barely moves data. It moves enforcement. Here is the failure mode that bites, the cost levers that actually change, and a decision procedure for working out whether the move is worth it at all.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/sagemaker-lakehouse-migration/">SageMaker Lakehouse vs Glue + Athena: Is Migration Worth It?</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A Glue job that had run every morning for two years failed on a Monday. Access denied on <code>GetTable</code>, thrown by Lake Formation. Nothing about the job had changed. Nothing about the IAM role had changed. What had changed was that someone spent Friday afternoon &#8220;moving us onto the lakehouse&#8221; and, as part of that, revoked a permission from a virtual group most people have never heard of.</p>



<p class="wp-block-paragraph">That is the shape of the risk. When people ask whether a SageMaker Lakehouse migration is worth it, they picture data moving. Almost none of it does. What moves is the permission model, and that is the part nobody tests.</p>



<p class="wp-block-paragraph">This post covers what the migration actually changes underneath, the failure mode that bites hardest, where the lakehouse earns its place, where staying on Glue plus Athena is the better call, how the cost levers shift, and a decision procedure you can run in an afternoon.</p>



<h2 class="wp-block-heading">First, understand what you are actually migrating</h2>



<p class="wp-block-paragraph">The framing &#8220;SageMaker Lakehouse versus Glue and Athena&#8221; is misleading, and it drives bad decisions.</p>



<p class="wp-block-paragraph">SageMaker Lakehouse is built on the AWS Glue Data Catalog and AWS Lake Formation. It is not a replacement metastore. It is a reorganisation of the catalog you already have, plus connectors, plus an Apache Iceberg REST interface, plus a console to look at all of it through.</p>



<p class="wp-block-paragraph">The unit it introduces is the <em>catalog</em>: a logical container of databases, tables and views, nestable to mirror the shape of the source system. A <strong>managed catalog</strong> holds data you own in S3 or Redshift Managed Storage. A <strong>federated catalog</strong> points at something external such as Redshift, DynamoDB, PostgreSQL, Oracle, BigQuery or Snowflake, and lets you query it in place.</p>



<p class="wp-block-paragraph">Your existing databases and tables do not disappear. They sit under the account catalog and keep working. AWS states in its own FAQ that you do not have to migrate, and that Athena, EMR, Glue and MWAA continue to work as they do today. That is an unusual thing for a vendor to publish about a flagship product, and it is the most useful sentence in the documentation set.</p>



<p class="wp-block-paragraph">So the question is not &#8220;should we move our data&#8221;. It is &#8220;should we move our governance&#8221;.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The failure mode that actually bites</h2>



<p class="wp-block-paragraph">Lake Formation grants <code>Super</code> permission to a virtual group called <code>IAMAllowedPrincipals</code> on every existing Data Catalog database and table by default. That group effectively means &#8220;anyone whose IAM policy already lets them in&#8221;. It exists purely for backward compatibility. While it holds <code>Super</code> on a resource, access is decided by IAM and S3 policies exactly as it was before Lake Formation existed. Your Lake Formation grants are recorded, but they are not what gets enforced.</p>



<p class="wp-block-paragraph">So the first time someone writes a careful column-level policy and tests it, the policy appears not to work. The analyst can still see the column. The natural next move is to revoke <code>IAMAllowedPrincipals</code> so the grants take effect.</p>



<p class="wp-block-paragraph">That revoke is the moment everything relying on IAM-only access stops working. Glue jobs. Notebooks. Scheduled dbt runs. The QuickSight dataset nobody has looked at since the person who built it left. None of them had a Lake Formation grant, because none of them ever needed one.</p>



<p class="wp-block-paragraph">It is invisible in advance because nothing warns you. Lake Formation cannot know which principals are silently depending on the compatibility group. There is no dry run.</p>



<h3 class="wp-block-heading">Hybrid access mode is the way through</h3>



<p class="wp-block-paragraph">Hybrid access mode exists exactly for this. It supports two permission pathways to the same catalog objects at once. Principals you explicitly opt in are governed by Lake Formation grants. Everyone else keeps reaching the data through their IAM and S3 policies, untouched.</p>



<p class="wp-block-paragraph">The detail worth internalising: when an S3 location is registered in hybrid access mode, Lake Formation enforces only <code>CREATE_TABLE</code>, <code>CREATE_PARTITION</code> and <code>UPDATE_TABLE</code> by default. The write path gets governed first. The read path stays on IAM until you opt a principal in. That asymmetry is what lets you migrate one team at a time instead of one account at a time.</p>



<p class="wp-block-paragraph">The safe sequence: register the location in hybrid mode, opt in one low-stakes principal, verify its queries respect the fine-grained grants, verify a non-opted-in principal still works, repeat. Only after every consumer is opted in and verified do you consider revoking the compatibility group. Plenty of teams never need that last step.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Where the lakehouse genuinely wins</h2>



<ul class="wp-block-list">
<li><strong>One permission surface across engines.</strong> If governance today is &#8220;Lake Formation for the lake, Redshift GRANTs for the warehouse, IAM for DynamoDB&#8221;, you maintain three access models that disagree with each other. Catalogs let you express column and row-level policy once and have Athena, Redshift, EMR and Spark honour it.</li>



<li><strong>Tag-based access control across federated sources.</strong> LF-TBAC extends to federated catalogs covering S3 Tables, Redshift and external sources. If you hand-maintain named-resource grants across dozens of databases, tag expressions are a real operational win.</li>



<li><strong>Querying across boundaries without copying.</strong> A federated catalog lets one statement touch a Redshift table and an S3 Iceberg table, with no ETL job whose only purpose is moving rows so they can sit next to each other. Every pipeline you delete is a pipeline that cannot break at 3am.</li>



<li><strong>The Iceberg REST interface.</strong> Exposing the catalog through Iceberg REST APIs lets engines outside AWS read your tables over a standard protocol. If you run Trino, Spark or DuckDB elsewhere, this is what makes the catalog genuinely portable rather than nominally open.</li>



<li><strong>Cross-account sharing without replication.</strong> For data mesh setups, sharing a catalog instead of shipping a copy per consumer removes a whole category of staleness bug. Cross-account grants on federated catalog objects require the cross-account data sharing version set to 4 or higher.</li>
</ul>



<p class="wp-block-paragraph">Notice what these share. Every one is a governance or interoperability problem. None is a query performance problem, and none is a cost problem.</p>



<h2 class="wp-block-heading">Where Glue plus Athena is still the right answer</h2>



<ul class="wp-block-list">
<li><strong>Single account, S3 only, one team.</strong> Catalogs solve a problem you do not have. You would be adding a governance layer to govern one group.</li>



<li><strong>Your access control is coarse and that is correct.</strong> Not every dataset needs column masking. If bucket-level IAM boundaries match your actual risk model, Lake Formation enforcement adds a failure mode without adding safety.</li>



<li><strong>You have heavy IAM-only automation.</strong> Every Glue job, Lambda, Airflow DAG and third-party tool touching the catalog is a principal you must eventually opt in and re-verify. That inventory is the real cost of migration, and it is always larger than the first estimate.</li>



<li><strong>Your bottleneck is query cost, not governance.</strong> Partitioning, file sizes, compaction and columnar layout drive Athena spend. A lakehouse migration touches none of them. Fix the small-files problem first; it pays back faster and it is reversible.</li>



<li><strong>You depend on a specific engine behaviour today.</strong> Fine-grained access control support varies by engine and table format. Confirm your exact combination is supported rather than assuming parity across Athena, Redshift, EMR and Glue Spark.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The cost levers that move, and the ones that don&#8217;t</h2>



<p class="wp-block-paragraph">Pricing pages change, so here are the billing mechanisms rather than numbers.</p>



<p class="wp-block-paragraph"><strong>Unchanged:</strong> metadata storage and API requests follow existing Glue Data Catalog pricing, including free tier allowances. Reorganising into catalogs does not put you on a new metadata meter. Athena still bills on data scanned. S3 storage and request charges are what they were.</p>



<p class="wp-block-paragraph"><strong>New line items:</strong></p>



<ul class="wp-block-list">
<li><strong>Lambda invocations behind federated catalogs.</strong> When a query hits a federated table, Lake Formation vends credentials that invoke a Lambda function defined in the Glue connection to retrieve metadata from the source. That is compute you did not previously pay for, and it is latency on the metadata path. Chatty BI tools that re-resolve schemas on every refresh are the ones to watch.</li>



<li><strong>Service-managed Redshift Serverless compute.</strong> Queries against a federated catalog pointing at an existing Redshift warehouse execute through a service-managed Redshift Serverless workgroup, which stages results before returning them. That compute is billed, and it does not appear where you expect it.</li>



<li><strong>Redshift Managed Storage.</strong> Managed catalogs backed by RMS rather than S3 put you on Redshift storage and Redshift Serverless compute economics, which behave nothing like S3 plus Athena.</li>
</ul>



<p class="wp-block-paragraph">The service-managed workgroup is the one that catches people, because the spend lands under Redshift Serverless with no obvious link to the analyst who ran the query. AWS publishes a generated cost allocation tag for exactly this, which you must activate in the billing console before it produces anything:</p>



<pre class="wp-block-code"><code># Cost allocation tag on lakehouse-managed Redshift Serverless compute.
# Activate under Billing -&gt; Cost allocation tags before it appears in reports.

Key:   aws:redshift-serverless:LakehouseManagedWorkgroup
Value: "True"</code></pre>



<p class="wp-block-paragraph">Activate it on day one, not after the first surprising invoice. Tags only apply to usage recorded after activation, so a late start leaves a blind period you cannot reconstruct. If you already run a cost visibility layer such as Vantage, CloudZero or Datadog&#8217;s cloud cost views, build a view isolating this tag before you create your first federated catalog. If you do not, the AWS Cost and Usage Report grouped by that tag key is enough to start.</p>



<h2 class="wp-block-heading">Is a SageMaker Lakehouse migration worth it? A decision procedure</h2>



<p class="wp-block-paragraph">Run these in order. Stop at the first one that gives a clear answer.</p>



<ol class="wp-block-list">
<li><strong>Count your non-S3 governed sources.</strong> Redshift clusters, DynamoDB tables, operational databases and external warehouses that analysts need and that each carry their own access model today. Zero or one means the federated catalog story does not apply to you yet. Three or more and the unified permission surface starts paying for itself.</li>



<li><strong>Inventory every principal touching the Data Catalog.</strong> Pull CloudTrail for Glue and Lake Formation API calls across a full billing cycle, long enough to catch monthly jobs. Every distinct role in that list is work. This number, not your data volume, is the migration estimate.</li>



<li><strong>Ask whether column or row-level control has actually been requested.</strong> If the answer is a compliance requirement with a name attached, proceed. If it is &#8220;it would be nice&#8221;, you are buying a permission model to solve a hypothetical.</li>



<li><strong>Check engine coverage against your real workloads.</strong> Confirm fine-grained access control support for each production engine in current documentation. One unsupported critical path makes hybrid mode permanent rather than transitional, which is a legitimate outcome but should be a decision, not a discovery.</li>



<li><strong>Price the federated path against a copy.</strong> For your highest-frequency cross-source query, estimate metadata Lambda invocations plus any service-managed compute, then compare against a nightly job that lands a copy in S3. Federation wins on freshness and on one less pipeline. It does not automatically win on cost.</li>



<li><strong>Decide the destination, not just the direction.</strong> &#8220;Hybrid access mode indefinitely, Lake Formation enforcement on the two regulated datasets&#8221; is a good end state. Write it down first, because migrations without a finish line become permanent half-migrations.</li>
</ol>



<h3 class="wp-block-heading">Three things to run before you commit</h3>



<p class="wp-block-paragraph">Enumerate what Athena can see today. This is your before picture, and it exposes a tooling gap worth knowing up front:</p>



<pre class="wp-block-code"><code>aws athena list-data-catalogs

# Returns catalogs registered directly with Athena, for example:
# {
#   "DataCatalogsSummary": [
#     { "CatalogName": "AwsDataCatalog", "Type": "GLUE" },
#     { "CatalogName": "cw_logs_catalog", "Type": "LAMBDA" }
#   ]
# }</code></pre>



<p class="wp-block-paragraph">S3 Tables catalogs use a <code>s3tablescatalog/bucket-name</code> naming convention and are not returned by this call, even though you can query them by naming the catalog explicitly. The console shows them through mechanisms the public API does not expose. Any inventory or drift-detection tooling built on <code>list-data-catalogs</code> will quietly under-report, so enumerate S3 Tables catalogs separately.</p>



<p class="wp-block-paragraph">Next, confirm the role you intend to use as data lake administrator can create catalogs at all. If it is not already a data lake admin, it needs an explicit grant:</p>



<pre class="wp-block-code"><code>aws lakeformation grant-permissions 
  --cli-input-json 
  '{
     "Principal": {
       "DataLakePrincipalIdentifier": "arn:aws:iam::123456789012:role/Admin"
     },
     "Resource": { "Catalog": {} },
     "Permissions": [ "CREATE_CATALOG", "DESCRIBE" ]
   }'</code></pre>



<p class="wp-block-paragraph">This avoids the common false start where catalog creation works in the console for an admin user and fails for the automation role you actually meant to use.</p>



<p class="wp-block-paragraph">Finally, if you read lakehouse tables from a Glue Spark job, fine-grained access control is not on by default. The job parameters that enable it:</p>



<pre class="wp-block-code"><code>--datalake-formats                          iceberg
--enable-lakeformation-fine-grained-access  true</code></pre>



<p class="wp-block-paragraph">Without the second one, the job reads through its IAM role and your column-level grants are not applied. That is a governance gap that looks exactly like a working pipeline, which is the worst kind.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<ul class="wp-block-list">
<li><strong>&#8220;It&#8217;s the same Glue Data Catalog, so migrating is low risk.&#8221;</strong> True about the metadata, false about enforcement. The catalog is the same object. The thing deciding whether a query returns rows is not.</li>



<li><strong>&#8220;We&#8217;ll migrate to avoid falling behind.&#8221;</strong> AWS states plainly that you can keep using Athena, EMR, Glue and MWAA as you do today. Nothing here is a deprecation. Migrating on an invented schedule is how you end up owning a permission model nobody needed.</li>



<li><strong>&#8220;Federated catalogs mean we can retire our ETL.&#8221;</strong> Some of it, sometimes. Federation removes copies that existed only for co-location. It does not remove transformation, deduplication, late-arriving data handling or historical snapshots. Those jobs stay.</li>



<li><strong>&#8220;Iceberg REST support means no lock-in.&#8221;</strong> The table format is portable and that is real. The Lake Formation grants, LF-Tags, hybrid mode opt-ins and catalog hierarchy are not, and they become the hardest part of your platform to reproduce elsewhere. Portability moved down a layer; it did not appear.</li>



<li><strong>&#8220;We can test this in a sandbox.&#8221;</strong> You can test the mechanics. You cannot test the thing that actually breaks, which is the set of production principals depending on IAM-only access. That inventory does not exist in a sandbox.</li>
</ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<p class="wp-block-paragraph">S3-only, one account, small team: I would not migrate. The same effort spent on partitioning and compaction returns more, and it keeps returning.</p>



<p class="wp-block-paragraph">Multiple governed sources, several consuming teams, a named compliance requirement for fine-grained control: migrate, and treat hybrid access mode as the destination rather than a waypoint. Enforce Lake Formation on the datasets that need it, leave the rest on IAM. The pressure to reach a &#8220;clean&#8221; fully-enforced state is aesthetic, not operational.</p>



<p class="wp-block-paragraph">Either way, start with a federated catalog over one source that is currently painful to reach. Prove the query path and the cost, and let the rest follow evidence rather than a plan drawn before anyone had numbers.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does SageMaker Lakehouse replace AWS Glue and Athena?</h3>



<p class="wp-block-paragraph">No. It is built on the Glue Data Catalog and Lake Formation, and Athena remains one of the engines you query it with. AWS documentation states you can continue using Athena, EMR, Glue and MWAA without migrating. It is a governance and connectivity layer, not a replacement stack.</p>



<h3 class="wp-block-heading">Will a SageMaker Lakehouse migration break my existing Athena queries?</h3>



<p class="wp-block-paragraph">Reorganising into catalogs does not break queries on its own. What breaks them is revoking <code>Super</code> from <code>IAMAllowedPrincipals</code> before every consuming principal has a Lake Formation grant. Use hybrid access mode, opt principals in one at a time, and existing queries keep working throughout.</p>



<h3 class="wp-block-heading">Do I have to move data into Redshift Managed Storage?</h3>



<p class="wp-block-paragraph">No. Managed catalogs can be backed by S3, including S3 Tables, and existing S3 data lakes integrate directly. RMS is a storage choice with different cost and performance behaviour, not a requirement of the architecture.</p>



<h3 class="wp-block-heading">How much extra does it cost?</h3>



<p class="wp-block-paragraph">The catalog layer itself is not a new meter: metadata storage and API requests follow existing Glue Data Catalog pricing including free tier allowances. New costs come from what you build on it, mainly Lambda invocations for federated metadata retrieval, service-managed Redshift Serverless compute for federated Redshift queries, and RMS storage if you choose it.</p>



<h3 class="wp-block-heading">Is SageMaker Unified Studio required?</h3>



<p class="wp-block-paragraph">No. Unified Studio is the integrated console experience. The catalogs are addressable from Athena, Redshift, EMR, Glue and Iceberg-compatible engines directly. Adopting the catalog structure without adopting the studio is a valid and common choice.</p>



<h3 class="wp-block-heading">What is the smallest useful first step?</h3>



<p class="wp-block-paragraph">Create one federated catalog over a single non-S3 source that analysts currently reach through an awkward workaround. Grant read access to one team. Measure query latency and the resulting Lambda and compute charges. That gives you real numbers on your own workload, and it is trivially reversible.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">A SageMaker Lakehouse migration is not a data migration. The Glue Data Catalog underneath is the same catalog, the tables are the same tables, Athena is still Athena. What changes is which system decides whether a query returns rows.</p>



<p class="wp-block-paragraph">That makes the call simpler than the marketing implies. Several governed sources, multiple consuming teams and a real fine-grained control requirement, and the unified permission surface is worth the work. Without them, you are taking on a permission model and a Lambda-backed metadata path in exchange for a nicer console.</p>



<p class="wp-block-paragraph">Whichever way you go: register in hybrid access mode, opt principals in one at a time, and do not revoke <code>IAMAllowedPrincipals</code> until you can name every principal that will notice.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help deciding, or unpicking a migration already underway?</h2>



<p class="wp-block-paragraph">An outside read on the actual inventory usually saves more time than any amount of architecture discussion. Things I help with:</p>



<ul class="wp-block-list">
<li>Building the principal inventory from CloudTrail so you know exactly who depends on IAM-only Data Catalog access before anything is revoked</li>



<li>Designing a hybrid access mode rollout that opts teams in incrementally, with a verification step and a rollback position at every stage</li>



<li>Modelling federated catalog costs against the copy-the-data alternative for your query patterns, including the service-managed compute nobody budgets for</li>



<li>Diagnosing Lake Formation access denied errors across Athena, Glue Spark and Redshift, including the fine-grained access flags that silently bypass your grants</li>



<li>Cost allocation tagging so lakehouse spend is attributable per catalog and per team from the first query rather than retrofitted later</li>



<li>Honest second opinions on whether a migration is worth doing at all, including the answer that it isn&#8217;t</li>
</ul>



<p class="wp-block-paragraph">Send me a failing query, a Lake Formation grant listing or a CloudTrail export and I&#8217;ll tell you what I&#8217;d do with it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>

<p>The post <a href="https://john-nessime.com/blog/cloud-computing/sagemaker-lakehouse-migration/">SageMaker Lakehouse vs Glue + Athena: Is Migration Worth It?</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Salesforce AWS Integration Architecture: The Four Paths and What Breaks Silently</title>
		<link>https://john-nessime.com/blog/devops/salesforce-aws-integration-architecture/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 19 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AppFlow]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Change Data Capture]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Event Relay]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[Named Credentials]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Platform Events]]></category>
		<category><![CDATA[PrivateLink]]></category>
		<category><![CDATA[Pub/Sub API]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[Serverless]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=209</guid>

					<description><![CDATA[<p>Most Salesforce AWS integrations don't fail loudly, they drift. This guide covers the four integration paths between Salesforce and AWS, when to use each, the silent failure mode hiding in every one of them, and the reconciliation job you have to build yourself.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-aws-integration-architecture/">Salesforce AWS Integration Architecture: The Four Paths and What Breaks Silently</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone in the ops channel asks why the opportunity count on the QuickSight dashboard doesn&#8217;t match the count in the Salesforce report. It&#8217;s off by a few hundred. Nobody can answer that, and worse, nobody can say when the two numbers stopped agreeing. Every pipeline is green. The scheduled flow reports successful runs. The EventBridge rule shows invocations. Everything is healthy except the data.</p>



<p class="wp-block-paragraph">That&#8217;s the shape of most Salesforce AWS integration problems. They&#8217;re rarely outages. They&#8217;re divergence, accumulating quietly while every status page stays green, and you find out about it when someone in finance runs a number past a customer.</p>



<p class="wp-block-paragraph">This guide covers the four integration paths between Salesforce and AWS, which one to reach for and when, the specific failure each one hides, and the reconciliation layer you&#8217;ll have to build yourself because neither vendor ships it.</p>



<h2 class="wp-block-heading">Why a Salesforce AWS integration fails silently</h2>



<p class="wp-block-paragraph">Start here, because it changes how you design everything downstream. There are three mechanisms that lose data without raising an error anywhere you&#8217;re looking.</p>



<p class="wp-block-paragraph"><strong>Events sent to an unassociated partner event bus are dropped on the floor.</strong> When you create an Event Relay, Salesforce creates a partner event source in your AWS account. Until you associate that source with an event bus, EventBridge discards anything the partner publishes to it. AWS is explicit that those events are not persisted at rest. There&#8217;s no queue holding them for you, no backlog to drain once you finish the setup. If the relay is running and the bus isn&#8217;t associated, you are streaming into a void.</p>



<p class="wp-block-paragraph"><strong>The Salesforce event bus retains high-volume events for 72 hours.</strong> Platform events and change data capture events sit on the event bus for three days, and each one carries a replay ID so a durable subscriber can resume from where it left off. Past that window, they&#8217;re purged. Salesforce doesn&#8217;t guarantee availability beyond it. So a relay that goes into an error state on a Friday afternoon is a data loss event by Monday morning, and the only symptom during those three days is an absence of messages, which looks identical to a quiet weekend.</p>



<p class="wp-block-paragraph"><strong>Deletes don&#8217;t show up in query-based syncs.</strong> A scheduled pull selects records that exist. Records that stopped existing between runs are simply not in the result set, so your S3 prefix or Redshift table keeps a row that Salesforce no longer has. Nothing errors. The row count on the AWS side just creeps upward relative to the CRM, forever, until someone notices.</p>



<p class="wp-block-paragraph">Every architecture decision below is really a decision about which of these three you&#8217;re accepting and how you&#8217;re going to detect it.</p>



<h2 class="wp-block-heading">A Salesforce AWS integration starts with a direction, not a service</h2>



<p class="wp-block-paragraph">The most common mistake I see is picking a service first, usually whichever one appeared in a conference demo, and then bending the requirement to fit it. Two questions decide the path, and they take about a minute to answer.</p>



<ol class="wp-block-list">
<li><strong>Which way does the data move?</strong> Salesforce to AWS, or AWS to Salesforce. If the answer is &#8220;both&#8221;, you have two integrations, not one, and they will have different failure modes and different owners.</li>

<li><strong>What shape is the workload?</strong> Batch, where you want a consistent set of records on a schedule, or event-driven, where you want a reaction within seconds of a change.</li>
</ol>



<p class="wp-block-paragraph">That gives you four cells, and each cell has a default answer.</p>



<h3 class="wp-block-heading">Salesforce to AWS, batch: Amazon AppFlow</h3>



<p class="wp-block-paragraph">Amazon AppFlow is the managed answer for pulling Salesforce objects into S3, Redshift, or the Glue Data Catalog on a schedule or on demand. It handles the OAuth connection, field mapping, and incremental pulls without you writing or hosting anything. For analytics loads it&#8217;s usually the right default, and it&#8217;s the path of least resistance if the destination is a data lake you&#8217;re already cataloguing with Glue and querying with Athena.</p>



<p class="wp-block-paragraph">The trade-off is that it&#8217;s a polling client, and it spends your Salesforce API allocation to do it. AWS documents this plainly with an example: an hourly flow pulling five pages of data makes 120 API calls a day, and all of them count against the org&#8217;s 24-hour API request limit. Multiply that by every object you&#8217;re syncing and every environment you&#8217;re syncing from, and AppFlow can become the largest single consumer of an allocation that Apex callouts, integration users, and third-party tools are also drawing on.</p>



<p class="wp-block-paragraph">The failure mode that catches people: if a field referenced in the flow&#8217;s mapping is deleted in Salesforce, the flow run fails. Someone cleans up an unused custom field on a Tuesday and your nightly load breaks on Wednesday, in a service that nobody on the Salesforce side knows exists. AppFlow will automatically pick up newly created Salesforce fields when the destination is S3, but the reverse case, a removed field, needs a human to edit the mapping. Put the flow names somewhere the Salesforce admins will see them before they run a field cleanup.</p>



<h3 class="wp-block-heading">Salesforce to AWS, event-driven: Event Relay and Amazon EventBridge</h3>



<p class="wp-block-paragraph">Event Relay subscribes to a channel on the Salesforce event bus and streams platform events and change data capture events into Amazon EventBridge as a partner event source. No middleware, no subscriber you have to keep alive, no polling and therefore no API allocation burn on the read path. From EventBridge you fan out to Lambda, Step Functions, SQS, Firehose, whatever the downstream is.</p>



<p class="wp-block-paragraph">Setup runs across both consoles, and the order matters. On the Salesforce side you create a named credential that points at your AWS account and region. It uses no authentication protocol, because trust is established by the partner event source relationship rather than by a credential in the request. The region goes in uppercase, which is the single most common reason a first attempt doesn&#8217;t work.</p>



<pre class="wp-block-code"><code>POST /services/data/vXX.X/tooling/sobjects/NamedCredential/

{
  "FullName": "AwsRelayCredential",
  "Metadata": {
    "label": "AwsRelayCredential",
    "endpoint": "arn:aws:US-EAST-1:111122223333",
    "principalType": "NamedUser",
    "protocol": "NoAuthentication",
    "generateAuthorizationHeader": true
  }
}</code></pre>



<p class="wp-block-paragraph">The endpoint is not a URL. It&#8217;s an ARN-shaped string carrying the AWS region and the twelve-digit account ID, and that&#8217;s the whole addressing scheme. Get the account wrong and the partner event source appears in a stranger&#8217;s console, not yours.</p>



<p class="wp-block-paragraph">You then create an event channel, add the platform events or change events you want as channel members, and create the relay configuration against that channel. A relay is created in a stopped state deliberately, so you get a chance to verify the AWS account and region before anything flows. That pause is the useful part of the design, not an inconvenience to click through.</p>



<p class="wp-block-paragraph">Now switch to AWS and associate the source. This is the step that, skipped, produces the silent drop described earlier.</p>



<pre class="wp-block-code"><code># Find the partner event source Salesforce created in your account
aws events list-event-sources --region us-east-1

# Create the matching bus. Both values must be the source name, character for character.
aws events create-event-bus 
  --name "aws.partner/salesforce.com/&lt;org-and-channel-suffix&gt;" 
  --event-source-name "aws.partner/salesforce.com/&lt;org-and-channel-suffix&gt;" 
  --region us-east-1

# Confirm the source moved from PENDING to ACTIVE
aws events describe-event-source 
  --name "aws.partner/salesforce.com/&lt;org-and-channel-suffix&gt;" 
  --region us-east-1</code></pre>



<p class="wp-block-paragraph"><code>describe-event-source</code> returning <code>ACTIVE</code> is your proof that a matching bus exists and is enabled. <code>PENDING</code> means either you haven&#8217;t created the bus or it&#8217;s deactivated, and every event published in that state is gone. This is the single check worth wiring into a synthetic monitor, because it&#8217;s cheap to run and it catches the most expensive mistake.</p>



<p class="wp-block-paragraph">Only once the source is active do you set the relay state to run. From the Salesforce side, the feedback object is where relay errors surface rather than in any log you&#8217;d normally tail:</p>



<pre class="wp-block-code"><code>SELECT Id, RemoteResource, Status, ErrorMessage, ErrorTime, ErrorIdentifier
FROM EventRelayFeedback
WHERE EventRelayConfigId = '7k2XXXXXXXXXXXXXXX'</code></pre>



<p class="wp-block-paragraph">If you&#8217;d rather not use Event Relay at all, the Pub/Sub API is the direct route: a gRPC interface to the same event bus, delivering Avro-encoded payloads, which you subscribe to from your own consumer. That&#8217;s more code and a process to keep running, on ECS or a VPS from a provider like InterServer or Hetzner if you&#8217;re keeping it outside AWS, but you get explicit control over replay ID checkpointing. Teams already running Kafka often go this way and land the stream through a managed connector from Confluent instead. Event Relay is the lower-operational-cost option; the Pub/Sub API is the one to reach for when you need the checkpoint under your own control.</p>



<h3 class="wp-block-heading">AWS to Salesforce, event-driven: EventBridge API Destinations</h3>



<p class="wp-block-paragraph">Going the other way, an API destination lets EventBridge call the Salesforce REST API directly, with no Lambda in the middle. You create a connection holding OAuth client credentials from a Salesforce connected app, point the destination at your org&#8217;s My Domain endpoint, and route matched events to it. Publishing back into an inbound platform event is the common pattern, so the write lands on the Salesforce event bus and Apex or Flow picks it up from there.</p>



<p class="wp-block-paragraph">Two constraints shape the design, and both are documented:</p>



<ul class="wp-block-list">
<li><strong>Five second client execution timeout.</strong> If Salesforce takes longer than that to respond, EventBridge times out the request and treats it as a retryable failure. Anything you make synchronous behind that endpoint, a trigger doing a callout, a validation chain, a busy org at peak, will eat into it.</li>

<li><strong>The default retry policy is generous, which is the problem.</strong> For event buses, EventBridge will retry a failed delivery for up to 24 hours and up to 185 times by default. If the endpoint is genuinely broken, that&#8217;s a day of retries against your org, consuming API calls, before the event is discarded with nowhere to go.</li>
</ul>



<p class="wp-block-paragraph">Tighten the policy and always attach a dead letter queue. Failed events land somewhere you can inspect and replay rather than evaporating at the end of the retry window:</p>



<pre class="wp-block-code"><code>{
  "Id": "salesforce-writeback",
  "Arn": "arn:aws:events:us-east-1:111122223333:api-destination/sf-platform-event/&lt;id&gt;",
  "RoleArn": "arn:aws:iam::111122223333:role/EventBridgeApiDestinationRole",
  "RetryPolicy": {
    "MaximumRetryAttempts": 20,
    "MaximumEventAgeInSeconds": 3600
  },
  "DeadLetterConfig": {
    "Arn": "arn:aws:sqs:us-east-1:111122223333:salesforce-writeback-dlq"
  }
}</code></pre>



<p class="wp-block-paragraph">Set the invocation rate limit on the destination to something your org can absorb. It exists specifically so you can align delivery with a third-party API&#8217;s limits, and Salesforce is very much a third-party API with limits.</p>



<p class="wp-block-paragraph">One thing worth knowing before you design around it: API destinations don&#8217;t support mutual TLS. If your security review requires mTLS on outbound calls, you&#8217;re putting a Lambda or a proxy in the path regardless.</p>



<h3 class="wp-block-heading">AWS to Salesforce, batch: Lambda and the Bulk API</h3>



<p class="wp-block-paragraph">For loading volume back into the org, nothing managed does this well and you write it yourself. A Lambda or a container job authenticates with the JWT bearer flow against a connected app, submits a Bulk API job, and polls for results. The REST API is fine for small writes; past a few thousand records the per-call overhead and the API allocation make the Bulk API the only sensible choice.</p>



<p class="wp-block-paragraph">This is the cell where you own everything: retries, idempotency, partial-failure handling, and backoff when the org throttles you. Use external ID fields and upsert semantics rather than insert, so a retried batch converges instead of duplicating. Store the job ID before you submit, not after, so a Lambda timeout mid-flight leaves you something to reconcile against. And keep the Salesforce credential in Secrets Manager with rotation wired up, because a hardcoded connected app secret in an environment variable is the finding that shows up in every audit.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The network layer: public internet, PrivateLink, or Private Connect</h2>



<p class="wp-block-paragraph">All four paths above work over the public internet with TLS, and for most workloads that&#8217;s genuinely fine. The question is whether your compliance posture or your risk appetite says otherwise.</p>



<p class="wp-block-paragraph">Salesforce Private Connect is the managed private path. Salesforce operates a transit VPC in each supported AWS region, and you create a PrivateLink connection between that transit VPC and yours. Traffic between the two never traverses the public internet. The terminology is worth getting right because it trips people up in design reviews:</p>



<ul class="wp-block-list">
<li><strong>Inbound</strong> means traffic flowing <em>into</em> Salesforce. AWS to Salesforce, or on-premises to Salesforce.</li>

<li><strong>Outbound</strong> means traffic flowing <em>out of</em> Salesforce. Salesforce calling a service in your VPC.</li>
</ul>



<p class="wp-block-paragraph">The direction is named from Salesforce&#8217;s point of view, not yours, and getting it backwards means provisioning the wrong connection.</p>



<p class="wp-block-paragraph">AppFlow can ride on this too. Choosing the PrivateLink option when you create the Salesforce connection has AppFlow route through Private Connect and manage the endpoint lifecycle for you, which is a meaningful simplification over building it by hand.</p>



<p class="wp-block-paragraph">Where I&#8217;d push back: Private Connect is a licensed add-on, and each VPC needs its own connection, so a multi-account AWS estate with separate VPCs per environment multiplies the cost. If the driver is a checkbox on a questionnaire rather than a real threat model, TLS with strict IP allowlisting on the Salesforce side and a well-scoped connected app gets you most of the way for nothing. If the driver is a regulator who wants to see that customer data never touches a public network path, Private Connect is the clean answer and the audit conversation ends quickly. Decide which one you&#8217;re in before you start pricing it.</p>



<h2 class="wp-block-heading">The reconciliation job nobody builds</h2>



<p class="wp-block-paragraph">This is the part that separates an integration that survives from one that quietly rots, and it&#8217;s about thirty lines of code plus a schedule.</p>



<p class="wp-block-paragraph">Run a count comparison on a window that both sides can express. Nightly is plenty:</p>



<pre class="wp-block-code"><code>-- Salesforce side, via the REST query endpoint
SELECT COUNT(Id) FROM Opportunity
WHERE LastModifiedDate = LAST_N_DAYS:1

-- AWS side, same window, in Athena
SELECT COUNT(DISTINCT id) FROM crm.opportunity
WHERE last_modified_date &gt;= date_add('day', -1, current_date)</code></pre>



<p class="wp-block-paragraph">If the two numbers differ by more than a small tolerance for clock skew at the window edges, alert. That single check catches a stopped relay, a failed flow, a mapping that silently dropped a field, and a partner event source sitting in <code>PENDING</code>. It catches all of them without you having to instrument each path separately, which is exactly why it&#8217;s worth building first.</p>



<p class="wp-block-paragraph">For the delete problem, Salesforce exposes a dedicated endpoint. Ask it for records removed in a window and apply those deletions on the AWS side:</p>



<pre class="wp-block-code"><code>GET /services/data/vXX.X/sobjects/Opportunity/deleted/
    ?start=&lt;ISO-8601 timestamp, URL encoded&gt;
    &amp;end=&lt;ISO-8601 timestamp, URL encoded&gt;</code></pre>



<p class="wp-block-paragraph">The window you can ask about is bounded by how long deleted records stay recoverable, so run it more often than that bound, not less. Change data capture delivers delete events too, which is the better answer when it&#8217;s available, but the endpoint is what you use to backfill after a gap, and after a 72-hour gap it&#8217;s the <em>only</em> thing that will tell you what you missed.</p>



<p class="wp-block-paragraph">Put both checks on one dashboard alongside your AppFlow run status and EventBridge failed-invocation metrics. CloudWatch will do it, and if you&#8217;re already running Grafana Cloud or Datadog for the rest of the estate, pull the Salesforce-side numbers in as a custom metric so one panel answers &#8220;are the two systems agreeing right now&#8221;.</p>



<h2 class="wp-block-heading">Troubleshooting a Salesforce AWS integration</h2>



<ul class="wp-block-list">
<li><strong>Relay is running, nothing arrives in EventBridge.</strong> Run <code>describe-event-source</code>. If the state is <code>PENDING</code>, the bus was never created or has been deactivated, and everything published so far is gone. Create the bus, then treat the gap as a backfill job.</li>

<li><strong>Relay won&#8217;t start, or flips to an error state.</strong> Query <code>EventRelayFeedback</code> for the config. The usual causes are a wrong AWS account ID in the named credential, a region in lowercase, or the channel having no members.</li>

<li><strong>Events arrive but no rule matches.</strong> Check the rule is attached to the partner event bus, not the default bus. This one costs people an afternoon. Match on the source prefix and the detail type, and test with an event pattern before wiring a real target.</li>

<li><strong>An AppFlow flow that ran for months suddenly fails.</strong> Look for a field in the mapping that no longer exists in the source object. Also check for a bad gateway error, which points at a connection configured with the wrong instance URL and is fixed by deleting the connection and recreating it rather than editing it.</li>

<li><strong>API destination returns 401 intermittently.</strong> The connection refreshes its OAuth token when it sees a 401 or 407. Persistent 401s usually mean the connected app has no usable scopes for the client credentials flow, or the run-as user was changed. Check the connected app&#8217;s scopes before you touch anything in AWS.</li>

<li><strong>Everything times out at almost exactly five seconds.</strong> That&#8217;s the API destination timeout, not Salesforce being slow. Move the work behind an inbound platform event so the HTTP call returns immediately and the processing happens asynchronously in the org.</li>

<li><strong>Unexplained API limit exhaustion.</strong> Sum the scheduled AppFlow runs against page counts before blaming a rogue integration user. A frequent flow across several objects adds up faster than most people expect.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Starting the relay before associating the partner event bus, and assuming the events queued somewhere.</li>

<li>Treating the 72-hour retention as a comfortable buffer. It&#8217;s three days including a weekend, and an unmonitored relay burns through it without a single alert.</li>

<li>Using change data capture as a replacement for an initial load. It streams changes from the moment you enable it; it doesn&#8217;t backfill history.</li>

<li>Building the whole integration bidirectionally through one mechanism, then discovering you&#8217;ve created an echo loop where a write from AWS fires a change event that triggers another write.</li>

<li>Sizing the AppFlow schedule for freshness without checking what it does to the org&#8217;s API allocation.</li>

<li>Leaving the default 24-hour EventBridge retry policy on an API destination with no dead letter queue.</li>

<li>Provisioning Private Connect for every VPC because it&#8217;s the secure-sounding option, without modelling the per-connection cost across environments.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Build the reconciliation check before you build the second integration path. It&#8217;s the only thing that tells you the whole system is working.</li>

<li>Monitor the partner event source state, not just rule invocations. Invocations go to zero on a quiet day and on a broken day, and the state field distinguishes them.</li>

<li>Give every integration its own connected app and its own integration user. Shared credentials make it impossible to attribute API consumption or revoke one path without breaking three.</li>

<li>Make writes idempotent with external IDs and upsert. Every path here retries, and some retry aggressively.</li>

<li>Version your event payloads. Platform events are immutable once published, and adding a required field later means a new event definition, not an edit.</li>

<li>Keep the AWS-side resource names discoverable from the Salesforce side. Put the flow name or event bus name in the named credential label so a Salesforce admin can see what&#8217;s attached before they change something.</li>

<li>Define the whole thing in Terraform or CloudFormation on the AWS side and in metadata on the Salesforce side, so a sandbox refresh doesn&#8217;t leave you clicking through two consoles from memory.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need MuleSoft to integrate Salesforce with AWS?</h3>



<p class="wp-block-paragraph">No. For the four patterns above, the native services cover it without an integration platform. MuleSoft, Boomi, and similar tools earn their place when you&#8217;re orchestrating across many systems, need transformation logic that lives outside both platforms, or have a governance requirement for a single integration layer. For a two-system integration, they&#8217;re usually more machinery than the problem needs.</p>



<h3 class="wp-block-heading">Does Event Relay use my Salesforce API allocation?</h3>



<p class="wp-block-paragraph">Event delivery counts against your platform event delivery allocation, which is a separate budget from the 24-hour API request limit that REST calls and AppFlow polling draw on. That separation is one of the strongest arguments for the event-driven path when API consumption is already tight in the org.</p>



<h3 class="wp-block-heading">Can I replay Salesforce events that AWS missed?</h3>



<p class="wp-block-paragraph">Within the 72-hour retention window, yes, using replay IDs from a durable subscriber. Beyond it, no. The events are purged and there&#8217;s no recovery path through the event bus. Backfilling from that point means querying the objects directly and using the deleted-records endpoint to catch removals, which is exactly why the reconciliation job matters.</p>



<h3 class="wp-block-heading">Should I use AppFlow or write my own Lambda extractor?</h3>



<p class="wp-block-paragraph">Start with AppFlow. It removes the auth handling, the pagination, and the scheduling, and for standard object pulls into S3 or Redshift it&#8217;s hard to justify writing that yourself. Move to a custom extractor when you need transformation logic AppFlow can&#8217;t express, when you&#8217;re hitting connector limitations on a specific object, or when you want tighter control over API consumption than a fixed schedule allows.</p>



<h3 class="wp-block-heading">How do I send data from AWS back into Salesforce without a Lambda?</h3>



<p class="wp-block-paragraph">EventBridge API destinations call the Salesforce REST API directly using OAuth client credentials from a connected app. Point them at an inbound platform event endpoint so the write returns fast and processing happens asynchronously in the org. You still need a Lambda if you require mTLS or payload transformation beyond what the input transformer handles.</p>



<h3 class="wp-block-heading">Is Private Connect required for a secure Salesforce AWS integration?</h3>



<p class="wp-block-paragraph">Not required. Everything runs over TLS by default. Private Connect removes the public internet from the path entirely, which matters when a regulator or a security review asks you to demonstrate it. It&#8217;s a licensed add-on with a per-connection cost, so model it against your actual VPC count before committing.</p>



<h3 class="wp-block-heading">Why does my EventBridge rule never fire even though events are arriving?</h3>



<p class="wp-block-paragraph">Almost always because the rule is on the default event bus rather than the partner event bus. Partner events land on their own bus and rules are bus-scoped, so a correct pattern on the wrong bus matches nothing and reports nothing.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A Salesforce AWS integration doesn&#8217;t usually break loudly. It drifts. A relay stops, a mapping goes stale, a delete never propagates, and the two systems keep answering questions confidently with different numbers. The services themselves are well built and the setup is genuinely not hard once you know the region goes in uppercase and the bus has to be associated first.</p>



<p class="wp-block-paragraph">So pick the path by direction and shape, wire the failure modes you&#8217;ve accepted into alerts, and build the nightly count comparison before you build anything clever. That one job is worth more than any amount of pipeline polish, because it&#8217;s the only thing in the architecture that can tell you the whole thing is still true.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your Salesforce and AWS setup?</h2>



<p class="wp-block-paragraph">I work with teams on the infrastructure side of CRM integrations, usually where the data is disagreeing and nobody can prove why. Things I can help with:</p>



<ul class="wp-block-list">
<li>Reviewing an existing Salesforce to AWS pipeline and finding where records are being lost between the two.</li>

<li>Standing up Event Relay into EventBridge end to end, including the partner bus association and the monitoring that proves it&#8217;s live.</li>

<li>Designing the AppFlow schedule and object set so it doesn&#8217;t eat the org&#8217;s API allocation.</li>

<li>Building the reconciliation and delete-propagation jobs, with alerting in CloudWatch, Grafana, or whatever you already run.</li>

<li>Writing the write-back path properly: Bulk API jobs, idempotent upserts, dead letter queues, and credential rotation through Secrets Manager.</li>

<li>Working out whether Private Connect is worth the licence in your specific environment, or whether TLS and tighter scoping gets you there.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;ve got a flow that&#8217;s failing, a relay stuck in an error state, or two record counts that won&#8217;t agree, send me the config or the error and I&#8217;ll tell you what I&#8217;d check first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-aws-integration-architecture/">Salesforce AWS Integration Architecture: The Four Paths and What Breaks Silently</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>SAP Data Integration with AWS Glue: The Failure Modes Nobody Demos</title>
		<link>https://john-nessime.com/blog/devops/sap-data-integration-aws-glue/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 18 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Change Data Capture]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Delta Token]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[ODP]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[SAP]]></category>
		<category><![CDATA[SAP OData]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[VPC]]></category>
		<category><![CDATA[Zero-ETL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=213</guid>

					<description><![CDATA[<p>The AWS Glue SAP OData connector demos beautifully and then drifts. A practical look at delta token invalidation, deletes that never delete, partitioning limits on ODP entities, and the licensing line between ODP-RFC and ODP-OData.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/sap-data-integration-aws-glue/">SAP Data Integration with AWS Glue: The Failure Modes Nobody Demos</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The Glue job had been green for weeks. Every run finished, every run wrote Parquet, nobody had a reason to look at it. Then someone from finance mentioned that a report was short by a few hundred rows, and the reconciliation that followed turned up the real problem: the job had not actually been reading changes for most of that time. It had been reading nothing, successfully.</p>



<p class="wp-block-paragraph">That is the shape of most trouble with SAP data integration with AWS Glue. The connector does not usually crash. It returns an empty or partial result set, the job exits zero, and your data lake quietly diverges from the source system. By the time anyone notices, you have no idea which run was the last correct one.</p>



<p class="wp-block-paragraph">This post covers the parts that bite after the proof of concept: how delta tokens actually behave, why deleted records keep showing up in your reports, where partitioning stops working, and the licensing distinction between two things that both get called &#8220;ODP&#8221; and are not the same thing at all. There is a troubleshooting section at the end for when a job is already broken.</p>



<h2 class="wp-block-heading">First, get the ODP terminology right, because licensing depends on it</h2>



<p class="wp-block-paragraph">Operational Data Provisioning is SAP&#8217;s framework for exposing extraction interfaces in a delta-aware way. A provider (a BW extractor, an ABAP CDS view, SLT, a HANA information view) writes changes into an Operational Delta Queue, and subscribers pull from that queue. This part is uncontroversial.</p>



<p class="wp-block-paragraph">What matters is that there are two ways to talk to ODP, and SAP treats them very differently. SAP Note 3255746 restricts third-party and customer applications from using the RFC modules of the ODP Data Replication API against ABAP sources. That note has been sharpened over several revisions, from &#8220;unsupported&#8221; to &#8220;not permitted&#8221;, and SAP has stated it reserves the right to introduce technical measures and audit unpermitted use. A number of established extraction tools were built directly on ODP-RFC and have had to move.</p>



<p class="wp-block-paragraph">The AWS Glue SAP OData connector takes the other path. It talks OData over HTTPS to an activated SAP Gateway service, and uses the ODP framework through that OData layer rather than through RFC. That is the route SAP points customers at. If you are being asked in an architecture review whether your pipeline is on the wrong side of Note 3255746, this distinction is the whole answer, and it is worth having the note number written down before the meeting.</p>



<p class="wp-block-paragraph">Two honest caveats. First, notes get revised, so read the current version rather than a vendor blog summarising it, including this one. Second, &#8220;permitted&#8221; is not the same as &#8220;free&#8221;: you still need the OData service activated, a service user with the right authorisations, and a Basis team willing to own the queue growth on their side.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: the delta token is shared state you do not own</h2>



<p class="wp-block-paragraph">This is the one that produces silent drift, and it is the reason the finance report was short.</p>



<p class="wp-block-paragraph">When you read an ODP-enabled entity with change data capture switched on, the connector asks SAP to initialise a delta queue subscription. SAP hands back a delta token. That token is your bookmark. On the next run you pass it back, and you get only what changed since.</p>



<pre class="wp-block-code"><code>sapodata_df = glueContext.create_dynamic_frame.from_options(
    connection_type="SAPOData",
    connection_options={
        "connectionName": "sap-prod-odata",
        "ENTITY_NAME": "entityName",
        "ENABLE_CDC": "true"
    },
    transformation_ctx="initial_load"
)</code></pre>



<p class="wp-block-paragraph"><code>ENABLE_CDC</code> defaults to false. Leave it off and you get a full load every single run, which is expensive but at least obvious. Turn it on and the token appears in the <code>DELTA_TOKEN</code> column, populated only on the last row of the frame. That last row is not a business record. It is a carrier for the token, and if you write your frame straight to S3 without stripping it you have just added a junk row to your table.</p>



<p class="wp-block-paragraph">The subsequent run passes the token back:</p>



<pre class="wp-block-code"><code>sapodata_df = glueContext.create_dynamic_frame.from_options(
    connection_type="SAPOData",
    connection_options={
        "connectionName": "sap-prod-odata",
        "ENTITY_NAME": "entityName",
        "DELTA_TOKEN": last_known_token
    },
    transformation_ctx="incremental_load"
)</code></pre>



<h3 class="wp-block-heading">Why this goes wrong</h3>



<p class="wp-block-paragraph">A delta token is bound to a service collection <em>and a user</em>. If a new initialisation runs with <code>ENABLE_CDC</code> set true for that same collection and user, SAP invalidates every token previously issued from the earlier initialisation. Not &#8220;expires eventually&#8221;. Invalidates, immediately.</p>



<p class="wp-block-paragraph">So picture two Glue jobs sharing one service account, both reading the same extractor. Someone redeploys job B with the CDC flag still set from testing. Job B re-initialises. Job A&#8217;s stored token is now dead, and the next time job A runs you get:</p>



<pre class="wp-block-code"><code>Could not open data access via extraction API RODPS_REPL_ODP_OPEN</code></pre>



<p class="wp-block-paragraph">An error is the good outcome. The bad outcome is code that catches the exception, falls back to a full load, and writes it over the top of an incremental table with no dedupe. Or code that re-initialises on failure, which fixes job A and breaks job B, and now the two jobs take turns invalidating each other forever while both keep exiting zero.</p>



<p class="wp-block-paragraph">Practical rules that come out of this:</p>



<ul class="wp-block-list">
<li>One SAP service user per consuming pipeline. Sharing a user is what couples independent jobs together.</li>

<li>Persist the token somewhere durable and versioned. DynamoDB, Parameter Store, S3 with versioning enabled, all fine. Job bookmarks are not a substitute, because the token is source state, not read position.</li>

<li>Never auto re-initialise on error. Fail loudly, page someone, and make the reinitialisation a deliberate human action with a documented backfill.</li>

<li>Alert on zero-change runs. A pipeline that returns no changes for several consecutive runs against a busy table is telling you something, and it is rarely &#8220;business is quiet&#8221;.</li>
</ul>



<h3 class="wp-block-heading">Retention on the SAP side is a hard deadline</h3>



<p class="wp-block-paragraph">The delta queue is not infinite. In transaction <code>ODQMON</code> the Basis team configures retention, and a reorganisation job cleans up data past that window. If your pipeline is down longer than the retention period, the changes you missed are gone from the queue and no token will bring them back. You are into a full reload and a reconciliation.</p>



<p class="wp-block-paragraph">Two things to find out before you go live: what the retention setting is, and whether the cleanup job is actually scheduled. Then make sure your pipeline&#8217;s alerting escalates well inside that window. A weekend outage on a queue with a short retention is a very expensive Monday.</p>



<p class="wp-block-paragraph">Related: if anyone has been testing extraction with <code>RODPS_REPL_TEST</code>, that program registers as a real additional subscriber rather than simulating one. Because the queue is shared and data is only cleaned once every subscriber has collected it, forgotten test subscriptions hold delta data indefinitely and the underlying queue tables grow. Ask whoever ran the tests to reset those subscriptions in ODQMON.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family two: deletes arrive as a column, not as a deletion</h2>



<p class="wp-block-paragraph">CDC extraction from ODP entities returns a couple of extra fields alongside your business columns. <code>DML_STATUS</code> carries <code>UPDATED</code> for inserts and updates, and <code>DELETED</code> for records removed at source. <code>GLUE_FETCH_SQ</code> is a per-record sequence derived from an epoch timestamp, in receipt order, and it is your tie-breaker when the same key changes twice inside one delta.</p>



<p class="wp-block-paragraph">Nothing acts on those fields for you. If your job appends the frame to S3 and calls it done, a deleted sales order is now a row in your lake with <code>DML_STATUS = DELETED</code> and every other column populated exactly as before. Any query that does not filter on that column counts it. Cancelled orders keep appearing in revenue. Closed accounts stay open. The pipeline is green, the numbers are wrong, and the discrepancy grows slowly enough that nobody catches it early.</p>



<p class="wp-block-paragraph">You have to apply the changes rather than accumulate them. Broadly two options:</p>



<ol class="wp-block-list">
<li><strong>Merge into a table format that supports it.</strong> Apache Iceberg, which Glue supports natively, gives you a real merge with delete handling. Land the raw delta, then merge on the business key with <code>GLUE_FETCH_SQ</code> deciding the winner within a batch, deleting where the status says deleted.</li>

<li><strong>Keep an append-only history and build a current-state view.</strong> Cheaper to write, and it preserves an audit trail. It also means every downstream consumer has to use the view, and someone eventually will not.</li>
</ol>



<p class="wp-block-paragraph">I reach for Iceberg first on anything that feeds reporting, because the correctness lives in the table rather than in the discipline of whoever writes the next query. Append-only is the right call when you genuinely need history, or when the consumer is a downstream warehouse like Amazon Redshift that will do its own merge anyway.</p>



<p class="wp-block-paragraph">Worth knowing: not every ODP provider emits deletes reliably. Some older extractors express them as reversal records rather than a delete flag. Confirm the behaviour for your specific entity with the functional owner before you build merge logic on an assumption.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family three: pagination and partitioning do not compose</h2>



<p class="wp-block-paragraph">The connector&#8217;s default page size is 50,000 records, and you can raise it up to 500,000. Anything above that is ignored and the maximum is used instead. Raising the page size cuts the number of round trips, which is usually the dominant cost on a wide table across a slow link.</p>



<p class="wp-block-paragraph">The catch is what your SAP Gateway will tolerate. A large page means a large response the ABAP stack has to assemble and hold, and Gateway or an intervening load balancer may cut it off before it completes. Symptoms are timeouts, connection resets, or a job that appears to paginate without ever finishing. Tune the page size with your Basis team, not in isolation, and treat it as a negotiation between round trips and response size rather than a value you set once.</p>



<p class="wp-block-paragraph">On partitioning, there is a specific documented limit that catches people out: ODP entities are not compatible with record-based partitioning, because pagination is already handled through skip and delta tokens. For record-based partitioning, the default concurrency is forced to null regardless of what you asked for. So the obvious lever for making a slow ODP extraction faster is not available.</p>



<p class="wp-block-paragraph">What you can still do on non-ODP OData entities is field-based partitioning, splitting a read across a bounded range:</p>



<ul class="wp-block-list">
<li><code>PARTITION_FIELD</code> is the field to split on, typically a date.</li>

<li><code>LOWER_BOUND</code> is inclusive, <code>UPPER_BOUND</code> is exclusive. Getting that backwards is how you duplicate or drop a boundary day.</li>

<li><code>NUM_PARTITIONS</code> defaults to 1, so leaving it out means you did not partition at all.</li>

<li>Where a limit and a partition are both applied, the limit wins. That is the usual cause of &#8220;I partitioned it and got fewer rows&#8221;.</li>
</ul>



<p class="wp-block-paragraph">For non-ODP entities you can also do timestamp-based incremental reads with <code>filteringExpression</code>, pointing at whatever last-changed field the entity exposes. Note the casing: that option is camelCase while most of the others are upper snake case, which trips people up when a filter appears to be ignored.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family four: connection, auth and network</h2>



<p class="wp-block-paragraph">These fail early and loudly, which makes them the easy ones. The trap is that they fail during setup, when you have three people on a call and someone suggests loosening something to move on.</p>



<p class="wp-block-paragraph">The connector supports two authentication types: CUSTOM, which is SAP basic authentication, and OAUTH2. The custom secret in AWS Secrets Manager takes this shape:</p>



<pre class="wp-block-code"><code>{
  "basicAuthUsername": "&lt;your SAP username&gt;",
  "basicAuthPassword": "&lt;your SAP password&gt;",
  "basicAuthDisableSSO": "&lt;True/False&gt;",
  "customAuthenticationType": "CustomBasicAuth"
}</code></pre>



<p class="wp-block-paragraph">For OAuth 2.0 the client secret goes into Secrets Manager under <code>USER_MANAGED_CLIENT_APPLICATION_CLIENT_SECRET</code>, and the client ID, authorisation code URL, token URL and scopes are supplied on the connection itself.</p>



<p class="wp-block-paragraph">Basic auth is genuinely easier for a scheduled pipeline: no interactive redirect, no refresh token to keep alive, and rotation is a Secrets Manager operation. OAuth gets you short-lived tokens and revocation you can actually reason about, which is what a security review will want. Both are defensible. What is not defensible is one shared service user across every pipeline, for the delta token reasons above as much as the audit ones.</p>



<p class="wp-block-paragraph">A few things that reliably waste a morning:</p>



<ul class="wp-block-list">
<li>The service path on the connection is the <em>catalog</em> service path, not the path to your object. The entity path goes in <code>ENTITY_NAME</code> on the job.</li>

<li>The OData service has to be activated in the SAP system. Until it is, nothing you do on the AWS side matters.</li>

<li>For a non-VPC connection, the application host URL must be reachable from the public internet. If your SAP system sits in a private network, configure VPC connectivity properly rather than opening it up.</li>

<li>The Glue role needs Secrets Manager read plus the EC2 network interface permissions when a VPC is involved. A missing <code>ec2:CreateNetworkInterface</code> shows up as a connection failure that looks like a network problem.</li>

<li>Private certificate authorities need the chain in a form Glue accepts. If you are wiring this up yourself, expect it to take a pass or two.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Choosing your path: Glue jobs, zero-ETL, or something else</h2>



<p class="wp-block-paragraph">The same connector is reachable through more than one product surface, and the choice is really about how much state you want to own.</p>



<ul class="wp-block-list">
<li><strong>Glue Studio or script jobs.</strong> You control page size, filters, output format, transformation and scheduling, and you own delta token persistence. Pick this when you need real transformation in flight, or when the merge logic has to live next to the extraction.</li>

<li><strong>Glue zero-ETL integrations.</strong> AWS manages token state internally, runs the initial full load and then the incremental cycle, and preserves the token on failure so retries do not lose data. Less code, less control. Worth noting that once an incremental update field is chosen for a connection and entity, a later integration on the same pair cannot pick a different one.</li>

<li><strong>Amazon AppFlow.</strong> Simpler, flow-shaped, well suited to modest volumes and quick prototypes. Less appropriate once you need Spark-scale processing.</li>

<li><strong>Commercial replication tools.</strong> Fivetran, Airbyte and SAP-specialist vendors all play here. They cost money and add a vendor, and in exchange they absorb exactly the state management this post is about. If your team has no SAP depth, that trade is often correct.</li>
</ul>



<p class="wp-block-paragraph">On cost, Glue bills for the resources a job consumes for as long as it runs, so the levers are the same as any Spark workload: fewer round trips, less data pulled, fewer idle executors, and not re-reading history you already have. Rates change and vary by region, so model it against current pricing rather than a figure from a blog post. If your incremental run is pulling the same volume as your initial load, the delta mechanism is not working and the bill is the symptom.</p>



<p class="wp-block-paragraph">Also worth writing back. The connector supports write operations to SAP OData, using <code>INSERT</code> and <code>UPDATE</code>, with <code>ID_FIELD_NAMES</code> required on update to identify the external ID field. Useful for enrichment loops, and something to treat with a good deal more caution than a read.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">&#8220;Could not open data access via extraction API RODPS_REPL_ODP_OPEN&#8221;</h3>



<p class="wp-block-paragraph">Your token is invalid or expired. Find out why before you re-initialise, because re-initialising destroys the evidence. Check ODQMON for a recent initialisation on that collection and user, check whether another job or a test program shares the credentials, and check whether the request you wanted has aged out of retention. Then re-initialise deliberately and reconcile the gap.</p>



<h3 class="wp-block-heading">The job succeeds but returns no rows</h3>



<p class="wp-block-paragraph">Most often a stale or wrong token, a filter predicate that matches nothing, or CDC never having been initialised so the connector has nothing to hand back. Run once without <code>ENABLE_CDC</code> against a small filtered slice and confirm you get data at all. That separates &#8220;no changes&#8221; from &#8220;not reading&#8221;.</p>



<h3 class="wp-block-heading">Pagination never terminates</h3>



<p class="wp-block-paragraph">Usually the SAP side enforcing its own page limit under your requested size, so the connector keeps requesting more. Reduce the page size, verify the filter is actually being applied server side rather than after retrieval, and have Basis confirm what Gateway is capping.</p>



<h3 class="wp-block-heading">Row counts drift over time</h3>



<p class="wp-block-paragraph">Almost always unhandled <code>DML_STATUS = DELETED</code> records, or a token invalidation that silently turned an incremental into a partial. Count distinct business keys in your target against a direct count in SAP for a bounded date range. Make that comparison a scheduled check, not something you run when someone complains.</p>



<h3 class="wp-block-heading">Duplicates after a rerun</h3>



<p class="wp-block-paragraph">A delta was consumed, the write failed, and the token advanced anyway. Store the token only after the write commits, never before. If you are appending rather than merging, add a dedupe on business key ordered by <code>GLUE_FETCH_SQ</code>.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Sharing one SAP service user across multiple pipelines, which couples their delta tokens together.</li>

<li>Automatic re-initialisation in an exception handler. It converts a loud failure into permanent silent drift.</li>

<li>Writing the token-carrier row into the target table as if it were data.</li>

<li>Appending CDC output and ignoring <code>DML_STATUS</code>.</li>

<li>Persisting the token before the write succeeds.</li>

<li>Assuming an entity is ODP-enabled because it looks analytical. Confirm it, because the incremental mechanism differs.</li>

<li>Leaving test subscriptions from <code>RODPS_REPL_TEST</code> registered against the queue.</li>

<li>Building the pipeline without ever asking Basis what the ODQ retention window is.</li>
</ul>



<h2 class="wp-block-heading">Best practices for SAP data integration with AWS Glue</h2>



<ul class="wp-block-list">
<li>Treat the delta token as production state. Versioned storage, a documented recovery procedure, and a written-down answer to &#8220;how do we backfill&#8221;.</li>

<li>One service user per pipeline, one connection per environment. Isolation is the cheapest fix for the worst failure.</li>

<li>Land raw, merge second. Keep the untouched connector output so you can replay a merge without going back to SAP.</li>

<li>Alert on absence, not just errors. Zero rows for several consecutive runs against a busy entity should page someone. Push Glue metrics into CloudWatch and forward them wherever you already look, whether that is Grafana Cloud, Datadog or your own Prometheus.</li>

<li>Reconcile on a schedule against SAP for a bounded window, and record the result. Drift you can see is a bug; drift you cannot see is an incident.</li>

<li>Start with a filtered slice and a small page size while you are building. Get it correct, then tune throughput.</li>

<li>Write down what each field means while the functional owner is still on the call. SAP field names do not explain themselves later.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does the AWS Glue SAP OData connector fall under SAP Note 3255746?</h3>



<p class="wp-block-paragraph">The note restricts the RFC modules of the ODP Data Replication API. The Glue connector uses OData over HTTPS against an activated Gateway service, which is the interface SAP directs customers toward. Read the current revision of the note yourself and get it confirmed with your SAP account team before you rely on any summary, including this one.</p>



<h3 class="wp-block-heading">Where should I store the delta token?</h3>



<p class="wp-block-paragraph">Anywhere durable and versioned that your job can write to transactionally: DynamoDB, Parameter Store, or S3 with versioning. What matters more than the store is the ordering. Write it only after the target write commits, and keep the previous value so you can roll back after a bad run.</p>



<h3 class="wp-block-heading">Can I run two Glue jobs against the same SAP entity?</h3>



<p class="wp-block-paragraph">Yes, but give them separate SAP service users. A delta token is scoped to the service collection and the user, so two jobs on one user will invalidate each other&#8217;s tokens whenever either re-initialises.</p>



<h3 class="wp-block-heading">Why do deleted SAP records still appear in my data lake?</h3>



<p class="wp-block-paragraph">Because the connector reports them rather than applying them. Deletes come back as rows with <code>DML_STATUS</code> set to <code>DELETED</code>. If your pipeline appends without a merge, those rows sit in the target looking like live data. Merge into Iceberg, or filter through a current-state view that every consumer actually uses.</p>



<h3 class="wp-block-heading">What page size should I use?</h3>



<p class="wp-block-paragraph">The default is 50,000 and the ceiling is 500,000. Larger pages mean fewer round trips but bigger responses for SAP to assemble, and Gateway or a load balancer may cut them short. Start at the default, raise it while watching for timeouts, and agree the ceiling with your Basis team.</p>



<h3 class="wp-block-heading">Can I partition an ODP extraction for speed?</h3>



<p class="wp-block-paragraph">Not with record-based partitioning. ODP entities handle pagination through skip and delta tokens, so concurrency is forced to null there regardless of your input. Field-based partitioning on a bounded range is available for non-ODP OData entities. For ODP, tune page size and narrow the entity instead.</p>



<h3 class="wp-block-heading">Can Glue write data back into SAP?</h3>



<p class="wp-block-paragraph">Yes. The SAP OData connector supports insert and update operations, with the external ID field named through <code>ID_FIELD_NAMES</code> on updates. Test that against a sandbox client with a functional owner watching, because a bad write into an ERP system is a very different class of incident to a bad read.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">SAP data integration with AWS Glue is not hard because the connector is difficult to configure. Configuration takes an afternoon. It is hard because the correctness of your pipeline depends on a token held in a queue on a system your team does not administer, subject to a retention policy you may not have been told about, and shared with every other subscriber using the same credentials.</p>



<p class="wp-block-paragraph">Design for that. Isolate the users, persist the token deliberately, apply deletes rather than collecting them, and alert on silence as loudly as you alert on errors. Do that and the connector is a good piece of engineering that will run for a long time without your attention. Skip it and you get a job that stays green while the numbers quietly stop being true.</p>



<h2 class="wp-block-heading">Need a hand with an SAP to AWS pipeline?</h2>



<p class="wp-block-paragraph">This is the kind of work I take on regularly. If you are standing one up, or you have one that has started drifting, I can help with:</p>



<ul class="wp-block-list">
<li>Designing delta token state management that survives failed runs, redeploys and concurrent pipelines.</li>

<li>Building merge logic that applies <code>DML_STATUS</code> deletes correctly into Iceberg or Redshift instead of accumulating ghost rows.</li>

<li>Diagnosing <code>RODPS_REPL_ODP_OPEN</code> failures, silent zero-row runs and row count drift, and working out how far back the damage goes.</li>

<li>Setting up SAP OData connections properly: Secrets Manager, least-privilege IAM, VPC connectivity and private certificate authorities.</li>

<li>Tuning page size, filters and job sizing so incremental runs cost what an incremental run should cost.</li>

<li>Adding reconciliation and absence alerting so drift shows up in a dashboard rather than in a finance report.</li>
</ul>



<p class="wp-block-paragraph">Send me a job log, a connection config, or the row counts that stopped matching, and I will tell you what I think is going on.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/sap-data-integration-aws-glue/">SAP Data Integration with AWS Glue: The Failure Modes Nobody Demos</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Zendesk Data Integration with AWS Glue Zero-ETL: The Delete Gap That Skews Your Numbers</title>
		<link>https://john-nessime.com/blog/technical-guides/zendesk-aws-glue-zero-etl/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 16 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Change Data Capture]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Customer Support Analytics]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Zendesk]]></category>
		<category><![CDATA[Zero-ETL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=223</guid>

					<description><![CDATA[<p>AWS Glue zero-ETL replicates seven Zendesk entities, but only three of them ever remove a row. Here is how that gap quietly skews CSAT and knowledge base counts, plus the three IAM layers to wire, the two settings you cannot change after creation, and the CloudWatch metrics that make drift visible before someone spots it in a meeting.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/zendesk-aws-glue-zero-etl/">Zendesk Data Integration with AWS Glue Zero-ETL: The Delete Gap That Skews Your Numbers</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The support lead pings you on a Tuesday. The CSAT figure on the exec dashboard is higher than the one in Zendesk Explore, and somebody noticed during the meeting. Nothing failed. The Glue integration is green. CloudWatch shows ingestion succeeding on every interval. And yet the warehouse and the source disagree, and have quietly disagreed for weeks.</p>



<p class="wp-block-paragraph">That divergence is not a bug. It is documented behaviour, sitting in one table in the AWS docs that almost everyone scrolls past. This post covers what Zendesk data integration with AWS Glue zero-ETL actually replicates, the entities where deletes silently never propagate, the configuration choices you cannot undo after creation, how to wire the three IAM roles involved, and how to monitor the thing so drift shows up as an alarm instead of a meeting.</p>



<h2 class="wp-block-heading">What Zendesk AWS Glue zero-ETL actually gives you</h2>



<p class="wp-block-paragraph">Zero-ETL is AWS&#8217;s name for managed replication. You create a Glue connection to Zendesk, pick the entities you want, pick a target, and Glue handles the initial snapshot, the schema mapping, and ongoing change data capture. No Spark job. No bookmark logic. No pagination code hammering the Zendesk API and no retry handling for it.</p>



<p class="wp-block-paragraph">Supported targets are a general purpose Amazon S3 bucket through the lakehouse architecture of Amazon SageMaker, S3 Tables through the same lakehouse, Redshift Managed Storage, or an Amazon Redshift data warehouse directly. Data lands as Apache Iceberg, which is what makes row-level updates and deletes viable on object storage at all.</p>



<p class="wp-block-paragraph">What you give up is control. No transform step, no server-side filter, no column pruning at ingest. You get the entity as the connector sees it and you shape it downstream. For most support analytics that trade is fine. It stops being fine the moment your compliance team asks why deleted records are still queryable.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The delete gap: four of seven entities never remove a row</h2>



<p class="wp-block-paragraph">AWS Glue zero-ETL supports seven Zendesk entities. Only three of them replicate deletes.</p>



<ul class="wp-block-list">
<li><strong>tickets</strong> &#8211; create, update and delete all propagate</li>



<li><strong>users</strong> &#8211; create, update and delete all propagate</li>



<li><strong>organizations</strong> &#8211; create, update and delete all propagate</li>



<li><strong>satisfaction-rating</strong> &#8211; create and update only, <strong>no delete</strong></li>



<li><strong>articles</strong> &#8211; create and update only, <strong>no delete</strong></li>



<li><strong>calls</strong> &#8211; create and update only, <strong>no delete</strong></li>



<li><strong>legs</strong> (call legs) &#8211; create and update only, <strong>no delete</strong></li>
</ul>



<p class="wp-block-paragraph">When a satisfaction rating disappears in Zendesk, the row stays in your lakehouse. Permanently. Nothing errors, <code>IngestionSucceeded</code> fires as usual, and the row count in that table only ever goes up.</p>



<p class="wp-block-paragraph">Now think about what your CSAT query does. It averages a score column across a set of rows. If removed ratings never leave the target, your average drifts further from the source every cycle, and the skew is rarely random because retracted or disputed ratings are exactly the ones most likely to be removed. The same mechanism inflates knowledge base article counts, call volume, and any call leg analysis you build. It stays invisible until two dashboards get compared side by side.</p>



<h3 class="wp-block-heading">How to design around it</h3>



<p class="wp-block-paragraph">You cannot make the connector track deletes it does not track. So stop trying, and split the problem instead.</p>



<ol class="wp-block-list">
<li><strong>Run two integrations, not one.</strong> Put tickets, users and organizations in a continuously synced integration. Put the four append-only entities in a second integration you can tear down and rebuild on a cadence that matches how much drift you can tolerate.</li>



<li><strong>Treat the append-only tables as an event log, not as state.</strong> Query them through a view that reconciles against a periodic authoritative count, rather than trusting row presence to mean the record still exists.</li>



<li><strong>Prefer status fields over row existence.</strong> Where an entity exposes a field describing its state, filter on that field downstream. Check which fields your entities actually return before you build on this, because coverage varies by entity.</li>



<li><strong>Reconcile on a schedule.</strong> Pull counts from the Zendesk API for the four entities and compare against the target. Alert on divergence, not on an absolute number.</li>
</ol>



<p class="wp-block-paragraph">Splitting the integration costs you a second set of IAM wiring and a second thing to monitor. It buys you the ability to rebuild the drifting half without touching the half that is correct. That is a good trade.</p>



<h2 class="wp-block-heading">Entity coverage is narrower than the Zendesk API</h2>



<p class="wp-block-paragraph">Seven entities is not the Zendesk API. Ticket comments, ticket metrics, ticket audits, groups, views, macros and SLA policy definitions are not part of the zero-ETL entity set. If your reporting needs first response time, the audit trail behind a status change, or custom field definitions, zero-ETL alone will not get you there.</p>



<p class="wp-block-paragraph">The useful part is that the Glue connection you create is reusable. The same connection backs a normal Glue ETL job, so you can run zero-ETL for the bulk entities and a scheduled Spark job for the ones it does not cover:</p>



<pre class="wp-block-code"><code># Read a Zendesk entity through the same connection
# used by the zero-ETL integration. ENTITY_NAME takes
# the entity name, not the label, e.g. "tickets".
zendesk_read = glueContext.create_dynamic_frame.from_options(
    connection_type="Zendesk",
    connection_options={
        "connectionName": "my-zendesk-connection",
        "ENTITY_NAME": "tickets",
        "API_VERSION": "v2"
    }
)</code></pre>



<p class="wp-block-paragraph">One schema detail worth knowing before you write a single downstream query: the connector converts struct and list types to strings. Zendesk custom fields arrive as arrays of objects, so they land as serialized text, not as a nested column you can address with dot notation. Partitioning is also not supported on the Zendesk source. Plan for a parse step in your silver layer rather than discovering it in a broken dashboard.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Two decisions you cannot change after creation</h2>



<p class="wp-block-paragraph">Most Glue settings are editable. These two are not, and getting them wrong means deleting the integration and starting over.</p>



<h3 class="wp-block-heading">Continuous sync versus on-demand snapshot</h3>



<p class="wp-block-paragraph">By default the integration syncs continuously. Enable the on-demand snapshot setting and it does a single one-time replication with no ongoing CDC. That setting is locked once the integration exists. If you enabled it for a proof of concept and then decided you wanted CDC, you are recreating the integration and re-running the full load.</p>



<h3 class="wp-block-heading">Refresh interval, and why your target choice locks it</h3>



<p class="wp-block-paragraph">The refresh interval controls how often CDC pulls run. It accepts anything from 15 minutes up to 8640 minutes, which is six days. Here is the part that catches people:</p>



<ul class="wp-block-list">
<li>If the target is <strong>Amazon Redshift</strong>, the refresh interval cannot be modified after creation.</li>



<li>For <strong>other targets</strong>, including the SageMaker lakehouse on S3, you can change it later.</li>
</ul>



<p class="wp-block-paragraph">So the target you pick is not only an architecture decision, it is a flexibility decision. If you are not certain what freshness the business actually needs, land in the lakehouse first and query Redshift over it. You keep the ability to tune the interval once you have real usage data instead of a guess made in week one.</p>



<p class="wp-block-paragraph">On cost: AWS does not charge separately for the integration itself. You pay for everything around it, target storage, Redshift or Athena query compute, S3 requests, Data Catalog usage and CloudWatch logs. A 15 minute interval on seven entities produces far more small-file churn and log volume than a 6 hour one. Set the interval from the freshness a human will actually act on, not from the smallest number the field accepts.</p>



<h2 class="wp-block-heading">Wiring the three roles</h2>



<p class="wp-block-paragraph">There are three separate permission layers here and they fail in different ways, which is why &#8220;check IAM&#8221; is useless advice on its own.</p>



<ul class="wp-block-list">
<li><strong>The Zendesk OAuth credential.</strong> The connector uses the authorization code grant, so you get redirected to Zendesk to log in and approve. You can rely on the Glue-managed client application and supply only your instance URL, or register your own OAuth app in the Zendesk admin center and provide your own client ID and secret. The resulting access token does not expire, which is convenient and also means a revoked app in Zendesk is something you will only discover through failed ingestions.</li>



<li><strong>The source role.</strong> This is what lets the integration read through the connection. It is a prerequisite for SaaS sources and it is the step most people miss, because creating the connection successfully does not mean the integration can use it.</li>



<li><strong>The target role and catalog policy.</strong> This is what lets the integration write. A misconfigured catalog resource policy puts the integration into <code>NEEDS_ATTENTION</code>, not into a clear error at creation time.</li>
</ul>



<p class="wp-block-paragraph">The source role policy looks like this. Note <code>glue:RefreshOAuth2Tokens</code>, which is the one people leave out and then spend an afternoon debugging:</p>



<pre class="wp-block-code"><code>{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "GlueConnections",
            "Effect": "Allow",
            "Action": [
                "glue:GetConnections",
                "glue:GetConnection"
            ],
            "Resource": [
                "arn:aws:glue:*:111122223333:catalog",
                "arn:aws:glue:us-east-1:111122223333:connection/*"
            ]
        },
        {
            "Sid": "GlueActionBasedPermissions",
            "Effect": "Allow",
            "Action": [
                "glue:ListEntities",
                "glue:RefreshOAuth2Tokens"
            ],
            "Resource": ["*"]
        },
        {
            "Sid": "CloudWatchLogging",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": ["*"]
        }
    ]
}</code></pre>



<p class="wp-block-paragraph">The trust policy allows <code>glue.amazonaws.com</code> to assume it. Then you attach the role to the connection, which is a CLI-only step and does not appear in the console flow:</p>



<pre class="wp-block-code"><code>aws glue create-integration-resource-property 
  --resource-arn arn:aws:glue:us-east-1:123456789012:connection/my-zendesk-connection 
  --source-processing-properties "{"RoleArn" : "arn:aws:iam::123456789012:role/zendesk-zetl-source"}" 
  --region us-east-1</code></pre>



<p class="wp-block-paragraph">If you skip this, the connection tests fine and the integration still cannot read. That mismatch is the single most common reason a first attempt sits in <code>NEEDS_ATTENTION</code> with no obvious cause.</p>



<h2 class="wp-block-heading">Setting up the integration</h2>



<ol class="wp-block-list">
<li>If you are using your own OAuth app, register it in the Zendesk admin center under the API settings and note the client ID and secret. Store the secret in AWS Secrets Manager, one secret per Glue connection.</li>



<li>In Glue Studio, create a connection under Data Connections. Choose Zendesk as the connection type, supply your instance URL and environment, and complete the OAuth redirect.</li>



<li>Create the source role with the policy above and attach it to the connection using <code>create-integration-resource-property</code>.</li>



<li>Prepare the target: the Glue database or S3 Table bucket, the target role, the catalog resource policy, and any Lake Formation grants your account&#8217;s permission model requires.</li>



<li>Create the integration. Select entities, set the refresh interval, and decide continuous sync versus on-demand snapshot. Both of those are locked from here.</li>



<li>Watch the first full load complete before you point anything at the tables. Full load and CDC are separate load types in the metrics, and the numbers look very different.</li>
</ol>



<p class="wp-block-paragraph">Once it works, move it into code. Glue zero-ETL integrations are supported by CloudFormation and the AWS CDK, which is the difference between a thing one person built in the console and a thing your team can redeploy into another account.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Monitoring: the metrics that expose drift</h2>



<p class="wp-block-paragraph">Zero-ETL publishes to the <code>AWS/Glue/ZeroETL</code> CloudWatch namespace, with dimensions for <code>integrationArn</code>, <code>loadType</code> and <code>tableName</code>. The metrics are <code>InsertCount</code>, <code>UpdateCount</code>, <code>DeleteCount</code>, <code>IngestionSucceeded</code>, <code>IngestionFailed</code> and <code>LastSyncTimestamp</code>.</p>



<p class="wp-block-paragraph">The obvious alarm is on failure:</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-alarm 
  --alarm-name zendesk-zetl-ingestion-failed 
  --namespace AWS/Glue/ZeroETL 
  --metric-name IngestionFailed 
  --dimensions Name=integrationArn,Value=&lt;your-integration-arn&gt; 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 1 
  --threshold 1 
  --comparison-operator GreaterThanOrEqualToThreshold 
  --treat-missing-data notBreaching</code></pre>



<p class="wp-block-paragraph">That one catches loud failures. It does not catch the quiet one, which is an integration that stops running entirely. For that, alarm on <code>IngestionSucceeded</code> with <code>--treat-missing-data breaching</code> over a window longer than your refresh interval. Absence of success is the signal, not presence of failure.</p>



<p class="wp-block-paragraph">And here is the one specific to this post. Because <code>DeleteCount</code> carries a <code>tableName</code> dimension, you can see the delete gap directly in a graph. Plot <code>DeleteCount</code> per table. Tickets, users and organizations will show a nonzero line. Satisfaction ratings, articles, calls and legs will sit flat at zero forever. That flat line is not a broken metric, it is the behaviour, and having it on a dashboard is the cheapest way to keep the whole team aware of it.</p>



<p class="wp-block-paragraph">Glue also writes a system table into the target database recording the outcome of each full load and CDC run, with per-run record, insert and delete counts. Query that when CloudWatch is not granular enough. If you already run Grafana Cloud or Datadog, pulling the <code>AWS/Glue/ZeroETL</code> namespace in alongside your existing infrastructure dashboards puts pipeline health next to everything else on call already watches.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<p class="wp-block-paragraph">Integrations move through Creating, Active, Modifying, Syncing, Needs attention, Failed and Deleting. What each one means in practice:</p>



<ul class="wp-block-list">
<li><strong>Syncing</strong> means it hit a recoverable error and is re-seeding data. Not an emergency. Let it finish before you touch anything.</li>



<li><strong>Needs attention</strong> means you have to fix something, usually connection credentials, the source role, the target role, or the catalog resource policy. Once you fix it, there is no manual recovery action. Glue retries automatically on an exponential backoff schedule, so the gap between attempts grows over time. If you fixed a policy and nothing happened after two minutes, that is expected. Wait it out.</li>



<li><strong>Failed</strong> is terminal. Delete the integration and recreate it. There is no repair path.</li>
</ul>



<p class="wp-block-paragraph">For anything else, the CloudWatch logs emitted after each full load or CDC run carry the actual root cause. The console status is a summary, and a fairly lossy one. Read the logs first.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Building CSAT, article count or call volume metrics on raw row counts from the four entities that never delete.</li>



<li>Creating the connection and assuming the integration can use it. The source role attachment is a separate CLI step.</li>



<li>Choosing Redshift as the direct target during a proof of concept and locking a refresh interval you picked arbitrarily.</li>



<li>Setting a 15 minute interval because it was available, then being surprised by small-file churn and CloudWatch log volume.</li>



<li>Writing downstream queries that address Zendesk custom fields as nested columns. They arrive as strings.</li>



<li>Alarming only on <code>IngestionFailed</code>, so a stalled integration produces silence and silence looks like health.</li>



<li>Reaching for zero-ETL when the requirement is ticket comments or ticket metrics, which are not in the entity set at all.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Split delete-tracking and append-only entities into separate integrations so you can rebuild one without disturbing the other.</li>



<li>Land in the SageMaker lakehouse rather than straight into Redshift unless you are certain about freshness. It keeps the refresh interval editable.</li>



<li>Put <code>DeleteCount</code> per table on a dashboard. It makes an invisible behaviour visible to everyone, not just whoever read the docs.</li>



<li>Define the integration in CloudFormation or CDK once it works, so the next environment is a deploy rather than a repeat of the console clicking.</li>



<li>Keep a silver layer between the replicated tables and your BI tool. That is where you parse the stringified structs and apply the reconciliation logic, and it means schema surprises break one view instead of every dashboard.</li>



<li>Document which entities do not track deletes somewhere your analysts will actually read, next to the tables, not in a wiki page nobody opens.</li>
</ul>



<h3 class="wp-block-heading">When to use something else</h3>



<p class="wp-block-paragraph">Zero-ETL is a good fit when your target is already AWS, your entities are in the supported set, and you want AWS to own the pipeline. It is the wrong fit if you need entities outside that list, need transformation at ingest, or need the same Zendesk data landing somewhere that is not AWS. Managed connector platforms like Fivetran or Airbyte cover a wider slice of the Zendesk API and will replicate to non-AWS destinations, at the cost of another vendor relationship, another bill, and data leaving your account boundary on the way through. If the seven entities cover your reporting, zero-ETL is cheaper and simpler. If they do not, forcing it is worse than paying for a tool that fits.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Which Zendesk entities does AWS Glue zero-ETL support?</h3>



<p class="wp-block-paragraph">Tickets, users, organizations, satisfaction ratings, articles, calls and call legs. Tickets, users and organizations replicate creates, updates and deletes. The other four replicate creates and updates only.</p>



<h3 class="wp-block-heading">Can I get ticket comments or ticket metrics through zero-ETL?</h3>



<p class="wp-block-paragraph">No. They are not in the entity set. Use a Glue ETL job against the same Zendesk connection, or a connector platform with broader API coverage, and join the result to your replicated tables downstream.</p>



<h3 class="wp-block-heading">How fresh can the data be?</h3>



<p class="wp-block-paragraph">The refresh interval goes down to 15 minutes and up to six days. This is near real time replication, not streaming. If you need sub-minute latency on Zendesk events, zero-ETL is the wrong mechanism and you want webhooks into an event pipeline instead.</p>



<h3 class="wp-block-heading">Can I change the refresh interval later?</h3>



<p class="wp-block-paragraph">Only if the target is not Redshift. With Redshift as the target the interval is fixed at creation. With the SageMaker lakehouse and other targets you can modify it afterwards.</p>



<h3 class="wp-block-heading">Why is my integration stuck in NEEDS_ATTENTION after I fixed the permissions?</h3>



<p class="wp-block-paragraph">Because recovery is automatic but not immediate. Glue retries on exponential backoff, so if the integration has been unhealthy for a while the next retry may be some way off. There is no manual recovery command. Confirm the fix is correct, then wait.</p>



<h3 class="wp-block-heading">Does zero-ETL cost extra on top of Glue?</h3>



<p class="wp-block-paragraph">AWS does not bill a separate charge for the integration itself. You pay for the surrounding services: target storage, Redshift or Athena query compute, S3 requests, Data Catalog usage and CloudWatch logs. Refresh interval is the main lever, since it drives how many small commits and log entries you generate.</p>



<h3 class="wp-block-heading">Why do my Zendesk custom fields look like text?</h3>



<p class="wp-block-paragraph">The connector converts struct and list types to strings. Custom fields come through as serialized values rather than nested columns, so parse them in a downstream layer instead of querying them directly.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">Zendesk AWS Glue zero-ETL removes a genuinely tedious pipeline from your plate. The setup is fiddly in the IAM layer and then it mostly runs itself, which is exactly the point of a managed integration.</p>



<p class="wp-block-paragraph">The one thing worth carrying out of this post: four of the seven supported entities never replicate a delete. Satisfaction ratings, articles, calls and call legs only grow. Nothing warns you, every metric stays green, and your numbers drift a little further from Zendesk with every cycle. Design for that on day one, put <code>DeleteCount</code> per table on a dashboard, and you will not be the person explaining the discrepancy in a meeting six months from now.</p>



<h2 class="wp-block-heading">Need help with your Zendesk to AWS data pipeline?</h2>



<p class="wp-block-paragraph">I work with teams building and fixing replication pipelines on AWS. On this kind of setup, that usually means:</p>



<ul class="wp-block-list">
<li>Auditing an existing Zendesk zero-ETL integration and quantifying how far the target has drifted from the source</li>



<li>Designing the split between continuously synced and append-only entities, including the rebuild cadence and reconciliation queries</li>



<li>Untangling the source role, target role and catalog policy layers when an integration sits in <code>NEEDS_ATTENTION</code> with no clear cause</li>



<li>Building the silver layer that parses stringified Zendesk custom fields into something your BI tool can actually use</li>



<li>Setting up CloudWatch alarms and dashboards that catch a stalled integration, not just a failed one</li>



<li>Moving a console-built integration into CloudFormation or CDK so it can be deployed across accounts</li>
</ul>



<p class="wp-block-paragraph">Send me the integration status, a CloudWatch log excerpt, or the query that is giving you the wrong number, and I will tell you what I think is going on.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/zendesk-aws-glue-zero-etl/">Zendesk Data Integration with AWS Glue Zero-ETL: The Delete Gap That Skews Your Numbers</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>How to Connect BI Tools to AWS Data Without a Runaway Bill</title>
		<link>https://john-nessime.com/blog/technical-guides/connect-bi-tools-to-aws-data/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Fri, 14 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Business Intelligence]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[Identity Propagation]]></category>
		<category><![CDATA[JDBC]]></category>
		<category><![CDATA[Lake Formation]]></category>
		<category><![CDATA[Power BI]]></category>
		<category><![CDATA[PrivateLink]]></category>
		<category><![CDATA[Query Result Reuse]]></category>
		<category><![CDATA[Row-Level Security]]></category>
		<category><![CDATA[Tableau]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=231</guid>

					<description><![CDATA[<p>Getting Tableau or Power BI to return rows from your data lake takes an hour. The problems arrive three weeks later. Here are the four access patterns worth considering, why fine-grained security and cheap dashboards pull against each other, and the cost levers that actually move the invoice.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/connect-bi-tools-to-aws-data/">How to Connect BI Tools to AWS Data Without a Runaway Bill</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The request usually arrives with a deadline already attached. Finance bought Tableau licences, the growth team wants Power BI pointed at the clickstream data, and someone in a meeting said the data is &#8220;already in S3, so it should be quick.&#8221;</p>



<p class="wp-block-paragraph">It is quick. You can have a working connection inside an hour. What isn&#8217;t quick is everything that surfaces three weeks later, when forty people are refreshing dashboards on a schedule nobody on your team controls and the Athena line on the invoice has quietly tripled.</p>



<p class="wp-block-paragraph">Getting a BI tool to authenticate against AWS and return rows is a solved problem with decent vendor documentation. What is not well covered is what happens afterwards. This post covers how to connect BI tools to AWS data in a way that survives contact with real users: the four access patterns worth considering, why fine-grained security and cheap dashboards pull in opposite directions, how identity actually flows through a shared service account, and the cost levers in the order they pay off.</p>



<h2 class="wp-block-heading">The failure that shows up on the invoice, not in the logs</h2>



<p class="wp-block-paragraph">Start with the shape of the problem, because it explains most of the design decisions that follow.</p>



<p class="wp-block-paragraph">A dashboard is not one query. Every tile is a query. A twelve-tile dashboard opened by forty analysts, with an hourly scheduled refresh on top, is a query generator running against your data lake. Athena bills on bytes scanned, so the cost of that pattern is set almost entirely by how much data each tile touches, not by how &#8220;small&#8221; the answer looks in the browser.</p>



<p class="wp-block-paragraph">Athena has a feature built for exactly this. Query result reuse lets a repeated query return the previous stored result instead of scanning again, within a maximum age you choose. If you don&#8217;t specify an age, Athena treats results older than 60 minutes as expired, and the longest age you can specify is the equivalent of seven days.</p>



<p class="wp-block-paragraph">Here is the part that catches people, and it is the whole reason I wrote this post.</p>



<p class="wp-block-paragraph">Result reuse is not available for tables that have Lake Formation row or column filters. It is also unavailable when the table&#8217;s S3 location is registered as a Lake Formation data location, for queries that reference more than one data catalog, for queries referencing more than twenty tables, and for federated catalogs or an external Hive metastore.</p>



<p class="wp-block-paragraph">So the sequence goes like this. Dashboards are connected and cheap. A security review asks for row-level security on the customer dimension. You add Lake Formation data filters, which is exactly the right call. Everyone signs off. And the reuse rate on those tables silently drops to zero. No error. No alert. No code change. Just a larger bill the following month and a very awkward conversation about what changed.</p>



<p class="wp-block-paragraph">You can still have both governance and cheap dashboards. You just can&#8217;t have both on the same table. The usual escape is to materialise pre-filtered, per-audience tables that carry no filters of their own, and grant access to those instead. That costs you a pipeline and some storage. It buys back the caching.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Pick the door before you pick the driver</h2>



<p class="wp-block-paragraph">Most guides start with driver installation. That&#8217;s backwards. The driver is a twenty-minute job once you&#8217;ve decided which door the BI tool walks through. There are four realistic doors, and they fail in different ways.</p>



<h3 class="wp-block-heading">Athena over the AWS Glue Data Catalog</h3>



<p class="wp-block-paragraph">The default, and usually the right first answer. Files stay where they are, the Glue Data Catalog is the single source of truth for schema, and anything that speaks JDBC or ODBC can connect. Tableau, Power BI, Looker, Metabase, Apache Superset and Grafana all handle this without exotic configuration.</p>



<p class="wp-block-paragraph">Where it wins: no cluster to size, no idle spend, and you can expose a new dataset by cataloguing it rather than by loading it anywhere.</p>



<p class="wp-block-paragraph">Where it doesn&#8217;t: per-query billing punishes chatty dashboards, and interactive latency is measured in seconds rather than milliseconds. If your users expect a filter dropdown to respond instantly, they will be disappointed and they will tell you so.</p>



<p class="wp-block-paragraph">One practical note that saves an afternoon: for Tableau Desktop, the Athena JDBC driver has to be dropped in a specific directory before the connector appears. On macOS that&#8217;s <code>~/Library/Tableau/Drivers</code>, and on Windows it&#8217;s <code>C:Program FilesTableauDrivers</code>. If the analyst says &#8220;I don&#8217;t see Amazon Athena in the list,&#8221; that&#8217;s almost always why.</p>



<h3 class="wp-block-heading">A Redshift datashare into a consumer warehouse</h3>



<p class="wp-block-paragraph">If the data already lives in Redshift, or if concurrency is the binding constraint, a datashare is the cleaner answer. The producer keeps ownership; the consumer gets live, transactionally consistent access and pays with its own compute, so a badly written dashboard can&#8217;t degrade your ETL warehouse.</p>



<pre class="wp-block-code"><code>-- On the producer
CREATE DATASHARE bi_share;
ALTER DATASHARE bi_share ADD SCHEMA reporting;
ALTER DATASHARE bi_share ADD TABLE reporting.fact_orders;
GRANT USAGE ON DATASHARE bi_share TO ACCOUNT '111122223333';</code></pre>



<p class="wp-block-paragraph">The schema has to be added before the tables inside it, which is a common first stumble. Cross-account sharing is a two-way handshake: the producer authorises the datashare, and an administrator in the consumer account associates it before anything is queryable. Both sides must be encrypted for cross-account sharing. Objects are read-only unless the producer explicitly grants write privileges and the consumer associates for writes, which is the behaviour you want for a BI audience.</p>



<p class="wp-block-paragraph">Two things to price in. Cross-region sharing moves data and is billed accordingly. And the consumer warehouse doesn&#8217;t hold statistics on the producer&#8217;s data, so plan expectations around query planning rather than assuming parity with local tables.</p>



<h3 class="wp-block-heading">The Iceberg REST catalog</h3>



<p class="wp-block-paragraph">If the consumer is another query engine rather than a dashboard tool, this is the interesting door. Amazon S3 Tables exposes an Iceberg REST endpoint at <code>https://s3tables.&lt;region&gt;.amazonaws.com/iceberg</code>, with your table bucket ARN as the warehouse location and SigV4 for authentication. Engines like Snowflake, Trino, StarRocks, Spark and PyIceberg can register that as an external catalog and read the tables in place.</p>



<p class="wp-block-paragraph">Worth knowing: AWS positions the S3 Tables endpoint for basic read and write access against a single table bucket, and recommends the AWS Glue Iceberg REST endpoint when you need unified table management, centralised governance and fine-grained access control. If Lake Formation is already your control plane, go through Glue.</p>



<p class="wp-block-paragraph">Where it doesn&#8217;t help: most BI tools don&#8217;t speak Iceberg REST. They speak SQL over JDBC. This is an engine-to-engine door, not a dashboard door, and treating it as the latter wastes a week.</p>



<h3 class="wp-block-heading">A materialised extract</h3>



<p class="wp-block-paragraph">The unglamorous option that is frequently correct. Build a narrow, pre-aggregated table on a schedule and land it where the BI tool already keeps data: QuickSight SPICE, a Power BI import model, a Tableau extract, or a small Postgres instance. Tools like dbt handle the modelling side; Fivetran or Airbyte handle it if the source is a SaaS system rather than your own lake.</p>



<p class="wp-block-paragraph">Where it wins: it is by a wide margin the cheapest and fastest option for stable reporting, because the expensive scan happens once per day instead of once per tile per user. If you self-host Metabase or Superset, a modest VPS from a provider like InterServer or Hetzner will comfortably serve a department off a nightly rollup.</p>



<p class="wp-block-paragraph">Where it doesn&#8217;t: freshness. And you now own a pipeline that can fail silently, which is its own category of on-call pain.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Identity: the shared service role problem</h2>



<p class="wp-block-paragraph">Nearly every BI deployment starts with one IAM role for the whole tool. It works immediately, which is why it survives longer than it should.</p>



<p class="wp-block-paragraph">The consequences accumulate quietly. CloudTrail records every query as the same principal, so audit answers stop being useful. Lake Formation row filters key off the principal, so if all traffic arrives as one role, your row-level security collapses to whatever that role can see. And cost attribution disappears, so when the bill jumps you cannot say which team caused it.</p>



<p class="wp-block-paragraph">Three ways out, in increasing order of effort:</p>



<ol class="wp-block-list">
<li><strong>One role and one Athena workgroup per consumer group.</strong> Coarse, but honest. Finance gets a role, marketing gets a role, each with its own workgroup. Scan volume becomes attributable and you can set different limits per audience.</li>

<li><strong>Real identity propagation.</strong> Federate the BI tool&#8217;s users through IAM Identity Center or SAML so Lake Formation grants apply to the actual person. This is the correct boundary and the most work, and driver and tool support varies. Test it with the exact tool version your users have before you promise it in a design review.</li>

<li><strong>Push the filter into the data.</strong> Per-tenant tables, views or partitions with separate grants. Unfashionable and more to maintain, but it works with every tool, survives driver upgrades, and keeps result reuse eligible because the tables themselves carry no filters.</li>
</ol>



<p class="wp-block-paragraph">Option three is the one I reach for first when the audience is a handful of stable groups, and option two when the audience is the whole company.</p>



<h2 class="wp-block-heading">The network path is a design decision, not a checkbox</h2>



<p class="wp-block-paragraph">For a SaaS BI tool, the public service endpoint with SigV4-signed requests over TLS is perfectly reasonable, and it is what most vendors document. Don&#8217;t let anyone talk you into complexity you don&#8217;t need.</p>



<p class="wp-block-paragraph">You need more when the tool runs inside a VPC, or on-premises via Direct Connect or VPN, and traffic must not traverse the internet. Athena supports an interface VPC endpoint, and the Athena documentation is explicit that you should pair it with an AWS Glue endpoint, because the catalog lookup is a separate API call that will otherwise still go out to the public endpoint.</p>



<pre class="wp-block-code"><code>aws ec2 create-vpc-endpoint 
  --vpc-id &lt;your-vpc-id&gt; 
  --service-name com.amazonaws.&lt;region&gt;.athena 
  --vpc-endpoint-type Interface 
  --subnet-ids &lt;subnet-id&gt; 
  --security-group-ids &lt;security-group-id&gt; 
  --private-dns-enabled</code></pre>



<p class="wp-block-paragraph">With private DNS enabled, the standard <code>athena.&lt;region&gt;.amazonaws.com</code> hostname resolves to the endpoint, so no client configuration changes. Without it, VPC gives you a hostname in the form <code>&lt;endpoint-id&gt;.athena.&lt;region&gt;.vpce.amazonaws.com</code>, which you have to configure in the tool by hand. Attach an endpoint policy while you&#8217;re there; that&#8217;s where you restrict which principals and resources can be reached.</p>



<p class="wp-block-paragraph">For Redshift, the equivalent is a Redshift-managed VPC endpoint, which requires either RA3 node types or Redshift Serverless. On provisioned clusters, cluster relocation has to be enabled first.</p>



<p class="wp-block-paragraph">One cost item people forget: if the BI tool lives outside AWS, every row it pulls back is data transfer out. A live dashboard returning aggregates is cheap. A nightly extract refresh dragging millions of rows across the boundary is not. Add an S3 gateway endpoint for the query results bucket so the driver&#8217;s result download stays on the AWS network where it can.</p>



<h2 class="wp-block-heading">Cost levers, in the order they pay off</h2>



<p class="wp-block-paragraph">Ordering matters here. Most teams reach for the guardrails first, which bounds the worst case without improving the base case.</p>



<ol class="wp-block-list">
<li><strong>Fix the storage layout.</strong> Columnar formats and sensible partitioning reduce bytes scanned on every query forever. Nothing else you do competes with this.</li>

<li><strong>Pre-aggregate for the dashboard.</strong> A tile that reads a purpose-built rollup is a different animal from one that scans a fact table and aggregates on the fly.</li>

<li><strong>Enable result reuse where governance allows it.</strong> Free wins on repeated queries, provided the tables aren&#8217;t disqualified by the limitations above.</li>

<li><strong>Give every consumer its own workgroup with a per-query cutoff.</strong> This is the guardrail, and it belongs after the first three, not before them.</li>

<li><strong>Move the highest-traffic dashboards to extract mode.</strong> Once a dashboard&#8217;s usage is predictable, importing beats querying live.</li>
</ol>



<p class="wp-block-paragraph">The workgroup guardrail in Terraform. The per-query cutoff cancels any single query that scans more than the limit, which is what stops an accidental <code>SELECT *</code> against a multi-terabyte table:</p>



<pre class="wp-block-code"><code>resource "aws_athena_workgroup" "bi_finance" {
  name  = "bi-finance"
  state = "ENABLED"

  configuration {
    enforce_workgroup_configuration    = true
    publish_cloudwatch_metrics_enabled = true
    bytes_scanned_cutoff_per_query     = 10737418240  # 10 GiB

    result_configuration {
      output_location = "s3://my-athena-results/bi-finance/"
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Two flags there do real work. <code>enforce_workgroup_configuration</code> stops clients overriding the results location and encryption settings, which is what keeps a BI tool from writing results somewhere unmanaged. <code>publish_cloudwatch_metrics_enabled</code> is what makes the whole thing observable, and without it you&#8217;re back to reading the invoice to find out what happened.</p>



<p class="wp-block-paragraph">Workgroup-wide limits are configured separately from the workgroup resource. Those let you set hourly or daily thresholds on aggregate data scanned and fire an SNS notification when a threshold is crossed, which can page someone or disable the workgroup outright.</p>



<p class="wp-block-paragraph">Result reuse can be set per query through the API, which is the mechanism the drivers expose as a connection option:</p>



<pre class="wp-block-code"><code>aws athena start-query-execution 
  --work-group "bi-finance" 
  --query-string "SELECT region, SUM(net) FROM reporting.orders GROUP BY 1" 
  --result-reuse-configuration 
    "ResultReuseByAgeConfiguration={Enabled=true,MaxAgeInMinutes=60}"</code></pre>



<p class="wp-block-paragraph">Set the max age to match how often the underlying data actually changes. If the table is rebuilt once a night, an hour is far too conservative.</p>



<p class="wp-block-paragraph">Finally, alarm on the scan metrics. Whether that lands in CloudWatch dashboards, Grafana or Datadog matters less than the alarm existing at all, owned by the team whose workgroup it is.</p>



<h2 class="wp-block-heading">Troubleshooting the connections that half work</h2>



<ul class="wp-block-list">
<li><strong>The tool connects but lists no tables.</strong> Almost always Lake Formation rather than IAM. Check that the role has been granted on the database and tables, and note that the <code>DESCRIBE</code> permission on the default database is required for Athena.</li>

<li><strong>&#8220;Bytes scanned limit exceeded.&#8221;</strong> That&#8217;s your per-query cutoff doing its job. Look at what the tile is actually querying before raising the limit. Usually the partition filter is missing.</li>

<li><strong>Query works in the console, fails from the tool.</strong> Three usual suspects: a different workgroup, a results location the role can&#8217;t write to, or a role that has catalog access but not S3 access to the underlying data.</li>

<li><strong>A view works for you but not for the analyst.</strong> Lake Formation requires permissions on the tables, columns and S3 locations the view is built on. Column-level permissions are not available on views, so a view is not a substitute for column filtering.</li>

<li><strong>Costs jumped with no deployment.</strong> Check whether anyone added Lake Formation filters, and check whether partition pruning quietly stopped working after a schema or layout change.</li>

<li><strong>The first query of the morning is slow, the rest are fine.</strong> Partition metadata listing. Partition projection removes most of that cost on large tables.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Granting the BI role broad S3 read access and calling it least privilege. It sees every bucket prefix it can list, catalog or not.</li>

<li>Running every consumer through the default primary workgroup, which makes attribution and per-team limits impossible.</li>

<li>Exposing raw fact tables to business users and expecting them to write efficient SQL. They will drag columns into a canvas, which is what the tool is for.</li>

<li>Setting dashboard refresh schedules without asking how fresh the underlying data actually is. Refreshing hourly against a nightly-loaded table burns money for identical results.</li>

<li>Promising row-level security through a shared service role. It will not behave the way the security review thinks it does.</li>

<li>Skipping the Glue VPC endpoint after building the Athena one, then spending a day wondering why traffic still leaves the VPC.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>Create the workgroup, the role and the tag before you hand over the connection string, not after the first invoice.</li>

<li>Publish a curated reporting layer. The BI tool should see purpose-built tables with business column names, not your raw landing zone.</li>

<li>Decide consciously whether a dataset is governed by Lake Formation filters or optimised for caching, and document which, because the two do not coexist on one table.</li>

<li>Apply cost allocation tags at creation time. Retrofitting them across a live estate is thankless work.</li>

<li>Version the connection setup as code. Terraform for the workgroup, roles and endpoints; a short runbook for the driver install on the client side.</li>

<li>Re-test the whole path after any driver upgrade. Authentication behaviour is the thing that changes between driver major versions.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">What is the fastest way to connect Power BI or Tableau to AWS data?</h3>



<p class="wp-block-paragraph">Athena over the Glue Data Catalog, using the Athena JDBC or ODBC driver. Catalogue the data, create a workgroup with a results bucket, create a role with catalog and S3 read access, install the driver on the client, connect. The connection itself is the easy part; the workgroup and role are what you&#8217;ll wish you had done deliberately.</p>



<h3 class="wp-block-heading">Should I use Athena or Redshift for BI dashboards?</h3>



<p class="wp-block-paragraph">Athena if query volume is moderate, the data already lives in S3, and users can tolerate a few seconds per query. Redshift if you need high concurrency, consistently low latency, or you already run a warehouse and can share into a separate consumer endpoint. The honest test is whether your peak is a spike or a plateau: Athena handles spikes without idle cost, Redshift handles plateaus without per-query cost.</p>



<h3 class="wp-block-heading">Does Lake Formation row-level security work with any BI tool?</h3>



<p class="wp-block-paragraph">The filtering itself is enforced by the query engine, so it works regardless of the tool. What varies is whether the tool can present each end user as a distinct principal. Through a single shared service role, every user gets the same filtered view, which is not what row-level security is for. Verify the identity path with your actual tool before designing around it.</p>



<h3 class="wp-block-heading">Why did my Athena bill increase after enabling fine-grained access control?</h3>



<p class="wp-block-paragraph">Because query result reuse is not supported for tables with Lake Formation row or column filters, or where the S3 location is registered as a Lake Formation data location. Queries that were being served from cached results started scanning again. Nothing errors, which is why it takes a billing cycle to notice.</p>



<h3 class="wp-block-heading">Do I need PrivateLink to connect a BI tool to AWS?</h3>



<p class="wp-block-paragraph">No, unless your requirements say traffic must not traverse the public internet, or the tool runs inside a VPC or on-premises network. Public endpoints with signed requests over TLS are the normal path for SaaS BI tools. Add interface endpoints when there&#8217;s a stated requirement, and remember to add the Glue endpoint alongside the Athena one.</p>



<h3 class="wp-block-heading">Can BI tools read Amazon S3 Tables directly?</h3>



<p class="wp-block-paragraph">Indirectly. S3 Tables integrates with the Glue Data Catalog, so Athena, EMR and Redshift can query them, and any BI tool that connects through Athena inherits that. The Iceberg REST endpoint is aimed at query engines and Iceberg clients rather than dashboard tools, which generally expect a SQL connection instead.</p>



<h3 class="wp-block-heading">Is QuickSight a better fit than an external BI tool?</h3>



<p class="wp-block-paragraph">It removes the network, driver and identity plumbing described above, and SPICE gives you an in-memory cache without building an extract pipeline. Against that, your organisation may already be committed to Tableau, Power BI or Looker for reasons that have nothing to do with infrastructure, and retraining an analytics team is a genuine cost. Evaluate it on that basis rather than on the connection mechanics alone.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">When you connect BI tools to AWS data, the connection is not the project. The project is deciding, deliberately, which datasets are governed by fine-grained filters and which are optimised for repeated cheap reads, then giving every consumer its own role, workgroup and cost tag before anyone gets a connection string.</p>



<p class="wp-block-paragraph">Do that and a surprise bill becomes a question with an answer. Skip it and you&#8217;ll be reading CloudTrail on a Friday afternoon trying to work out which dashboard scanned four terabytes, with no way to prove it.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help wiring this up properly?</h2>



<p class="wp-block-paragraph">This is a large part of what I do. If you&#8217;re opening up AWS data to an analytics team and want it done so it still makes sense in six months, I can help with:</p>



<ul class="wp-block-list">
<li>Choosing between Athena, a Redshift datashare, an Iceberg REST catalog or a scheduled extract for your specific query pattern and user count</li>

<li>Setting up per-consumer IAM roles, Athena workgroups, scan limits and cost allocation tags as Terraform rather than console clicks</li>

<li>Designing a Lake Formation permission model that gives you row and column security without silently destroying your caching</li>

<li>Building the curated reporting layer and partition strategy that makes dashboards fast and cheap at the source</li>

<li>Private network paths with interface VPC endpoints, Redshift-managed endpoints and endpoint policies for tools that can&#8217;t use public endpoints</li>

<li>Diagnosing an Athena or Redshift bill that grew without an obvious cause, and putting alarms in place so it doesn&#8217;t happen twice</li>
</ul>



<p class="wp-block-paragraph">Send me a workgroup config, a slow dashboard query, or a Cost Explorer screenshot you can&#8217;t explain, and I&#8217;ll tell you what I&#8217;d look at first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/connect-bi-tools-to-aws-data/">How to Connect BI Tools to AWS Data Without a Runaway Bill</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Data Lake vs Data Warehouse for CRM Analytics: Volume Is the Wrong Question</title>
		<link>https://john-nessime.com/blog/devops/data-lake-vs-data-warehouse-crm-analytics/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 06 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Business Intelligence]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[CRM Analytics]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Warehouse]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[SQL]]></category>
		<category><![CDATA[Zero-ETL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=153</guid>

					<description><![CDATA[<p>Everyone argues this one on data volume, and volume is the argument that matters least: CRM data is small enough that both architectures handle it comfortably. What actually decides data lake vs data warehouse for CRM analytics is how much point-in-time history you need, how fast the schema churns, what shape your queries are, and who is going to maintain the thing. Includes a decision procedure you can run in an afternoon.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/data-lake-vs-data-warehouse-crm-analytics/">Data Lake vs Data Warehouse for CRM Analytics: Volume Is the Wrong Question</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question never arrives as an architecture question. It arrives as someone in RevOps asking whether they can get pipeline coverage by rep, by week, for the last two years, including the deals that were deleted along the way.</p>



<p class="wp-block-paragraph">You look at the CRM reporting tab. It cannot do that. So now you are picking a platform, and within about ten minutes someone will say &#8220;we should build a data lake,&#8221; and someone else will say &#8220;just put it in the warehouse,&#8221; and the deciding argument will be about how much data there is.</p>



<p class="wp-block-paragraph">That argument is almost always the wrong one. This post covers data lake vs data warehouse specifically for CRM analytics: why volume rarely decides it, what actually does, a decision procedure you can run in an afternoon, and where zero-ETL has changed the maths recently enough that older advice is stale.</p>



<h2 class="wp-block-heading">Start with the number, then stop using it</h2>



<p class="wp-block-paragraph">Go and count your rows. Not estimate. Count.</p>



<p class="wp-block-paragraph">A mid-sized company&#8217;s entire Salesforce org is usually a few million rows across the objects anyone actually reports on: accounts, contacts, opportunities, activities, cases. Add HubSpot, Zendesk and a billing system and you are still typically in the low tens of gigabytes once it is written as compressed Parquet. Activity and event history is the one object that can genuinely run large, and it is worth checking separately.</p>



<p class="wp-block-paragraph">Numbers at that scale do not stress either architecture. A data warehouse will not break a sweat. A lake will not break a sweat. The lake-versus-warehouse debate as it is usually framed, the one about petabytes and elasticity and separating storage from compute, is a debate about a scale problem you do not have.</p>



<p class="wp-block-paragraph">So count the rows, confirm you are small, and then stop using volume as an argument. Everything below is about what decides it instead.</p>



<h2 class="wp-block-heading">Data lake vs data warehouse: the three options, honestly</h2>



<h3 class="wp-block-heading">Warehouse only</h3>



<p class="wp-block-paragraph">Load CRM objects straight into Redshift, Snowflake, BigQuery or Postgres. Model with SQL. Point the BI tool at it.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> it is the fastest path to an analyst being productive, and that is not a small thing. Query performance on joins is better than a lake gives you without tuning. Access control is mature and row-level. Every BI tool connects natively. The people who will maintain this already know SQL and probably do not know Spark.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> semi-structured data is awkward, and CRM payloads are full of it. Custom field bags, JSON blobs, email bodies, call transcripts. You can store JSON in a modern warehouse but you will not enjoy querying it at any depth. Storing many years of daily snapshots gets expensive on warehouse storage in a way it does not on object storage. And you are more locked in: moving between warehouses is a migration project, not a repoint.</p>



<h3 class="wp-block-heading">Lake only</h3>



<p class="wp-block-paragraph">Land raw API responses to S3, catalogue them, query with Athena or Spark. Iceberg or Delta as the table format if you want anything resembling a database.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> raw retention is nearly free, so keeping every version of every record for years is a storage line item rather than a decision. Schema evolution is genuinely easier: when a SaaS admin adds a field, the lake absorbs it. Unstructured content lives alongside structured tables, which matters more now that people want to run models over call notes and support tickets. And an open table format means the data is not hostage to one query engine.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> you have taken on work a warehouse does for you. Small-file compaction, partitioning, table maintenance, catalogue hygiene. Query latency on interactive dashboards is worse unless you invest in layout. Governance is a build, not a setting. And the per-query cost model punishes exactly the workload CRM analytics generates: lots of small, repeated, unpredictable queries from a BI tool. Athena bills on data scanned with a per-query minimum, which means a dashboard refreshing thirty tiles is thirty billable scans whether or not anyone is looking.</p>



<h3 class="wp-block-heading">Lakehouse</h3>



<p class="wp-block-paragraph">Raw and historical data in object storage as Iceberg tables; curated marts in the warehouse or queried in place by a warehouse engine. This is where most serious setups end up, and it is what AWS, Databricks and Snowflake have all converged on.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> you get cheap deep history and expensive fast queries in the right places. Raw payloads stay replayable when a vendor changes a field type. Analysts still get warehouse-shaped tables.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> it is two systems. Two cost models, two sets of permissions, two places a table can be defined and drift. For a five-person company with one CRM, this is architecture as procrastination. The complexity is real and it should be earning something.</p>



<h2 class="wp-block-heading">What actually decides it</h2>



<p class="wp-block-paragraph">Four levers, in the order they usually matter for CRM data.</p>



<h3 class="wp-block-heading">1. How much history you need, and whether deletes matter</h3>



<p class="wp-block-paragraph">This is the big one and it is the one people skip.</p>



<p class="wp-block-paragraph">CRM systems mutate in place. An opportunity&#8217;s stage, amount and close date are overwritten as the deal moves. If you only ever sync current state, you can report on the pipeline as it is today and you cannot report on what it looked like last March. Every question about pipeline movement, forecast accuracy, stage velocity or &#8220;what changed since the QBR&#8221; needs point-in-time history, and the CRM will not give it to you retroactively. Deleted records are worse: they are simply gone.</p>



<p class="wp-block-paragraph">If you need history, you need somewhere cheap to keep daily snapshots or a change feed. That pushes toward object storage, or toward a warehouse with a genuinely cheap storage tier. If a rolling twelve months of current-state data covers every question anyone asks, the warehouse handles it alone.</p>



<h3 class="wp-block-heading">2. How fast the schema churns</h3>



<p class="wp-block-paragraph">Count the custom fields added to your main CRM object in the last six months. If it is two, schema churn is not your problem. If it is forty, and half of them are from a business unit you do not talk to, a rigid warehouse schema becomes a standing maintenance tax and the lake&#8217;s tolerance for drift starts paying for itself.</p>



<h3 class="wp-block-heading">3. The shape of the queries</h3>



<p class="wp-block-paragraph">Scheduled dashboards hitting the same tables repeatedly favour a warehouse, because per-second compute amortises across many queries and results are cached. Redshift Serverless, for example, bills in processing-unit-hours per second with a short minimum and shuts down when idle, which suits bursty BI traffic.</p>



<p class="wp-block-paragraph">Exploratory one-off analysis across wide, rarely-touched history favours a lake, because you only pay when someone asks. Most CRM analytics is overwhelmingly the first kind. That fact alone settles a lot of these arguments.</p>



<h3 class="wp-block-heading">4. Who is going to maintain it</h3>



<p class="wp-block-paragraph">The least technical criterion and often the most decisive. If the answer is &#8220;a RevOps analyst who is very good at SQL and has no platform team behind them,&#8221; build the warehouse. A lake handed to someone who cannot maintain it becomes an unqueryable pile of Parquet in about eight months. That is not a hypothetical failure mode; it is the normal one.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">A decision procedure you can run this afternoon</h2>



<ol class="wp-block-list">
<li><strong>Count rows per object</strong> for everything anyone reports on. Note which objects are over about ten million rows. Usually it is activities and nothing else.</li>

<li><strong>Write down the ten questions</strong> the business actually asks. Not the ones you imagine. Mark each one as needing current state or point-in-time history.</li>

<li><strong>Count schema changes</strong> on your main objects over the last six months.</li>

<li><strong>Estimate query volume:</strong> number of dashboards, tiles per dashboard, refresh frequency, plus rough ad-hoc queries per week. This is the number that drives cost, not storage.</li>

<li><strong>Model both costs against those numbers.</strong> For the lake: bytes scanned per query times queries per month. For the warehouse: compute-hours per month plus storage. Do it on a napkin. At CRM scale the answer is usually obvious and usually favours the warehouse.</li>

<li><strong>Name the maintainer.</strong> If you cannot name a person, pick the option that needs less of them.</li>
</ol>



<p class="wp-block-paragraph">Read together, most CRM-only shops land on: warehouse for the curated layer, cheap object storage for raw history, and no ambition beyond that until something forces it. That is a lakehouse, but the small honest kind, and it is a very different commitment from standing up a full lake platform.</p>



<h2 class="wp-block-heading">Where zero-ETL changed the maths</h2>



<p class="wp-block-paragraph">A lot of writing on this topic predates a genuine shift, so it is worth stating plainly.</p>



<p class="wp-block-paragraph">Managed zero-ETL integrations now replicate CRM and support platforms into AWS targets without you building a pipeline. AWS Glue supports SaaS sources including Salesforce, SAP, ServiceNow, Zendesk and Zoho CRM, landing into Amazon Redshift, the SageMaker lakehouse, or S3 Tables with Iceberg underneath. Salesforce has pushed the other direction with zero-copy federation, letting Data Cloud read Iceberg tables in Databricks, Snowflake, BigQuery or Redshift in place, and letting those platforms read Data Cloud objects back.</p>



<p class="wp-block-paragraph">Two consequences for this decision:</p>



<ul class="wp-block-list">
<li><strong>Ingestion is no longer the differentiator.</strong> &#8220;The lake is easier to load into&#8221; was a real argument when you were hand-rolling connectors. It is much weaker when the same managed integration targets either destination.</li>

<li><strong>Iceberg makes the choice less permanent.</strong> If your raw layer is Iceberg tables in object storage, adding a warehouse engine later is a configuration change rather than a migration. That is a genuine reason to prefer open table formats even if you never build a &#8220;lake&#8221; as such.</li>
</ul>



<p class="wp-block-paragraph">Two honest caveats. Zero-ETL integrations cover a curated list of sources and regions, so check yours before designing around them. And a managed replica is still current-state replication plus change capture, not a modelled history; you still have to decide how snapshots and slowly changing dimensions get built.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<ul class="wp-block-list">
<li><strong>&#8220;We need a lake because we&#8217;ll do AI later.&#8221;</strong> Reasonable in spirit, wrong in sequence. Build the thing that answers this quarter&#8217;s questions, keep raw payloads in object storage so you have not destroyed your options, and revisit. Raw retention is the cheap insurance; a lake platform is not.</li>

<li><strong>&#8220;Storage is cheaper in a lake.&#8221;</strong> True and almost always irrelevant. At CRM volumes, storage is a rounding error next to compute and, more to the point, next to the salary of whoever maintains the thing.</li>

<li><strong>&#8220;The warehouse is a single point of vendor lock-in.&#8221;</strong> Fair, and it is why open table formats matter. But if you replicate raw data to object storage, the lock-in is on your transformation logic, not your data, and that is portable if you keep it in SQL and version control.</li>

<li><strong>&#8220;We&#8217;ll just query the CRM API directly and skip both.&#8221;</strong> Works until a dashboard fans out into rate limits and the whole thing becomes a data pipeline you did not design. If more than one person is asking questions, you need a copy.</li>

<li><strong>&#8220;Let&#8217;s do both properly from day one.&#8221;</strong> This is the expensive mistake. Two systems, two cost models, and tables that quietly disagree, in service of a workload that fits comfortably in one.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is a data lake overkill for one CRM?</h3>



<p class="wp-block-paragraph">For a single CRM with current-state reporting needs, usually yes. The point at which it stops being overkill is when you need years of point-in-time history, when unstructured content like call transcripts becomes part of the analysis, or when you are integrating enough systems that a common raw layer is cheaper than a dozen warehouse loaders.</p>



<h3 class="wp-block-heading">How do I get historical CRM snapshots if I only have current state?</h3>



<p class="wp-block-paragraph">You mostly cannot get it retroactively, which is why this decision is urgent rather than academic. Start capturing now: either daily full snapshots to partitioned object storage, or a change feed if the platform offers one, then build slowly changing dimension tables from that. Field history tracking inside the CRM covers a limited number of fields and is not a substitute.</p>



<h3 class="wp-block-heading">Does the lakehouse make this comparison obsolete?</h3>



<p class="wp-block-paragraph">It blurs the storage layer, not the operational reality. You still choose a query engine, a cost model, a governance model and a maintainer. The useful question has shifted from &#8220;lake or warehouse&#8221; to &#8220;how much of this do I want to operate myself,&#8221; which is a better question anyway.</p>



<h3 class="wp-block-heading">Where does reverse ETL fit?</h3>



<p class="wp-block-paragraph">If you plan to push computed values back into the CRM, such as a lead score or account health field, that pushes toward the warehouse. Reverse ETL tools connect to warehouses far more readily than to query engines over object storage, and the write path wants a stable, modelled table rather than a raw one.</p>



<h3 class="wp-block-heading">Can I start with one and move later?</h3>



<p class="wp-block-paragraph">Moving from lake to warehouse is straightforward: you are loading modelled tables into a new engine. Moving from warehouse to lake is harder, because the history only exists inside the warehouse and the transformations are written in its dialect. The asymmetric fix is to land raw data in object storage regardless of which you query from. That single habit preserves the option cheaply.</p>



<h3 class="wp-block-heading">What about just using the CRM&#8217;s own analytics product?</h3>



<p class="wp-block-paragraph">Worth pricing seriously before building anything. It is genuinely the right answer when all your questions are about one system, and it removes an entire pipeline from your life. It stops being the right answer the moment a question spans CRM plus billing plus product usage, because you will be paying to ingest the other systems into a platform priced for CRM data.</p>



<h2 class="wp-block-heading">The short version</h2>



<p class="wp-block-paragraph">In the data lake vs data warehouse question for CRM analytics, volume is the argument everyone reaches for and it is the one that matters least. Your CRM data is small. Both options handle it.</p>



<p class="wp-block-paragraph">What decides it is how much history you need, how fast the schema moves, what shape the queries are, and who is going to keep it running. Answer those four and the architecture falls out on its own. And whatever you pick, land the raw payloads in object storage from day one, because that is the cheap decision that keeps every other decision reversible.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second opinion on your CRM analytics stack?</h2>



<p class="wp-block-paragraph">I help teams make this call without over-building, and unpick it when it has already been over-built. Things I can help with:</p>



<ul class="wp-block-list">
<li>Running the sizing and cost model above against your real query volume, not a vendor calculator.</li>

<li>Designing point-in-time history capture for CRM objects, including deletes and slowly changing dimensions.</li>

<li>Setting up zero-ETL or connector-based ingestion from Salesforce, HubSpot, Zendesk or Zoho into Redshift, Snowflake or an Iceberg lake.</li>

<li>Migrating an unmaintainable pile of Parquet into something an analyst can actually query.</li>

<li>Cutting query costs on an existing setup: partitioning, table layout, dashboard refresh patterns.</li>

<li>Building the curated CRM models that BI tools sit on, in SQL and in version control.</li>
</ul>



<p class="wp-block-paragraph">If you are mid-argument about this, send me your row counts and the ten questions the business wants answered. That is usually enough to settle it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/data-lake-vs-data-warehouse-crm-analytics/">Data Lake vs Data Warehouse for CRM Analytics: Volume Is the Wrong Question</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Redshift Cost Optimization for SaaS Analytics: The Levers That Actually Move the Bill</title>
		<link>https://john-nessime.com/blog/devops/redshift-cost-optimization-saas-analytics/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 04 Aug 2026 09:18:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Business Intelligence]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Warehouse]]></category>
		<category><![CDATA[Embedded Analytics]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Multi-Tenant]]></category>
		<category><![CDATA[Row-Level Security]]></category>
		<category><![CDATA[SQL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=123</guid>

					<description><![CDATA[<p>In a SaaS analytics product, the Redshift bill tracks how often queries arrive, not how much data they touch. Here is how the meter actually works, why connection pools bill you while nobody is using the product, how to attribute spend to a tenant, and which isolation choices quietly cost more than they save.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/redshift-cost-optimization-saas-analytics/">Redshift Cost Optimization for SaaS Analytics: The Levers That Actually Move the Bill</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually comes from whoever owns the AWS bill, and it is never dramatic. &#8220;Redshift is up again this month. Did we onboard someone big?&#8221; Nobody onboarded anyone. Nobody shipped a new dashboard. Query volume looks flat on the Grafana board. The bill moved anyway.</p>



<p class="wp-block-paragraph">That gap between what you think you are paying for and what you are actually paying for is what makes Redshift cost optimization awkward in a SaaS analytics product. You are not running one nightly batch against a warehouse that sleeps the rest of the day. You are serving hundreds of small, latency-sensitive queries that fire whenever a customer opens a dashboard, plus ingestion, plus whatever your BI layer and your connection pool are doing when nobody is watching.</p>



<p class="wp-block-paragraph">This post covers the levers that genuinely move that number: how the meter works, why idle-looking connections still bill, how to work out which tenant is expensive, and which isolation choices cost more than they save. Where the popular advice is wrong for SaaS specifically, I will say so.</p>



<h2 class="wp-block-heading">How Amazon Redshift actually charges you</h2>



<p class="wp-block-paragraph">Three buckets, and they behave very differently.</p>



<ul class="wp-block-list"><li><strong>Compute.</strong> On Redshift Serverless this is RPU-hours, metered per second. On provisioned clusters it is node-hours, plus separate line items for concurrency scaling and Spectrum.</li><li><strong>Storage.</strong> Redshift Managed Storage, billed by GB per month, independent of compute. Snapshots are storage too.</li><li><strong>Everything else.</strong> Cross-region data sharing and snapshot replication, machine learning, data transfer outside the usual in-region S3 paths.</li></ul>



<p class="wp-block-paragraph">In a SaaS analytics workload compute dominates, often overwhelmingly. And the important part: compute is a function of how long the warehouse is awake and at what capacity, not how many rows you touched. Two teams can scan identical data volumes and get bills that differ by a factor of five, purely because of how their queries arrive.</p>



<h2 class="wp-block-heading">The billing mechanic that catches SaaS teams out</h2>



<p class="wp-block-paragraph">Read the serverless billing notes properly once and a lot of mysterious spend stops being mysterious. The parts that matter:</p>



<ul class="wp-block-list"><li>The minimum charge is 60 seconds of resource usage, metered per second beyond that. This is a minimum for the warehouse, not for each individual query.</li><li>Usage is recorded when a transaction <em>completes</em>, rolls back, or is stopped. A transaction that runs for hours shows up in your usage view only at the end.</li><li>Cancel a query before it finishes and you still pay for the time it ran.</li><li>Querying system tables is billed like any other query. Your monitoring loop is a workload.</li><li>After a burst, capacity can stay elevated for a period after the load drops. Scale-down is not instant.</li></ul>



<p class="wp-block-paragraph">Put those together and you reach a conclusion that irritates most engineers: on serverless, ten small queries crammed into one minute are cheaper than the same ten queries spread across ten minutes. Every wake-up costs you a minimum billing window multiplied by your base capacity. That is the opposite of the instinct you have from tuning an OLTP service, where you smooth load out to protect tail latency.</p>



<p class="wp-block-paragraph">Before you change anything, get the real numbers out of the warehouse rather than out of Cost Explorer, which lags and aggregates.</p>



<pre class="wp-block-code"><code>-- Daily billed RPU-seconds converted to RPU-hours.
-- Multiply by your region's on-demand RPU-hour rate for dollars.
SELECT trunc(start_time) AS day,
       sum(charged_seconds) / 3600::double precision AS rpu_hours
FROM   sys_serverless_usage
GROUP  BY 1
ORDER  BY 1 DESC;</code></pre>



<p class="wp-block-paragraph"><code>charged_seconds</code> is the column to build cost reporting on. <code>compute_seconds</code> is informative but it is not what the invoice is derived from, and the two can disagree within a given interval. Two constraints worth knowing before you wire this into a dashboard: the view holds roughly a week of history, and it is visible only to superusers. If you want month-over-month trends, UNLOAD it to S3 on a schedule and query the archive with Amazon Athena instead.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Lever one: connections that look idle and are not</h2>



<p class="wp-block-paragraph">This is the one that bites hardest and shows up last, because there is nothing to see. AWS documents it plainly: Redshift Serverless treats all incoming queries as billable user activity, including lightweight health-check queries sent by connection pools. It does not matter whether the statement came from your application, a JDBC driver, or a pooling framework doing its job.</p>



<p class="wp-block-paragraph">So a pool that fires <code>SELECT 1</code> every thirty seconds to validate connections is a warehouse that never gets to sleep. Your product has no users at 3am and you are still paying the minimum window, over and over, multiplied by base capacity. HikariCP, Apache Commons DBCP and PgBouncer all have some form of this behaviour, and the defaults are tuned for OLTP databases where a validation query costs nothing.</p>



<p class="wp-block-paragraph">Open transactions are the same problem wearing a different hat. A <code>BEGIN</code> without a matching <code>COMMIT</code> or <code>ROLLBACK</code> keeps consuming RPUs until the session ends. Session timeouts exist precisely because this happens.</p>



<p class="wp-block-paragraph">What I would check, in this order:</p>



<ol class="wp-block-list"><li>Disable the pool&#8217;s validation or heartbeat query entirely if the driver allows it. If it does not, stretch the interval as far as your failure tolerance permits.</li><li>Drop idle pool size to something honest. A pool sized for peak that stays warm overnight is pure waste on this pricing model.</li><li>Fix any code path that opens a transaction and returns early on error without ending it.</li><li>Set a session timeout per application role so a leaked connection cannot bill indefinitely.</li></ol>



<pre class="wp-block-code"><code>-- Cap idle sessions for the application role.
-- Value is in seconds; the documented range is 60 to 1,728,000.
ALTER USER analytics_app SESSION TIMEOUT 1800;

-- Cap how many connections a single role can hold open at once.
ALTER USER analytics_app CONNECTION LIMIT 40;

-- What is connected right now, and with what timeout.
SELECT * FROM stv_sessions;</code></pre>



<p class="wp-block-paragraph">Session timeout changes apply to new sessions only, so recycle the pool afterwards or you will conclude the setting does nothing.</p>



<h2 class="wp-block-heading">Lever two: base capacity, max capacity and usage limits are three different things</h2>



<p class="wp-block-paragraph">These get conflated constantly, and two of them will not save you a cent on their own.</p>



<ul class="wp-block-list"><li><strong>Base capacity (base RPU).</strong> The floor. It multiplies every billed second, including that 60-second minimum. Halving base capacity roughly halves the cost of a warehouse dominated by short queries. It also halves the compute those queries get, so watch p95 latency alongside the bill.</li><li><strong>Max capacity (MaxRPU).</strong> A ceiling on how far automatic scaling can go. It caps compute available to the workgroup, it does not stop queries and it does not interrupt anything running. Useful as a guard rail against a runaway scan, useless as a budget.</li><li><strong>Usage limits.</strong> An actual budget, expressed in RPU-hours over a daily, weekly or monthly period. The breach actions are: log to a system table, raise an SNS alert, or turn off user queries.</li></ul>



<p class="wp-block-paragraph">Only the third one can stop you spending money, and only the third one can take your product down at 2pm on a Tuesday. Set it to alert first, live with it for a full billing cycle so you learn the shape of a normal week, then decide whether you are genuinely willing to have queries turned off. In a customer-facing SaaS product the answer is usually no, and the limit stays as an alarm feeding PagerDuty or whatever you already page from.</p>



<p class="wp-block-paragraph">There is also the price-performance target, the slider that hands scaling decisions to AWS in exchange for a stated cost or speed preference. AWS recommends it for mid-range base capacities and advises against it at the very bottom and very top of the RPU scale, so check the current guidance against your base setting before enabling it. It is worth trying on a staging workgroup with a replayed query mix; it is not worth switching on blind in production.</p>



<p class="wp-block-paragraph">On provisioned clusters the equivalent controls are per-feature usage limits: concurrency scaling measured in time, Spectrum measured in data scanned, cross-region data sharing, and extra compute for automatic optimization. Each takes a breach action of log, emit a metric, or disable the feature. Concurrency scaling also earns free credits as the main cluster runs, which is why a moderately bursty provisioned cluster often shows no concurrency scaling charge at all until it suddenly does.</p>



<h2 class="wp-block-heading">Lever three: Redshift cost optimization starts with knowing which tenant is expensive</h2>



<p class="wp-block-paragraph">Be clear-eyed about what is possible here. On serverless you cannot get an exact dollar figure per query, because billing happens at the warehouse level and the minimum charge is shared across whatever else was running in that window. What you can build is a defensible apportionment, and that is enough to find the customer whose scheduled export is quietly eating your margin.</p>



<p class="wp-block-paragraph">Start by labelling every statement your API issues on a tenant&#8217;s behalf.</p>



<pre class="wp-block-code"><code>-- Set in the pool's per-checkout init SQL, or per request.
SET query_group TO 'tenant_4417';

SELECT metric_date, sum(events)
FROM   fact_events
WHERE  tenant_id = 4417
  AND  metric_date &gt;= dateadd(day, -30, current_date)
GROUP  BY 1;

RESET query_group;</code></pre>



<p class="wp-block-paragraph">The label lands in the query log and surfaces as <code>query_label</code> in the SYS monitoring views. Keep it short: the older query log views truncate the label to 30 characters, so a tenant slug beats a UUID with prefixes bolted on.</p>



<pre class="wp-block-code"><code>-- Seven days of activity grouped by tenant label.
-- Note: time columns in the SYS views are microseconds;
-- confirm units before converting anything to money.
SELECT trim(query_label)   AS tenant,
       count(*)            AS queries,
       sum(execution_time) AS exec_time,
       sum(queue_time)     AS queue_time
FROM   sys_query_history
WHERE  start_time &gt; dateadd(day, -7, sysdate)
  AND  query_label LIKE 'tenant_%'
GROUP  BY 1
ORDER  BY exec_time DESC;</code></pre>



<p class="wp-block-paragraph">Three columns in that view earn their keep beyond the obvious ones. <code>result_cache_hit</code> tells you which dashboard queries are already free, which is often a bigger share than people expect. The split between <code>queue_time</code> and <code>execution_time</code> tells you whether you have a tuning problem or a capacity problem, and those have opposite fixes. And <code>user_query_hash</code> groups repeated queries with different literals, which is exactly what an embedded dashboard produces, so it is the fastest way to find the one panel that fifty tenants are running badly.</p>



<p class="wp-block-paragraph">From there, apportion the day&#8217;s <code>charged_seconds</code> by each tenant&#8217;s share of execution time. It is an approximation and you should label it as one when you show it to finance. It is still the difference between &#8220;Redshift costs us a lot&#8221; and &#8220;eleven percent of our warehouse spend is one customer pulling an unbounded date range every fifteen minutes.&#8221;</p>



<h2 class="wp-block-heading">Lever four: the isolation model you picked is a cost decision</h2>



<p class="wp-block-paragraph">AWS&#8217;s SaaS guidance describes three partitioning models, and each one has a distinct cost signature on Redshift.</p>



<ul class="wp-block-list"><li><strong>Pool.</strong> All tenants share tables with a tenant identifier column. Cheapest by a wide margin, one warehouse to keep warm, one set of statistics. You pay for it in noisy-neighbour risk and in the access-control work you now have to do yourself.</li><li><strong>Bridge.</strong> Separate schemas or databases inside one cluster. Sounds like a compromise, behaves like neither. AWS&#8217;s own whitepaper is fairly blunt that the isolation profile does not usually justify it, since cluster-level access grants reach across the databases anyway.</li><li><strong>Silo.</strong> A warehouse per tenant. Clean boundaries and per-tenant cost visibility for free. On serverless it is also the most expensive thing you can do, because every workgroup carries its own base capacity floor and its own 60-second minimums. Twenty small tenants means twenty warehouses waking up independently.</li></ul>



<p class="wp-block-paragraph">Data sharing sits between these and is the pattern I reach for when workload interference is the real problem. One producer handles ingestion and transformation; consumers read the shared data without copying it, and a consumer&#8217;s load does not touch the producer. Genuinely useful for separating a heavy ETL window from customer-facing reads. But be honest about the arithmetic: every consumer is its own billable warehouse. Data sharing buys you performance isolation, not cheaper compute.</p>



<p class="wp-block-paragraph">In a pooled model, the thing I set up first is a sort key that leads with the tenant identifier followed by the time column everyone filters on. That lets Redshift prune blocks before it reads them instead of scanning broadly and filtering afterwards. Combine it with row-level security so the tenant predicate cannot be forgotten by an application bug, and you have removed both the largest cost driver and the scariest failure mode in one change.</p>



<h2 class="wp-block-heading">Lever five: scan less, refresh less</h2>



<p class="wp-block-paragraph">Classic warehouse hygiene still applies, it just pays differently here. Shorter queries mean fewer billed seconds at your base capacity.</p>



<ul class="wp-block-list"><li><strong>Sort keys that match your real predicates.</strong> Not the ones from the design doc. Pull the top twenty query hashes and read their WHERE clauses.</li><li><strong>Materialized views for the panels every tenant loads.</strong> Real savings on the read path, but refresh is compute you pay for. A view refreshed every five minutes and read twice an hour is a net loss.</li><li><strong>Let the result cache work.</strong> Identical query text against unchanged data is free. Anything your BI layer does that injects a timestamp or a random parameter into otherwise identical SQL is throwing that away. Worth checking in Amazon QuickSight, Metabase or whatever sits in front.</li><li><strong>Tune zero-ETL refresh intervals.</strong> The refresh interval on the target database is adjustable via <code>ALTER DATABASE</code>. Shorter is fresher and more expensive. For reporting and historical analysis, a longer interval is usually the right call and nobody notices.</li><li><strong>Keep cold history out of managed storage.</strong> Partitioned Parquet or Apache Iceberg tables in S3, catalogued in AWS Glue, queried through the lake. On serverless those queries bill at the same RPU rate rather than as a separate Spectrum line, so the win is in scan efficiency and storage cost, not in dodging a charge.</li></ul>



<p class="wp-block-paragraph">One reassuring detail: the automatic optimization work Redshift does in the background is not billed by default. It becomes billable only if you explicitly enable extra compute resources so those operations can run during busy periods. That is a deliberate trade, not an accident, and it is worth knowing before you turn it on.</p>



<h2 class="wp-block-heading">Provisioned or serverless: how I would decide</h2>



<p class="wp-block-paragraph">Both have a genuine case and the honest answer depends on the shape of your load, not on which is newer.</p>



<p class="wp-block-paragraph">Serverless wins when demand is spiky or concentrated in business hours, when you cannot forecast capacity, and for dev and test environments that sit idle most of the week. It also folds concurrency scaling and data-lake queries into a single rate, which removes two line items people routinely forget to model.</p>



<p class="wp-block-paragraph">Provisioned RA3 wins when load is steady around the clock, because a reserved commitment on nodes can beat accumulated on-demand RPU-hours, and because you get the full workload management surface: queues, query priority, query monitoring rules with the complete set of controls. If you need to guarantee that a tenant&#8217;s export can never starve the interactive path, that machinery is more expressive than a price-performance slider.</p>



<p class="wp-block-paragraph">Commitment discounts now exist on both sides, including reservations for serverless managed at the payer account level. Rates and terms change, so price it against your own numbers rather than a blog post.</p>



<p class="wp-block-paragraph">The tell is simple. Pull a week of <code>charged_seconds</code> bucketed by hour and plot it. A flat line means you are paying serverless rates for provisioned behaviour. A sawtooth with long dead zones means the opposite.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting: the bill moved and nothing shipped</h2>



<ol class="wp-block-list"><li><strong>Get hourly billed seconds first.</strong> Aggregate <code>charged_seconds</code> by hour from the usage view. If the increase is spread evenly across all 24 hours, it is background activity: a pool, a monitor, a health check. If it is concentrated, it is a workload.</li><li><strong>Check for anything running or queued right now.</strong> A single stuck statement explains a lot of otherwise inexplicable spend.</li><li><strong>Look for transactions that never ended.</strong> A deploy that changed error handling can leave transactions open without a single failed request in your logs.</li><li><strong>Compare query counts against query cost.</strong> Flat count with rising cost points at base capacity changes, scale-down lag, or data growth making the same queries slower.</li><li><strong>Group by <code>user_query_hash</code> and diff against last week.</strong> New shapes appearing means a shipped change. Old shapes getting slower means data or statistics.</li><li><strong>Only then look at storage.</strong> Managed storage grows quietly and it is rarely the cause of a sudden jump, but it is often the cause of a slow one.</li></ol>



<pre class="wp-block-code"><code>-- Anything currently running or waiting.
SELECT user_id, query_id, transaction_id, session_id, status,
       trim(database_name) AS database_name,
       start_time, queue_time, execution_time
FROM   sys_query_history
WHERE  status IN ('running','queued')
ORDER  BY start_time;</code></pre>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list"><li>Treating max capacity as a spending cap. It caps compute, not cost, and it will not stop a workload that simply runs for a long time.</li><li>Optimising individual slow queries while ignoring a connection pool that wakes the warehouse every thirty seconds all night.</li><li>Smoothing scheduled jobs out across the hour to be gentle on the warehouse. On serverless this is backwards; batching into fewer windows costs less.</li><li>Building cost dashboards on <code>compute_seconds</code> instead of <code>charged_seconds</code>, then wondering why the totals never reconcile with the invoice.</li><li>Giving every tenant their own workgroup for isolation, then discovering that base capacity floors and minimum charges multiply by tenant count.</li><li>Setting a usage limit to &#8220;turn off user queries&#8221; on the first day, before anyone knows what a normal week looks like.</li><li>Leaving the monitoring loop itself unbounded. Polling system views every few seconds is a workload that bills like any other.</li></ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list"><li>Label every tenant-originated query with <code>query_group</code> from day one. Retrofitting attribution is far more painful than adding a SET statement to your pool&#8217;s init SQL.</li><li>UNLOAD the serverless usage view to S3 on a schedule. Seven days of retention is not enough to argue about a monthly invoice.</li><li>Keep at least one usage limit configured as an alert, permanently, even if you never set a hard cap.</li><li>Review base capacity quarterly against p95 latency, not just against cost. The right number moves as your workload changes.</li><li>Put a hard date bound on every customer-facing query in the application layer. Unbounded ranges are the single most common source of surprise spend in embedded analytics.</li><li>Model concurrency scaling and data-lake charges explicitly if you are on provisioned. They are the line items people forget until they appear.</li><li>Tag workgroups and clusters consistently so cost tooling, whether that is AWS Cost Explorer or something like CloudZero or Vantage, can split spend by environment without guesswork.</li></ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Does Redshift Serverless really charge me when nobody is using the product?</h3>



<p class="wp-block-paragraph">Idle time itself is not billed, but anything that sends a query is. AWS states explicitly that health-check queries from connection pools count as billable user activity. If your pool validates connections on a timer overnight, you are paying minimum billing windows all night. Check the pool before you conclude the pricing model is broken.</p>



<h3 class="wp-block-heading">How do I calculate the cost of a single query?</h3>



<p class="wp-block-paragraph">You cannot, exactly. Serverless bills the warehouse, and the 60-second minimum is shared with whatever else ran in that window. The workable approach is apportionment: take <code>charged_seconds</code> for a period and divide it by each labelled tenant&#8217;s share of execution time from the query history view. Useful for finding outliers, not precise enough for per-customer invoicing.</p>



<h3 class="wp-block-heading">Should I lower base capacity to save money?</h3>



<p class="wp-block-paragraph">Often yes, and it is the single highest-leverage change for a workload made of many short queries, because base capacity multiplies every billed second including the minimum. The catch is that it also reduces the compute each query gets. Change it in one step, watch p95 latency and queue time together for a full week, then decide whether to go further.</p>



<h3 class="wp-block-heading">Is a warehouse per tenant a good idea?</h3>



<p class="wp-block-paragraph">Only when tenants are large enough to keep a warehouse genuinely busy, or when a contract requires that level of separation. For a long tail of small tenants it is the most expensive option available, since each warehouse carries its own capacity floor and its own minimum charges. Pooled tables with row-level security and a tenant-leading sort key gets you most of the isolation for a fraction of the compute.</p>



<h3 class="wp-block-heading">Does concurrency scaling cost extra?</h3>



<p class="wp-block-paragraph">On Redshift Serverless, no, scaling is included in the RPU rate. On provisioned clusters it is a separate charge, offset by credits that accrue while the main cluster runs. That difference catches out teams migrating between the two, in both directions.</p>



<h3 class="wp-block-heading">Will a usage limit take my product down?</h3>



<p class="wp-block-paragraph">It will if you configure the breach action to turn off user queries. The logging and alerting actions are safe and are what you want in a customer-facing system. Treat the hard stop as a deliberate business decision about which is worse, an unexpected invoice or an outage, rather than as a default setting.</p>



<h3 class="wp-block-heading">Why does my cost report never match the AWS invoice?</h3>



<p class="wp-block-paragraph">Usually one of three things: using <code>compute_seconds</code> rather than <code>charged_seconds</code>, forgetting that usage is recorded only when a transaction completes so long transactions land in a later interval, or leaving storage and cross-region transfer out of the model entirely.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Redshift cost optimization for a SaaS analytics product is mostly not a query tuning exercise. It is a question of how often something wakes the warehouse up and at what capacity. Query tuning matters, sort keys matter, materialized views matter, but a connection pool with default settings will quietly outspend all of them combined.</p>



<p class="wp-block-paragraph">So start at the meter. Pull hourly billed seconds, look at the overnight hours when your product has no users, and see whether the line goes to zero. If it does not, you have found your first and cheapest win before touching a single line of SQL.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help getting your Redshift bill under control?</h2>



<p class="wp-block-paragraph">I work with SaaS and data teams on exactly this problem, usually somewhere between the warehouse and the application that is hammering it. Things I can help with:</p>



<ul class="wp-block-list"><li>Auditing an existing Redshift Serverless or RA3 workload and producing a ranked list of what is actually driving spend, with the numbers pulled from your own system views.</li><li>Building per-tenant cost attribution: query labelling, a usage archive in S3, and a dashboard your product and finance teams can both read.</li><li>Fixing the connection and session layer, including pool configuration, validation queries, session timeouts and transaction hygiene.</li><li>Right-sizing base and max capacity against measured latency, and setting usage limits and alerts that warn without risking an outage.</li><li>Reviewing multi-tenant data models: sort and distribution keys, row-level security, and whether data sharing or a pooled model fits your tenant mix.</li><li>Deciding between provisioned and serverless with a workload profile behind the recommendation rather than a rule of thumb.</li></ul>



<p class="wp-block-paragraph">If you have a week of usage data, a suspicious hourly cost chart, or a pool configuration you are not sure about, send it over and I will tell you what I see in it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/redshift-cost-optimization-saas-analytics/">Redshift Cost Optimization for SaaS Analytics: The Levers That Actually Move the Bill</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Embedded Analytics on AWS: The Four Decisions That Bite Later</title>
		<link>https://john-nessime.com/blog/devops/embedded-analytics-on-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 04 Aug 2026 06:07:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Business Intelligence]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Warehouse]]></category>
		<category><![CDATA[Embedded Analytics]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[Multi-Tenant]]></category>
		<category><![CDATA[QuickSight]]></category>
		<category><![CDATA[Row-Level Security]]></category>
		<category><![CDATA[SPICE]]></category>
		<category><![CDATA[SQL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=120</guid>

					<description><![CDATA[<p>Rendering a dashboard inside your app is the easy part. Tenant isolation, session cost and query mode are what break. A practical walkthrough of the four decisions behind embedded analytics on AWS, the API constraints that lock you in, and the errors you will actually see.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/embedded-analytics-on-aws/">Embedded Analytics on AWS: The Four Decisions That Bite Later</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Getting a dashboard to render inside your own application is the easy part. You publish it, call the embed API, drop the iframe in, and it shows up. The hard question arrives about a day later, usually from someone in security or from the first customer who logs in: how exactly does tenant B not see tenant A&#8217;s rows?</p>



<p class="wp-block-paragraph">That is where embedded analytics on AWS stops being a front-end task and becomes an architecture decision. The awkward part is that the choice you make first, how the viewer is identified, quietly decides which isolation mechanisms remain available to you afterwards. Get that order backwards and you rebuild the data layer, not the iframe.</p>



<p class="wp-block-paragraph">This post covers the four decisions that determine whether the build holds: identity model, tenant isolation, query mode, and session economics. Then the embed handshake itself, the errors you will actually see in the browser console, and what to check first when it fails.</p>



<h2 class="wp-block-heading">Before anything else: the product got renamed</h2>



<p class="wp-block-paragraph">Amazon QuickSight was folded into a broader platform called Amazon Quick Suite, and the BI product inside it is now called Amazon Quick Sight. AWS documentation has since moved again under an &#8220;Amazon Quick&#8221; umbrella. You will land on all three naming conventions depending on which search result you click, which makes finding the right doc page genuinely annoying.</p>



<p class="wp-block-paragraph">The practical upshot: the APIs, SDKs and IAM action names did not change. You are still calling <code>quicksight:GenerateEmbedUrlForRegisteredUser</code> against ARNs in the <code>quicksight</code> namespace, and the JavaScript SDK is still published as <code>amazon-quicksight-embedding-sdk</code>. Nothing in your code breaks. Only your bookmarks do. I mention it because half the confusion in a first embedded build comes from following a doc page that describes a UI menu that has since been reorganised.</p>



<h2 class="wp-block-heading">Decision one: registered users or anonymous sessions</h2>



<p class="wp-block-paragraph">Two API operations generate embed URLs. <code>GenerateEmbedUrlForRegisteredUser</code> issues a session for a user who exists inside the BI account. <code>GenerateEmbedUrlForAnonymousUser</code> issues a session for someone who does not, and never will.</p>



<p class="wp-block-paragraph">This reads like a convenience choice. It is not. Row-level security using session tags, the mechanism most SaaS products want, is supported <em>only</em> for anonymous embedding. It does not work with <code>GenerateEmbedUrlForRegisteredUser</code>, it does not work with the older <code>GetDashboardEmbedUrl</code> operation, and it is not supported with the IAM identity type. That constraint is documented, easy to miss, and it is the single most expensive thing to discover late.</p>



<p class="wp-block-paragraph">So the fork is really this. If you register every viewer, you get per-user features (bookmarks, threshold alerts, scheduled snapshots) and you enforce isolation with username or group rules on the dataset. You also inherit the job of provisioning, deprovisioning and reconciling a user directory that mirrors your own. If you go anonymous, you skip all of that and filter with session tags at embed time, but per-user features are off the table because there is no persistent user to hang them on.</p>



<p class="wp-block-paragraph">The registered-user request body is small. Everything interesting is in <code>ExperienceConfiguration</code>:</p>



<pre class="wp-block-code"><code>POST /accounts/&lt;aws-account-id&gt;/embed-url/registered-user

{
  "UserArn": "arn:aws:quicksight:&lt;region&gt;:&lt;account&gt;:user/default/&lt;user&gt;",
  "SessionLifetimeInMinutes": 60,
  "AllowedDomains": ["https://app.example.com"],
  "ExperienceConfiguration": {
    "Dashboard": {
      "InitialDashboardId": "&lt;dashboard-id&gt;",
      "FeatureConfigurations": {
        "Bookmarks": { "Enabled": true }
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">One trap on the anonymous path that deserves its own sentence. Anonymous sessions belong to a namespace, and any dashboard shared with that namespace is reachable by a session in it, whether or not you listed the dashboard in <code>AuthorizedResourceArns</code>. If you were treating that parameter as your allowlist, it is not. Namespace membership is the real boundary.</p>



<h2 class="wp-block-heading">Decision two: where tenant isolation actually lives</h2>



<p class="wp-block-paragraph">There are three places you can put the filter, and only one of them scales.</p>



<ul class="wp-block-list">
<li><strong>A dashboard per tenant.</strong> Works for five customers. Becomes a deployment problem at fifty and a change-management disaster at five hundred, because every visual fix is now a fan-out.</li>



<li><strong>A dataset per tenant, filtered in SQL.</strong> Better isolation guarantees, genuinely defensible in a compliance review, but you multiply refresh jobs and in-memory footprint by tenant count.</li>



<li><strong>One dashboard, one dataset, row-level security.</strong> The standard answer. One artifact to maintain, filtering applied per session.</li>
</ul>



<p class="wp-block-paragraph">With anonymous embedding, RLS is driven by tags. You declare tag keys against columns on the dataset, then supply values at embed time. The filter is evaluated server-side against the session, so a viewer poking at the iframe cannot lift it.</p>



<pre class="wp-block-code"><code>POST /accounts/&lt;aws-account-id&gt;/embed-url/anonymous-user

{
  "Namespace": "default",
  "SessionLifetimeInMinutes": 60,
  "AuthorizedResourceArns": [
    "arn:aws:quicksight:&lt;region&gt;:&lt;account&gt;:dashboard/&lt;dashboard-id&gt;"
  ],
  "SessionTags": [
    { "Key": "tenant_id", "Value": "acme-corp" },
    { "Key": "region",    "Value": "emea" }
  ],
  "AllowedDomains": ["https://app.example.com"],
  "ExperienceConfiguration": {
    "Dashboard": { "InitialDashboardId": "&lt;dashboard-id&gt;" }
  }
}</code></pre>



<p class="wp-block-paragraph">The value in <code>SessionTags</code> must come from your server-side session, never from a request parameter, a cookie your client can write, or a JWT claim you have not verified. This is the whole security boundary. Tag rules support combining conditions, so a manager who should see several sites is expressible without a second dashboard.</p>



<p class="wp-block-paragraph">One quiet limit worth knowing before it bites: when RLS is applied to in-memory datasets, each field has a maximum length in Unicode characters, and fields exceeding it are truncated during ingestion rather than rejected. If your tenant identifiers are long opaque strings, test that a truncated value cannot collide with another tenant&#8217;s. Silent truncation plus a prefix collision is exactly the kind of bug that produces a cross-tenant data leak with no error anywhere in the logs.</p>



<h2 class="wp-block-heading">Decision three: SPICE or direct query against your AWS data</h2>



<p class="wp-block-paragraph">Every dataset runs in one of two modes, and the difference shows up on a bill somewhere else in your account.</p>



<p class="wp-block-paragraph"><strong>Direct query</strong> sends a live query to the source each time a visual renders. Against Amazon Athena that means an S3 scan per dashboard open, billed by bytes scanned. Against Amazon Redshift it means a concurrent query slot per viewer. Freshness is perfect. The failure mode is that dashboard load is now coupled to warehouse load, and your analytics traffic competes with everything else running there. Two hundred people opening a dashboard at 9am is two hundred queries, and Redshift concurrency is finite.</p>



<p class="wp-block-paragraph"><strong>SPICE</strong> imports a snapshot into an in-memory engine and serves every viewer from it. One scan on refresh, then arbitrarily many reads. For an embedded product where the same aggregate is served to thousands of sessions, this is usually the right call, and the Athena cost difference between &#8220;scan once per refresh&#8221; and &#8220;scan once per pageview&#8221; is not subtle. What you give up is freshness, bounded by your refresh schedule, plus a capacity dimension to manage and incremental refresh to configure if the dataset is large.</p>



<p class="wp-block-paragraph">The pattern I reach for first on a data-lake backend is a hybrid: recent partitions in SPICE with an incremental refresh on a look-back window, historical data left on direct query for the rare deep query. It costs more design effort up front and it is the thing most teams skip, but it is the only shape that keeps both the bill and the load time flat as history grows.</p>



<p class="wp-block-paragraph">Whichever you pick, note that visual generation has a timeout, and data-source-specific timeouts apply on top of it. A query that is merely slow in a console tab renders as a broken visual in a customer&#8217;s browser. Model your worst partition, not your average one.</p>



<h2 class="wp-block-heading">Decision four: what a session actually costs</h2>



<p class="wp-block-paragraph">I am not going to quote figures, because AWS changes them and you should read the current pricing page. The mechanism is what matters, and it is genuinely different from seat-based BI licensing.</p>



<ul class="wp-block-list">
<li>A reader session is a fixed 30-minute window. Not a pageview, not a query. Reopening the dashboard twenty minutes later is still the same session.</li>



<li><strong>Per-user pricing</strong> charges per session with a monthly cap per reader. Predictable when the same people return daily.</li>



<li><strong>Capacity pricing</strong> buys sessions in bulk with no user provisioning at all. This is the model built for embedding, and it is the one that pairs with anonymous sessions.</li>



<li>Capacity pricing is also the prerequisite for programmatic dashboard refresh, so if near-real-time rendering is a product requirement, that decision is already made for you.</li>



<li>Annual commitments to capacity unlock removing the &#8220;Powered by&#8221; attribution footer. If white-labelling is a contractual requirement, factor that in early rather than discovering it during a customer demo.</li>



<li>Enabling certain Pro-tier and generative Q&amp;A capabilities triggers an account-level monthly infrastructure fee that exists whether or not anyone uses the feature.</li>
</ul>



<p class="wp-block-paragraph">The cost failure mode nobody plans for is architectural rather than commercial. If you embed the dashboard on a tab that loads by default, you bill a session for every user who lands on that page and looks at something else. Lazy-load the iframe on explicit interaction. That one change is often the largest single lever on the bill, and it costs an afternoon.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The embed handshake, and the three things that break it</h2>



<p class="wp-block-paragraph">The flow is short. Your backend authenticates the user with your own identity system, calls the embed URL API with the right tags or user ARN, returns the URL to the browser, and the SDK mounts an iframe against it.</p>



<ol class="wp-block-list">
<li>The generated URL carries a temporary bearer token valid for five minutes, and it is single use once redeemed. Generate it per page load from your backend. Never cache it, never put it in a build artifact, never log it.</li>



<li>Session lifetime is separate from URL validity, set with <code>SessionLifetimeInMinutes</code>, and ranges from fifteen minutes to ten hours with ten hours as the default. Ten hours is almost never what you want for a customer-facing product. Match it to your own session, or shorter.</li>



<li>Domains must be allowed explicitly. An administrator configures static domains in the admin menu, and <code>AllowedDomains</code> on the API call can override that with up to three domains or subdomains per request. Add an <code>AllowedEmbeddingDomains</code> condition to the IAM policy of the calling role, or any developer with that permission can list any domain on the internet.</li>
</ol>



<p class="wp-block-paragraph">On the browser side, the v2 SDK creates an embedding context (which appends its own zero-pixel iframe to <code>body</code> for message passing) and then mounts the experience:</p>



<pre class="wp-block-code"><code>import { createEmbeddingContext } from 'amazon-quicksight-embedding-sdk';

const context = await createEmbeddingContext();

await context.embedDashboard(
  {
    url: embedUrl,                        // fetched from your backend, just now
    container: '#analytics',
    height: '600px',                      // acts as loading height below
    resizeHeightOnSizeChangedEvent: true,
  },
  {
    toolbarOptions: { export: false, undoRedo: false, reset: false },
    attributionOptions: { overlayContent: true },
    onMessage: async (event) =&gt; {
      if (event.eventName === 'ERROR_OCCURRED') {
        console.error(event.message.errorCode);
      }
    },
  }
);</code></pre>



<p class="wp-block-paragraph">Two details in there earn their place. <code>resizeHeightOnSizeChangedEvent</code> turns the <code>height</code> value into a loading placeholder and lets the frame grow to fit content, which is what stops the dashboard rendering into a 600px letterbox with an inner scrollbar. And <code>overlayContent</code> tells the layout to overlay the attribution footer rather than reserve extra height at the bottom for it.</p>



<h2 class="wp-block-heading">Troubleshooting embedded analytics on AWS</h2>



<p class="wp-block-paragraph">Almost every failure lands in one of these. Read the error code out of the <code>ERROR_OCCURRED</code> message before doing anything else.</p>



<ul class="wp-block-list">
<li><strong><code>Forbidden</code></strong> means the URL&#8217;s authentication code expired. You held the URL longer than five minutes, or you served it from a cache, or a retry redeemed it twice. Fix the generation path, not the permissions.</li>



<li><strong><code>Unauthorized</code></strong> means the session obtained from that code expired. Different problem, different fix: your <code>SessionLifetimeInMinutes</code> is shorter than how long people keep the tab open. Handle it by re-fetching a fresh URL and re-mounting rather than letting the frame sit there dead.</li>



<li><strong>Frame never appears at all.</strong> Check the <code>onChange</code> handler for <code>NO_CONTAINER</code> or <code>INVALID_CONTAINER</code>, which usually means you mounted before your target element existed, and for <code>INVALID_URL</code>, which means the URL shape does not match the experience method you called.</li>



<li><strong>Frame appears, dashboard does not.</strong> Nine times out of ten this is the domain allowlist. The request domain has to match what was allowed, including scheme and any subdomain, and a staging hostname that nobody added is the usual culprit.</li>



<li><strong>Modals render off-screen.</strong> A known consequence of auto-resizing height: an export dialog can open above the visible viewport. Listen for <code>MODAL_OPENED</code> and scroll the parent page to the frame position.</li>



<li><strong>Toolbar features silently missing.</strong> Bookmarks, threshold alerts and scheduling require both the SDK toolbar flag and the matching entry under <code>FeatureConfigurations</code> in the embed URL request, and they only exist on the registered-user path. Setting the client flag alone does nothing.</li>



<li><strong>First render is slow, later ones are fine.</strong> Direct query against a cold warehouse. Compare the same query in Athena or Redshift directly to confirm before blaming the BI layer.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Choosing registered-user embedding for the identity story, then discovering session-tag RLS is unavailable on that path.</li>



<li>Treating <code>AuthorizedResourceArns</code> as the security boundary instead of namespace membership.</li>



<li>Deriving a session tag value from anything the client can influence.</li>



<li>Generating the embed URL at build time, or caching it in a CDN, and then not understanding the <code>Forbidden</code> errors.</li>



<li>Leaving session lifetime at the ten-hour default in a customer-facing app.</li>



<li>Putting the dashboard on a default-loaded tab and paying for sessions nobody asked for.</li>



<li>Building the first version on direct query against Athena because it is quicker to wire up, then meeting the scan bill.</li>



<li>Forgetting that embedding and row-level security sit in the Enterprise tier, so a Standard-tier proof of concept proves nothing.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Decide the identity model before you build a single dataset. Everything downstream inherits it.</li>



<li>Put the embed URL call behind one server-side endpoint that reads tenant scope from your own session and nowhere else. One function, one place to audit.</li>



<li>Constrain the calling IAM role with an <code>AllowedEmbeddingDomains</code> condition and scope resources to specific namespaces rather than a wildcard.</li>



<li>Write an automated test that requests tenant A&#8217;s embed URL and asserts tenant B&#8217;s rows are absent. Run it on every dataset change, because RLS breaks silently.</li>



<li>Default to SPICE with a refresh schedule matched to a stated freshness SLA, and only reach for direct query where the SLA genuinely demands it.</li>



<li>Track refresh failures as a first-class alert in CloudWatch or whatever you already run, whether that is Grafana, Datadog or something in-house. A stale dashboard that still renders is worse than one that errors, because nobody notices.</li>



<li>Lazy-load the iframe on user intent, not on page mount.</li>



<li>Keep the embedded surface read-only unless authoring is a real product requirement. Console embedding is a much larger permissions surface than dashboard embedding.</li>
</ul>



<h2 class="wp-block-heading">Is managed BI even the right call?</h2>



<p class="wp-block-paragraph">Worth asking honestly, because the answer is not always yes. The case for the AWS-native route is real: no connector layer to maintain against Athena, Redshift, S3 and Aurora, IAM you already understand, and a usage-based cost model that beats per-seat licensing when your viewers are bursty. If most of your data already sits in AWS, that adds up.</p>



<p class="wp-block-paragraph">The case against is equally real. Visual customisation is limited compared to charting directly against your own API, the attribution footer needs a commitment to remove, and if you want full control of the front end you may be better served by Apache Superset or Metabase self-hosted, or by Grafana where the workload is closer to operational metrics than customer-facing BI. Those come with an operational burden you now own. That is the trade: you either run the BI layer or you rent it, and renting it means living inside its constraints.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do my users need AWS accounts to view an embedded dashboard?</h3>



<p class="wp-block-paragraph">No. With anonymous embedding they need no AWS account and no BI user record at all. Your application authenticates them however you already do, and your backend maps that identity to session tags when it requests the embed URL.</p>



<h3 class="wp-block-heading">Can I use row-level security with registered-user embedding?</h3>



<p class="wp-block-paragraph">Yes, but only with username or group based rules, not with session tags. Tag-based RLS is restricted to the anonymous embedding path. If you need tags, you need anonymous sessions.</p>



<h3 class="wp-block-heading">How long does an embed URL stay valid?</h3>



<p class="wp-block-paragraph">The URL itself carries a bearer token valid for five minutes and usable once. The session it opens is separate and lasts between fifteen minutes and ten hours depending on <code>SessionLifetimeInMinutes</code>, defaulting to ten hours.</p>



<h3 class="wp-block-heading">Should I use SPICE or direct query for embedded analytics on AWS?</h3>



<p class="wp-block-paragraph">SPICE for anything with many viewers per refresh, which describes most embedded products. Direct query where the data must be current to the second, or where the dataset exceeds what you want to hold in memory. A hybrid split by data age is often the right answer and is under-used.</p>



<h3 class="wp-block-heading">Why do I get a Forbidden error when the dashboard worked yesterday?</h3>



<p class="wp-block-paragraph"><code>Forbidden</code> points at the URL, not at permissions. The most common causes are caching the URL, generating it more than five minutes before use, or a client retry redeeming the same single-use token twice. If it is <code>Unauthorized</code> instead, the session expired and you need a fresh URL.</p>



<h3 class="wp-block-heading">Can I white-label the embedded dashboard completely?</h3>



<p class="wp-block-paragraph">Largely. Themes control colours and typography, the SDK hides toolbar controls, and parameters let your own UI drive the dashboard. Removing the attribution footer entirely is tied to an annual capacity commitment, so confirm that against current terms before you promise it to a customer.</p>



<h3 class="wp-block-heading">Does natural-language querying work in an embedded context?</h3>



<p class="wp-block-paragraph">Yes. The SDK exposes a generative Q&amp;A experience alongside dashboards and visuals, driven by curated topics rather than raw tables. It is billed on its own capacity dimension and gates behind the Pro tiers, so treat it as a separate cost decision rather than a free addition.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Embedded analytics on AWS is not a rendering problem. The iframe is the last five percent. The part that decides whether the build survives contact with a second customer is the identity model, because it silently determines which isolation mechanism you are allowed to use, and that in turn shapes your dataset design, your refresh strategy and your bill.</p>



<p class="wp-block-paragraph">Pick that first. Write the cross-tenant test before you write the dashboard. Everything else is recoverable in an afternoon.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with an embedded analytics build?</h2>



<p class="wp-block-paragraph">This is the kind of work I do. Things I can help with directly:</p>



<ul class="wp-block-list">
<li>Reviewing an existing embed integration for cross-tenant leakage, including the session-tag path and the namespace boundary.</li>



<li>Designing the identity and row-level security model before you commit to a dataset layout.</li>



<li>Cutting Athena scan and Redshift concurrency cost by moving the right datasets into SPICE with incremental refresh.</li>



<li>Building the backend embed-URL service with scoped IAM roles, domain conditions and sane session lifetimes.</li>



<li>Setting up refresh failure alerting so a stale dashboard does not quietly serve last week&#8217;s numbers.</li>



<li>Automated cross-tenant isolation tests wired into CI, so an RLS regression fails the build instead of the customer.</li>
</ul>



<p class="wp-block-paragraph">Send me the actual thing: your embed URL request payload with secrets stripped, the browser console error, or the dataset RLS rules. It is much faster to reason about a real payload than a description of one.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/embedded-analytics-on-aws/">Embedded Analytics on AWS: The Four Decisions That Bite Later</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>It Counted the Chunks: Preparing CRM Data for Amazon Bedrock</title>
		<link>https://john-nessime.com/blog/devops/crm-data-amazon-bedrock/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 04 Aug 2026 00:55:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Warehouse]]></category>
		<category><![CDATA[Embeddings]]></category>
		<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[Vector Database]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=116</guid>

					<description><![CDATA[<p>Asked how many open opportunities an account had, the assistant said five. It had nineteen. Retrieval returned five chunks and the model counted them. CRM data is structured and RAG is built for text, so the preparation work is mostly deciding which questions are retrieval and which are queries.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/crm-data-amazon-bedrock/">It Counted the Chunks: Preparing CRM Data for Amazon Bedrock</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The demo goes well until someone from sales asks the assistant how many open opportunities Acme has. It answers immediately: five. Confident, well-phrased, cited.</p>



<p class="wp-block-paragraph">Acme has nineteen. The assistant retrieved five chunks that happened to mention Acme, because five is roughly how many chunks a retrieval returns, and then it counted them. There is no bug. Nothing failed. Similarity search returned the most similar things and the model described what it was given.</p>



<p class="wp-block-paragraph">That is the defining problem with <strong>preparing CRM data for Amazon Bedrock</strong>, and it is upstream of chunk sizes and embedding models. CRM data is structured. Retrieval-augmented generation is built for unstructured text. Feed a table of opportunities through an embedding pipeline and you get a system that is genuinely good at &#8220;what did we discuss with this account&#8221; and quietly terrible at &#8220;how many&#8221;, &#8220;how much&#8221; and &#8220;since when&#8221;.</p>



<p class="wp-block-paragraph">This covers the split that fixes it, what a good document actually looks like, the permissions problem nobody notices until it is a problem, and how to know whether any of it works.</p>



<h2 class="wp-block-heading">Route the question before you build anything</h2>



<p class="wp-block-paragraph">Your users will ask two kinds of question and they need two different mechanisms.</p>



<ul class="wp-block-list">
<li><strong>Narrative questions.</strong> &#8220;What were the objections on the Acme renewal?&#8221; &#8220;Summarise our history with this account.&#8221; &#8220;What did the customer say about pricing?&#8221; These live in notes, emails, call summaries and descriptions. Semantic retrieval is exactly right for them.</li>
<li><strong>Analytical questions.</strong> &#8220;How many open deals over fifty thousand?&#8221; &#8220;What is total pipeline this quarter?&#8221; &#8220;Which accounts have had no activity in ninety days?&#8221; These need aggregation, filtering and joins. Retrieval cannot do them and will not tell you it cannot.</li>
</ul>



<p class="wp-block-paragraph">The useful part is that you do not have to build the second half yourself. Bedrock Knowledge Bases supports structured data retrieval: you point it at Amazon Redshift or a Glue Data Catalog, and it generates and executes SQL from natural language against your data in place, with no copying or embedding. The counting question goes to SQL, the narrative question goes to the vector store, and both answers are right for the right reasons.</p>



<p class="wp-block-paragraph">So the first architectural decision is not which chunking strategy to use. It is: which of your fields are text worth embedding, and which are facts worth querying. Get that wrong and no amount of tuning downstream will save it.</p>



<p class="wp-block-paragraph">One practical note if you take the structured route. Bedrock&#8217;s execution role authenticates to Redshift as an IAM identity, and you grant it access the same way you would any other reader. Point it at curated views rather than base tables, exactly as you would a BI tool:</p>



<pre class="wp-block-code"><code>-- Give the knowledge base the same narrow access you'd give a
-- reporting user. A view is a contract; a schema grant is not.
GRANT USAGE ON SCHEMA analytics
  TO "IAMR:AmazonBedrockExecutionRoleForKnowledgeBase";

GRANT SELECT ON analytics.v_opportunities
  TO "IAMR:AmazonBedrockExecutionRoleForKnowledgeBase";</code></pre>



<p class="wp-block-paragraph">Named columns and clear table names matter more here than anywhere else, because the model is reading your schema to write the SQL. A column called <code>flag_2</code> will produce exactly the query quality it deserves.</p>



<h2 class="wp-block-heading">What a good document looks like</h2>



<p class="wp-block-paragraph">For the narrative half, the instinct is to export records to CSV and let the chunker deal with it. That produces chunks that are half a row of one record and half a row of another, with no context about which is which.</p>



<p class="wp-block-paragraph">Build documents deliberately instead. Three rules that do most of the work:</p>



<p class="wp-block-paragraph"><strong>One record, one document.</strong> An opportunity and its notes become a single file. A chunk from it is then always about one thing.</p>



<p class="wp-block-paragraph"><strong>Denormalise the context in.</strong> A note that says &#8220;they want a two-year term&#8221; is useless in isolation. Embed the account name, the opportunity name, the stage and the owner into the same document, so a chunk carries enough context to be retrieved and to be understood once retrieved.</p>



<p class="wp-block-paragraph"><strong>Write it as prose, not as fields.</strong> Embedding models were trained on language. <code>Stage__c: Negotiation</code> embeds worse than &#8220;The opportunity is at the Negotiation stage.&#8221; This feels silly and it measurably improves retrieval.</p>



<p class="wp-block-paragraph">On chunking itself: start with the default strategy and change it only when evaluation tells you to. If your documents are one record each and reasonably short, semantic or hierarchical chunking buys you less than people expect. The gains that matter come from document construction and metadata, not from chunk size.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The permissions problem</h2>



<p class="wp-block-paragraph">This is the one that turns a nice project into an incident, and it is easy to miss because nothing about it looks broken.</p>



<p class="wp-block-paragraph">Salesforce has an elaborate sharing model: role hierarchies, sharing rules, territory management, field-level security. A rep sees their accounts and not everyone else&#8217;s. Your vector store has none of that. Once a record is embedded, it is a vector like any other, and anyone who can query the knowledge base can retrieve it.</p>



<p class="wp-block-paragraph">So a rep asks about a competitor deal they were never on the team for, and the assistant helpfully summarises it. Or someone asks about compensation-adjacent records and gets an answer. No alert fires, because from the system&#8217;s perspective retrieval worked perfectly.</p>



<p class="wp-block-paragraph">Three ways out, in increasing order of effort:</p>



<ol class="wp-block-list">
<li><strong>Only embed what everyone may see.</strong> Blunt, easy, and often correct for a first version. Exclude anything sensitive and be explicit that the assistant does not know about it.</li>
<li><strong>Filter at retrieval time using metadata.</strong> Attach owner, team or region to every document, and pass a filter with each query derived from the caller&#8217;s identity. This is the practical answer for most teams.</li>
<li><strong>Separate knowledge bases per audience.</strong> Heavier to operate, and the only option that gives you a hard boundary rather than a filter you have to remember to apply.</li>
</ol>



<p class="wp-block-paragraph">Whichever you pick, decide it before ingestion, because metadata has to be configured at ingest time. Retrofitting a filter you did not plan for means rebuilding the index.</p>



<h2 class="wp-block-heading">Metadata is most of the value</h2>



<p class="wp-block-paragraph">Metadata filtering narrows the candidate set before similarity search runs. That improves relevance, reduces tokens in the prompt, and gives you the access control lever above. For S3 sources it lives in a sidecar file named after the object it describes:</p>



<pre class="wp-block-code"><code>// acme-renewal.txt.metadata.json, alongside acme-renewal.txt in S3
{
  "metadataAttributes": {
    "object_type": "Opportunity",
    "owner_id": "0051t00000XYZ",
    "account_id": "0011t00000ABC",
    "region": "EMEA",
    "is_closed": false,
    "last_modified_epoch": 1750000000
  }
}</code></pre>



<p class="wp-block-paragraph">Include, at minimum: what kind of record it is, who owns it, which account it belongs to, whether it is still open, and when it last changed. Those five turn &#8220;search everything&#8221; into &#8220;search the open opportunities this person owns, changed recently&#8221;, which is a different product.</p>



<h2 class="wp-block-heading">Freshness, and the vectors that outlive the record</h2>



<p class="wp-block-paragraph">An embedding is a copy. When the source record changes, the copy is stale. When the source record is deleted, the copy is not.</p>



<p class="wp-block-paragraph">The consequences are worse in a CRM than in a document store, because CRM records change constantly and confidently answering with last quarter&#8217;s stage is indistinguishable from answering correctly. An assistant that cites a deal which closed-lost in March, or a contact who left the company, is not obviously wrong to the person reading it.</p>



<ul class="wp-block-list">
<li><strong>Sync on a schedule that matches how the data is used.</strong> Daily is fine for account summaries and useless for live pipeline questions, which should be going to SQL anyway.</li>
<li><strong>Handle deletes explicitly.</strong> Whatever removes a record from your lake must also remove its document from the source bucket, or the vector survives.</li>
<li><strong>Put the record&#8217;s last-modified timestamp in the metadata</strong> and surface it in the answer. &#8220;As of three days ago&#8221; is a small change that prevents a large class of misunderstanding.</li>
<li><strong>Reconcile document count against source record count</strong> periodically. Divergence means orphans.</li>
</ul>



<h2 class="wp-block-heading">What not to embed</h2>



<p class="wp-block-paragraph">CRM free-text fields are where people put things. Notes fields contain personal details, occasionally payment information, opinions about customers that were never meant to leave a private conversation, and years of accumulated noise.</p>



<p class="wp-block-paragraph">Once embedded, all of it is retrievable by anyone with query access, and a vector is not something you can easily grep for a mistake.</p>



<ul class="wp-block-list">
<li>Run detection over free-text fields before ingestion and redact rather than hope.</li>
<li>Exclude fields nobody asks questions about. Every extra field is noise competing for retrieval slots.</li>
<li>Consider excluding records closed long ago; they dilute results and rarely answer anything.</li>
<li>Add Bedrock Guardrails on the output side as a second layer, not as your only control.</li>
<li>Keep the whole thing on private networking with VPC endpoints if the data warrants it.</li>
</ul>



<p class="wp-block-paragraph">There is a governance question here too, and it is worth raising before someone else does: your customers&#8217; personal data is now in a vector store feeding a language model. Whether your privacy notice covers that is not an engineering decision, but it is an engineering responsibility to ask.</p>



<h2 class="wp-block-heading">Evaluate it before you show anyone</h2>



<p class="wp-block-paragraph">A demo proves the pipeline runs. It does not tell you whether the answers are right, and this is a system whose failure mode is being wrong fluently.</p>



<p class="wp-block-paragraph">Write thirty real questions from the people who will use it, with correct answers you have verified by hand. Include the analytical ones specifically, because those are where retrieval fails silently. Then run the set after every meaningful change to chunking, metadata or the document format, and compare.</p>



<p class="wp-block-paragraph">Use the Retrieve API rather than RetrieveAndGenerate while you are tuning. Seeing which chunks came back tells you whether a bad answer is a retrieval problem or a generation problem, and those have completely different fixes.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Embedding structured records and expecting aggregation to work.</li>
<li>Exporting to CSV and letting the chunker split rows arbitrarily.</li>
<li>Field-value dumps instead of prose, so embeddings have little language to work with.</li>
<li>No metadata, so every query searches everything and access control is impossible.</li>
<li>Assuming Salesforce sharing rules follow the data into the vector store.</li>
<li>Deciding on filtering after ingestion, then discovering it has to be configured at ingest.</li>
<li>Syncing changes but never removing documents for deleted records.</li>
<li>Embedding every field because it was easier than choosing.</li>
<li>No PII detection over free-text notes.</li>
<li>Tuning chunk size before fixing document construction.</li>
<li>Judging the system on a demo instead of a scored question set.</li>
<li>Granting the execution role a whole schema rather than specific views.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Route analytical questions to structured retrieval and narrative questions to the vector store.</li>
<li>One record per document, with context denormalised in, written as prose.</li>
<li>Metadata on every document: type, owner, account, open state, last modified.</li>
<li>Decide the access model before ingestion, and filter at retrieval time by the caller&#8217;s identity.</li>
<li>Redact free-text fields before embedding, and exclude fields nobody queries.</li>
<li>Delete documents when source records are deleted, and reconcile the counts.</li>
<li>Surface record age in answers so staleness is visible rather than implied.</li>
<li>Expose curated views to the execution role, never whole schemas.</li>
<li>Meaningful table and column names, because the model reads your schema to write SQL.</li>
<li>A scored evaluation set, run on every change, including questions you expect it to fail.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Why does it get counts wrong?</h3>



<p class="wp-block-paragraph">Because retrieval returns the most similar chunks, not all matching records, and the model counts what it was handed. Counting is a query, not a similarity search. Send those questions to structured retrieval over your warehouse instead.</p>



<h3 class="wp-block-heading">Should I use the Salesforce connector or export to S3 first?</h3>



<p class="wp-block-paragraph">The connector is quicker to stand up. Going via S3 gives you control over document construction, metadata and redaction, which is where most of the quality lives. Start with the connector to learn what people ask, then move to S3 once you know what the documents should look like.</p>



<h3 class="wp-block-heading">Which chunking strategy is best?</h3>



<p class="wp-block-paragraph">Usually the default, honestly. If each document is one record, chunking has less work to do. Change it in response to evaluation results rather than in advance, because chunking is the most over-discussed and least decisive variable in a CRM RAG setup.</p>



<h3 class="wp-block-heading">How do I stop people seeing records they shouldn&#8217;t?</h3>



<p class="wp-block-paragraph">Metadata filters applied per query based on the caller&#8217;s identity, separate knowledge bases per audience, or simply not embedding sensitive records. There is no automatic inheritance of CRM sharing rules, and assuming otherwise is the most consequential mistake in this post.</p>



<h3 class="wp-block-heading">How often should I re-sync?</h3>



<p class="wp-block-paragraph">Match the questions. Account histories and notes tolerate a daily sync comfortably. Anything needing current numbers should not be coming from embeddings at all. Whatever the interval, make sure deletions propagate.</p>



<h3 class="wp-block-heading">Do I need a vector database at all?</h3>



<p class="wp-block-paragraph">Only for the narrative half. If every question your users ask is analytical, structured retrieval over Redshift answers all of them with no embedding pipeline, no vector store and no staleness. Plenty of CRM assistants would be better products if they had noticed that.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">The model will answer either way. It will answer the narrative question well and the counting question badly, in the same tone, with the same citations, and nothing in the response distinguishes them.</p>



<p class="wp-block-paragraph">So the preparation work is mostly triage: decide which questions are retrieval and which are queries, build documents that carry their own context, attach metadata before you ingest rather than after, and test with real questions whose answers you already know. Chunk size is the last thing to worry about and the first thing everybody argues about.</p>



<h2 class="wp-block-heading">Building one of these?</h2>



<p class="wp-block-paragraph">The pipeline is usually the easy part; the decisions before it are where these projects succeed or quietly disappoint. Work I take on:</p>



<ul class="wp-block-list">
<li>Designing the split between semantic retrieval and structured querying, so counting questions get correct answers.</li>
<li>Building the document preparation pipeline from CRM data: record-level documents, denormalised context, metadata, redaction.</li>
<li>Access control design for knowledge bases, including per-caller metadata filtering and audience separation.</li>
<li>Freshness and deletion handling so the index does not diverge from the source.</li>
<li>Curated Redshift views and IAM grants for structured data retrieval.</li>
<li>Evaluation harnesses with scored question sets, run on every change rather than once before launch.</li>
</ul>



<p class="wp-block-paragraph">Send me ten questions your users would actually ask, and I will tell you which of them retrieval can answer.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/crm-data-amazon-bedrock/">It Counted the Chunks: Preparing CRM Data for Amazon Bedrock</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>The Column That Never Shows Up: Fixing Redshift and Zoho Analytics Schema Mismatches</title>
		<link>https://john-nessime.com/blog/technical-guides/zoho-analytics-redshift-schema-mismatch/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 02 Aug 2026 08:30:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Business Intelligence]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Warehouse]]></category>
		<category><![CDATA[Database Optimization]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[SQL]]></category>
		<category><![CDATA[Zoho Analytics]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=76</guid>

					<description><![CDATA[<p>You added a column in Redshift, clicked Sync Design, and it reported success. The column still isn't in the dashboard. That's documented behaviour: one unresolved mismatch anywhere blocks new column information everywhere. Here's how to clear them and stop them coming back.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/zoho-analytics-redshift-schema-mismatch/">The Column That Never Shows Up: Fixing Redshift and Zoho Analytics Schema Mismatches</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone from finance asks why <code>discount_amount</code> isn&#8217;t showing up in the dashboard. You added it to the Redshift table last week. You went into the connection settings, clicked Sync Design, waited, and it reported success. The column is still not there.</p>



<p class="wp-block-paragraph">Nothing failed. Sync Design ran exactly as designed. What it did not do is fetch any new column information, because there is an unresolved mismatch sitting in the connection, on a different table, from a rename somebody did two months ago that nobody noticed. Until that is cleared, design sync will keep running and keep declining to pick up anything new.</p>



<p class="wp-block-paragraph">That behaviour is documented, not a bug, and it is the single most useful thing to know about a <strong>Zoho Analytics Redshift schema mismatch</strong>. Mismatches are not independent little problems you can leave lying around. One of them jams the mechanism for all of them.</p>



<p class="wp-block-paragraph">This walks through the three families these problems come in, names, types and time, the fix for each, and the structural change that stops them recurring: stop pointing the BI tool at your base tables.</p>



<h2 class="wp-block-heading">First: work out which mode you are actually in</h2>



<p class="wp-block-paragraph">Zoho Analytics connects to Redshift two completely different ways, and half the confusion in this area comes from people reading advice written for the other one. Whoever set the connection up may have left, so check rather than assume.</p>



<p class="wp-block-paragraph"><strong>Data Import</strong> copies the data into Zoho Analytics on a schedule. Reports are fast because they run against Zoho&#8217;s own storage. In this mode:</p>



<ul class="wp-block-list">
<li>Column additions and deletions are synchronised automatically.</li>
<li>You <em>can</em> change a column&#8217;s data type inside Zoho Analytics, but the type has to stay compatible with the Redshift column or subsequent syncs fail. Zoho&#8217;s own guidance is to change it in both places, which is worth taking literally.</li>
<li>You can create query tables, and import a filtered subset using a custom query.</li>
</ul>



<p class="wp-block-paragraph"><strong>Live Connect</strong> keeps nothing locally and queries Redshift when a report loads. It is available on the paid tiers only. In this mode:</p>



<ul class="wp-block-list">
<li>Column additions, deletions and renames are <em>not</em> synchronised automatically. You have to trigger Sync Design from the Edit Redshift Settings page.</li>
<li>You <em>cannot</em> change a column&#8217;s data type in Zoho Analytics at all. Whatever Redshift says, that is what you get.</li>
<li>No query tables, and you cannot pull other data sources into that workspace.</li>
<li>Foreign keys defined in Redshift become lookup relationships automatically, which is a genuine advantage over Data Import, where you build those by hand.</li>
<li>Report loading time is your Redshift cluster&#8217;s problem now.</li>
</ul>



<p class="wp-block-paragraph">The practical consequence: in Live Connect, every schema fix has to happen in Redshift. There is no BI-side escape hatch. That constraint sounds annoying and is actually the thing that pushes you toward the right architecture, which is the last section of this post.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Family 1: names</h2>



<p class="wp-block-paragraph">Zoho keeps its own copy of the table and column names it expects. When Redshift&#8217;s names drift away from that copy, the difference shows up in the Mismatch tab of the connection settings, and the two most common causes are both silent.</p>



<h3 class="wp-block-heading">Case folding</h3>



<p class="wp-block-paragraph">Redshift lowercases unquoted identifiers. Somebody writes what looks like a camel-case column name, Redshift stores something else, and the BI tool is now looking for a column that does not exist under that name.</p>



<pre class="wp-block-code"><code>-- These two do NOT create the same column.
ALTER TABLE analytics.orders ADD COLUMN DiscountAmount DECIMAL(12,2);
-- ...stored as: discountamount

ALTER TABLE analytics.orders ADD COLUMN "DiscountAmount" DECIMAL(12,2);
-- ...stored as: DiscountAmount</code></pre>



<p class="wp-block-paragraph">Pick one convention, lowercase with underscores, and enforce it. Mixed quoting across a schema means some columns are case-sensitive and some are not, and you will spend an afternoon working out which.</p>



<h3 class="wp-block-heading">See what Redshift actually has</h3>



<p class="wp-block-paragraph">Before touching anything in Zoho, get the ground truth. Use <code>SVV_COLUMNS</code> rather than <code>PG_TABLE_DEF</code>, because the latter only returns rows for schemas that happen to be in your <code>search_path</code> and silently returns nothing otherwise, which has wasted a lot of people&#8217;s time.</p>



<pre class="wp-block-code"><code>SELECT table_name,
       column_name,
       ordinal_position,
       data_type,
       character_maximum_length,
       numeric_precision,
       numeric_scale
FROM svv_columns
WHERE table_schema = 'analytics'
ORDER BY table_name, ordinal_position;</code></pre>



<p class="wp-block-paragraph">Export that, put it next to the Mismatch tab, and work down the list. Guessing from memory is how you resolve four mismatches and leave the fifth.</p>



<h3 class="wp-block-heading">Renames and drops</h3>



<p class="wp-block-paragraph">A rename upstream reads to Zoho as one column disappearing and an unrelated one appearing. If a report or formula referenced the old name, you will also see the alert about a view that cannot be accessed because of changes made to the table. The fix there is to re-synchronise the table from the connection settings, but re-syncing will not help while the Mismatch tab still has entries.</p>



<p class="wp-block-paragraph">So the order is fixed: <strong>clear every mismatch first, then Sync Design, then fix reports.</strong> Doing it in any other order produces the &#8220;I clicked sync and nothing happened&#8221; experience.</p>



<h2 class="wp-block-heading">Family 2: types</h2>



<p class="wp-block-paragraph">Redshift has a rich type system. A BI tool has maybe a dozen column types. The mapping is lossy in places, and the losses are quiet.</p>



<h3 class="wp-block-heading">TEXT and BPCHAR are not what they look like</h3>



<p class="wp-block-paragraph">This one catches people migrating from PostgreSQL, where <code>TEXT</code> is unbounded. In Redshift it is an alias that becomes <code>VARCHAR(256)</code>, and <code>BPCHAR</code> becomes <code>CHAR(256)</code>. Longer values get rejected or truncated depending on how they arrive, and the column reaching Zoho is a 256-character string rather than the free text you thought you had.</p>



<pre class="wp-block-code"><code>-- Looks unbounded. Is not.
CREATE TABLE staging.notes (body TEXT);        -- VARCHAR(256)

-- Say what you mean. 65535 bytes is the VARCHAR maximum.
CREATE TABLE staging.notes (body VARCHAR(65535));</code></pre>



<h3 class="wp-block-heading">VARCHAR length is measured in bytes</h3>



<p class="wp-block-paragraph">Not characters. An accented Latin character costs two bytes, most CJK characters three, an emoji four. A <code>VARCHAR(50)</code> holds fifty English letters or twelve emoji. Names, addresses and free-text fields with international data hit this constantly, and the symptom in the dashboard is a truncated string rather than an error.</p>



<pre class="wp-block-code"><code>-- LENGTH counts characters, OCTET_LENGTH counts bytes.
-- The second number is the one that has to fit.
SELECT MAX(LENGTH(customer_name))       AS max_chars,
       MAX(OCTET_LENGTH(customer_name)) AS max_bytes
FROM analytics.customers;</code></pre>



<p class="wp-block-paragraph">Widening a VARCHAR is one of the few in-place alterations Redshift allows. Narrowing one, or changing a column&#8217;s type outright, generally means rebuilding the table, so size these deliberately at creation rather than planning to fix them later.</p>



<h3 class="wp-block-heading">Numbers and precision</h3>



<p class="wp-block-paragraph">A <code>DECIMAL(38,10)</code> is a perfectly reasonable warehouse column and an awkward BI column. Currency stored as a float is worse, because you get rounding that appears only in the total row and only sometimes, which is a genuinely unpleasant thing to debug in front of a finance team.</p>



<p class="wp-block-paragraph">Cast money to a fixed scale before it leaves Redshift. Two decimal places, <code>DECIMAL</code> not <code>FLOAT</code>, decided once in the warehouse rather than per-report in the BI tool.</p>



<h3 class="wp-block-heading">SUPER, and anything else with no BI equivalent</h3>



<p class="wp-block-paragraph">Semi-structured <code>SUPER</code> columns, <code>VARBYTE</code>, <code>GEOMETRY</code>, <code>HLLSKETCH</code>: there is no sensible column type on the other side. Do not expose them. Flatten what you need into typed scalar columns in a view and leave the rest in the warehouse.</p>



<h3 class="wp-block-heading">If you are in Data Import mode</h3>



<p class="wp-block-paragraph">You have the option of overriding a column&#8217;s type on the Zoho side. Use it sparingly. Zoho&#8217;s requirement is that the type stays compatible with Redshift&#8217;s, and &#8220;compatible&#8221; is doing quiet work in that sentence: an override that works today breaks the next sync when a value arrives that the Zoho type cannot hold. Changing it in both places, as Zoho recommends, is the version that keeps working.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Family 3: time</h2>



<p class="wp-block-paragraph">This one does not appear as a mismatch anywhere. It appears as a reconciliation problem, which is worse, because you spend the first hour looking for missing rows.</p>



<p class="wp-block-paragraph">Redshift has <code>TIMESTAMP</code>, which carries no timezone and means whatever the writer intended, and <code>TIMESTAMPTZ</code>, which is stored in UTC. A BI tool has a timezone setting of its own. Between those, a row written at 23:40 local time can be counted on a different day at each end.</p>



<p class="wp-block-paragraph">The tell is specific and worth memorising: <strong>daily totals match, monthly totals do not.</strong> Nothing is missing. A few hours&#8217; worth of rows at each month boundary are being attributed to the neighbouring month.</p>



<pre class="wp-block-code"><code>-- Run the dashboard's aggregate directly against Redshift and
-- compare. Relative bounds so this keeps working next month.
SELECT DATE_TRUNC('day', created_at) AS day,
       COUNT(*)                      AS orders,
       SUM(total_amount)             AS revenue
FROM analytics.orders
WHERE created_at &gt;= DATEADD(month, -1, DATE_TRUNC('month', GETDATE()))
  AND created_at &lt;  DATE_TRUNC('month', GETDATE())
GROUP BY 1
ORDER BY 1;</code></pre>



<p class="wp-block-paragraph">The fix is to stop making the BI tool guess. Convert in Redshift, expose both the UTC instant and a pre-computed local date, and build every report on the pre-computed one:</p>



<pre class="wp-block-code"><code>-- CONVERT_TIMEZONE is the Redshift idiom. Doing this once here
-- beats doing it in every report and getting it right in most.
SELECT
    created_at                                            AS created_at_utc,
    CONVERT_TIMEZONE('UTC', 'Europe/London', created_at)  AS created_at_local,
    CAST(CONVERT_TIMEZONE('UTC', 'Europe/London', created_at) AS DATE)
                                                          AS order_date_local
FROM analytics.orders;</code></pre>



<p class="wp-block-paragraph">Named zones rather than fixed offsets, so daylight saving is handled for you. A hardcoded offset is correct for roughly half the year.</p>



<h2 class="wp-block-heading">The structural fix: give Zoho a contract, not your tables</h2>



<p class="wp-block-paragraph">Everything above is treatment. This is prevention, and it is the part worth doing even if nothing is currently broken.</p>



<p class="wp-block-paragraph">Pointing a BI tool at base tables means every upstream change is a potential BI incident. Someone widening a column, renaming a field, or adding a <code>SUPER</code> column for a new feature has no idea a dashboard depends on it. Put a view layer in between and that stops being true: the view is the interface, the tables underneath are free to change, and you decide when the interface changes.</p>



<pre class="wp-block-code"><code>CREATE OR REPLACE VIEW analytics.v_orders_bi AS
SELECT
    -- Explicit casts pin the types Zoho will see, so an upstream
    -- change cannot quietly alter the shape of the report.
    CAST(o.order_id      AS BIGINT)         AS order_id,
    CAST(o.order_status  AS VARCHAR(64))    AS order_status,
    CAST(o.total_amount  AS DECIMAL(18,2))  AS total_amount,

    o.created_at                            AS created_at_utc,
    CAST(CONVERT_TIMEZONE('UTC','Europe/London', o.created_at) AS DATE)
                                            AS order_date_local,

    -- SUPER flattened to something a BI column can hold.
    CAST(o.attributes.channel AS VARCHAR(64)) AS channel
FROM analytics.orders o
WITH NO SCHEMA BINDING;</code></pre>



<p class="wp-block-paragraph"><code>WITH NO SCHEMA BINDING</code> creates a late-binding view: it does not hold a dependency on the underlying table, so you can drop and recreate <code>analytics.orders</code> without Redshift refusing or the view vanishing. For a warehouse where tables get rebuilt by a nightly load, that is the difference between a maintenance window and a broken dashboard.</p>



<p class="wp-block-paragraph">List columns explicitly. Never <code>SELECT *</code> in a view a BI tool depends on, because then any upstream column addition changes the contract without anybody deciding to.</p>



<p class="wp-block-paragraph">Give the connection its own read-only Redshift user, granted access to the views and nothing else. That also means the credentials in the BI tool cannot read tables you did not intend to publish:</p>



<pre class="wp-block-code"><code>CREATE USER zoho_reader PASSWORD 'use-a-generated-one';
GRANT USAGE ON SCHEMA analytics TO zoho_reader;
GRANT SELECT ON analytics.v_orders_bi TO zoho_reader;</code></pre>



<p class="wp-block-paragraph">The honest cost: a view layer is a thing to maintain, and adding a column now means editing the view as well as the table. That is the point. The friction is the control. If your views are getting numerous, managing them with dbt or an equivalent gives you version control and review on what is otherwise a pile of undocumented SQL.</p>



<h2 class="wp-block-heading">A repeatable resolution procedure</h2>



<ol class="wp-block-list">
<li><strong>Get ground truth from Redshift.</strong> Run the <code>SVV_COLUMNS</code> query and save the output.</li>
<li><strong>Open the Mismatch tab</strong> in the connection settings and list every entry, including ones on tables nobody reports on.</li>
<li><strong>Resolve every mismatch.</strong> All of them. A single leftover blocks design sync for everything else.</li>
<li><strong>Trigger Sync Design</strong> and confirm the new columns actually appear before moving on.</li>
<li><strong>Check types, not just names.</strong> A column can sync successfully and still be the wrong type. Spot-check the ones carrying money and dates.</li>
<li><strong>Reconcile a known number.</strong> Run the same aggregate in both places for a closed period. Daily and monthly. If daily matches and monthly does not, go back to the timezone section.</li>
<li><strong>Fix broken reports last</strong>, once the data underneath them is right.</li>
</ol>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Sync Design runs but the new column never appears</h3>



<p class="wp-block-paragraph">An unresolved mismatch is blocking it, almost certainly on a table you were not looking at. Clear the Mismatch tab completely, then sync again.</p>



<h3 class="wp-block-heading">&#8220;This view cannot be accessed due to some changes made in the table&#8221;</h3>



<p class="wp-block-paragraph">Something the report depends on was renamed or deleted in Redshift. Re-synchronise that table from the connection settings. If it recurs after every deployment, that is the argument for the view layer.</p>



<h3 class="wp-block-heading">The connection failed entirely</h3>



<p class="wp-block-paragraph">Check three things in order: whether the Redshift database was renamed or dropped, whether Zoho&#8217;s IP addresses are still allowlisted in your security group, and whether the credentials still work. A renamed database needs the connection edited; a dropped one means starting over.</p>



<h3 class="wp-block-heading">Reports show old data after a schema fix</h3>



<p class="wp-block-paragraph">In Live Connect, caching can be enabled per workspace with its own refresh interval, and it applies to reports rather than tables. If the numbers look stale after you fixed something, check that setting before you conclude the fix did not work.</p>



<h3 class="wp-block-heading">Text is truncated in the dashboard but complete in Redshift</h3>



<p class="wp-block-paragraph">Byte-length truncation on multi-byte characters, or a <code>TEXT</code> column that quietly became <code>VARCHAR(256)</code>. Compare <code>LENGTH</code> against <code>OCTET_LENGTH</code> and widen the column.</p>



<h3 class="wp-block-heading">Totals are close but not equal</h3>



<p class="wp-block-paragraph">Timezone if the gap sits at period boundaries. Precision if it is a consistent tiny drift across everything. Filters that differ between the report and your reconciliation query if it is neither. Check them in that order, because the first two are far more common than a genuinely missing row.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Fixing one mismatch, running Sync Design, and assuming the rest can wait.</li>
<li>Not knowing whether the connection is Data Import or Live Connect, and applying advice for the wrong one.</li>
<li>Overriding a column type in Zoho without changing it in Redshift, so the next sync fails.</li>
<li>Using <code>TEXT</code> in Redshift and expecting PostgreSQL behaviour.</li>
<li>Sizing <code>VARCHAR</code> by character count when the limit is in bytes.</li>
<li>Storing currency as a float.</li>
<li>Mixing quoted and unquoted identifiers, so some column names are case-sensitive and some are not.</li>
<li>Letting reports use raw timestamps and setting the timezone per report.</li>
<li>Using a fixed UTC offset instead of a named timezone.</li>
<li>Pointing the BI tool at base tables that a nightly job rebuilds.</li>
<li><code>SELECT *</code> in a view that a dashboard depends on.</li>
<li>Connecting with an admin-level Redshift user because it was quicker.</li>
<li>Declaring the fix done without reconciling a number against the warehouse.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Expose late-binding views to the BI tool, never base tables.</li>
<li>Cast every column explicitly in the view so the types are decided, not inferred.</li>
<li>Do timezone conversion in Redshift and publish a pre-computed local date.</li>
<li>Lowercase, underscore-separated identifiers everywhere, unquoted.</li>
<li>Fixed-scale <code>DECIMAL</code> for money, never floating point.</li>
<li>Size <code>VARCHAR</code> against <code>OCTET_LENGTH</code> of real data, with headroom.</li>
<li>A dedicated read-only Redshift user granted access only to the reporting views.</li>
<li>Treat the Mismatch tab as a queue to empty, not a list to triage.</li>
<li>Reconcile at least one aggregate against the warehouse after every schema change.</li>
<li>Version-control the view definitions, with dbt or just a repository of SQL files.</li>
<li>Tell whoever owns the upstream tables that a view depends on them.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">What exactly is a mismatch in Zoho Analytics?</h3>



<p class="wp-block-paragraph">A disagreement between the table and column names Zoho Analytics expects and the ones Redshift currently has. They are listed in the Mismatch tab of the Redshift connection settings. The important property is that leaving one unresolved stops Sync Design from fetching new column information at all.</p>



<h3 class="wp-block-heading">Why does Sync Design not pick up my new column?</h3>



<p class="wp-block-paragraph">Because there is at least one mismatch outstanding. Clear the Mismatch tab entirely and run it again. It is not a caching issue and re-running it more times will not help.</p>



<h3 class="wp-block-heading">Should I use Data Import or Live Connect?</h3>



<p class="wp-block-paragraph">Data Import when you want fast dashboards, query tables, and the ability to blend data, and can accept the data being as fresh as the last sync. Live Connect when the numbers must be current and you would rather not duplicate the data, accepting that report speed becomes a Redshift performance question and every schema change needs a manual sync.</p>



<h3 class="wp-block-heading">Can I change a column&#8217;s data type in Zoho Analytics?</h3>



<p class="wp-block-paragraph">In Data Import, yes, provided it stays compatible with the Redshift type. In Live Connect, no. Either way the durable fix is to cast the column correctly in a view on the Redshift side, so both ends agree without anyone having to remember an override exists.</p>



<h3 class="wp-block-heading">Why do my dashboard totals not match the warehouse?</h3>



<p class="wp-block-paragraph">If daily figures agree and monthly ones do not, it is timezone handling at period boundaries. If everything is off by a tiny consistent amount, it is numeric precision. Genuinely missing rows are the least likely of the three and the one people check first.</p>



<h3 class="wp-block-heading">How do I handle SUPER columns?</h3>



<p class="wp-block-paragraph">Do not expose them. Extract the specific fields you report on, cast them to scalar types in a view, and let the rest stay in the warehouse. A BI tool has nowhere to put a nested document.</p>



<h3 class="wp-block-heading">Will a view layer slow down Live Connect reports?</h3>



<p class="wp-block-paragraph">A little, since the casts and conversions run per query. In practice the dominant cost is how much data the query scans, so sort keys and distribution keys on the underlying tables matter far more than the view. Measure before optimising, and if a particular view is genuinely expensive, materialise it as a table refreshed by your load job.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">A Zoho Analytics Redshift schema mismatch is not a small isolated problem you can leave in the queue. One unresolved entry stops new columns arriving at all, which is why the symptom people report is almost never &#8220;there&#8217;s a mismatch&#8221; and almost always &#8220;I added a column and nothing happened&#8221;.</p>



<p class="wp-block-paragraph">Clear them all, then sync, then reconcile a real number rather than trusting that it worked. And once it is working, spend the afternoon putting a view layer in between, because the alternative is having this conversation again the next time someone upstream renames a field they had no idea you were reading.</p>



<h2 class="wp-block-heading">Need this sorted out properly?</h2>



<p class="wp-block-paragraph">Warehouse-to-BI connections tend to be set up once, by someone who has since moved on, and then quietly degrade. Work I take on:</p>



<ul class="wp-block-list">
<li>Auditing an existing Redshift to Zoho Analytics connection and clearing the mismatch backlog properly.</li>
<li>Building a reporting view layer in Redshift with explicit casts, timezone handling and late binding, so upstream changes stop breaking dashboards.</li>
<li>Reconciling dashboard figures against the warehouse and finding where the difference comes from.</li>
<li>Redshift schema work: type corrections, column sizing, flattening <code>SUPER</code> data into reportable columns.</li>
<li>Least-privilege database users and network access for BI tools, including security group and allowlist configuration.</li>
<li>Putting the view definitions under version control with dbt so schema changes get reviewed instead of discovered.</li>
</ul>



<p class="wp-block-paragraph">Send me the output of the <code>SVV_COLUMNS</code> query above and a screenshot of your Mismatch tab, and I will tell you what is actually wrong.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/zoho-analytics-redshift-schema-mismatch/">The Column That Never Shows Up: Fixing Redshift and Zoho Analytics Schema Mismatches</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
