<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Zero-ETL | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/zero-etl/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/zero-etl/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sun, 27 Sep 2026 06:01:58 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Zero-ETL | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/zero-etl/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>SageMaker Lakehouse vs Glue + Athena: Is Migration Worth It?</title>
		<link>https://john-nessime.com/blog/cloud-computing/sagemaker-lakehouse-migration/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 27 Sep 2026 06:01:58 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Cost Allocation Tags]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Cross-Account Access]]></category>
		<category><![CDATA[Data Catalog]]></category>
		<category><![CDATA[Data Governance]]></category>
		<category><![CDATA[Federated Catalogs]]></category>
		<category><![CDATA[Hybrid Access Mode]]></category>
		<category><![CDATA[Iceberg REST Catalog]]></category>
		<category><![CDATA[Lake Formation]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[Migration Assessment]]></category>
		<category><![CDATA[Operational Complexity]]></category>
		<category><![CDATA[S3 Tables]]></category>
		<category><![CDATA[SageMaker Unified Studio]]></category>
		<category><![CDATA[Total Cost of Ownership]]></category>
		<category><![CDATA[Zero-ETL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=595</guid>

					<description><![CDATA[<p>SageMaker Lakehouse is built on the Glue Data Catalog and Lake Formation, so a migration barely moves data. It moves enforcement. Here is the failure mode that bites, the cost levers that actually change, and a decision procedure for working out whether the move is worth it at all.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/sagemaker-lakehouse-migration/">SageMaker Lakehouse vs Glue + Athena: Is Migration Worth It?</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A Glue job that had run every morning for two years failed on a Monday. Access denied on <code>GetTable</code>, thrown by Lake Formation. Nothing about the job had changed. Nothing about the IAM role had changed. What had changed was that someone spent Friday afternoon &#8220;moving us onto the lakehouse&#8221; and, as part of that, revoked a permission from a virtual group most people have never heard of.</p>



<p class="wp-block-paragraph">That is the shape of the risk. When people ask whether a SageMaker Lakehouse migration is worth it, they picture data moving. Almost none of it does. What moves is the permission model, and that is the part nobody tests.</p>



<p class="wp-block-paragraph">This post covers what the migration actually changes underneath, the failure mode that bites hardest, where the lakehouse earns its place, where staying on Glue plus Athena is the better call, how the cost levers shift, and a decision procedure you can run in an afternoon.</p>



<h2 class="wp-block-heading">First, understand what you are actually migrating</h2>



<p class="wp-block-paragraph">The framing &#8220;SageMaker Lakehouse versus Glue and Athena&#8221; is misleading, and it drives bad decisions.</p>



<p class="wp-block-paragraph">SageMaker Lakehouse is built on the AWS Glue Data Catalog and AWS Lake Formation. It is not a replacement metastore. It is a reorganisation of the catalog you already have, plus connectors, plus an Apache Iceberg REST interface, plus a console to look at all of it through.</p>



<p class="wp-block-paragraph">The unit it introduces is the <em>catalog</em>: a logical container of databases, tables and views, nestable to mirror the shape of the source system. A <strong>managed catalog</strong> holds data you own in S3 or Redshift Managed Storage. A <strong>federated catalog</strong> points at something external such as Redshift, DynamoDB, PostgreSQL, Oracle, BigQuery or Snowflake, and lets you query it in place.</p>



<p class="wp-block-paragraph">Your existing databases and tables do not disappear. They sit under the account catalog and keep working. AWS states in its own FAQ that you do not have to migrate, and that Athena, EMR, Glue and MWAA continue to work as they do today. That is an unusual thing for a vendor to publish about a flagship product, and it is the most useful sentence in the documentation set.</p>



<p class="wp-block-paragraph">So the question is not &#8220;should we move our data&#8221;. It is &#8220;should we move our governance&#8221;.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The failure mode that actually bites</h2>



<p class="wp-block-paragraph">Lake Formation grants <code>Super</code> permission to a virtual group called <code>IAMAllowedPrincipals</code> on every existing Data Catalog database and table by default. That group effectively means &#8220;anyone whose IAM policy already lets them in&#8221;. It exists purely for backward compatibility. While it holds <code>Super</code> on a resource, access is decided by IAM and S3 policies exactly as it was before Lake Formation existed. Your Lake Formation grants are recorded, but they are not what gets enforced.</p>



<p class="wp-block-paragraph">So the first time someone writes a careful column-level policy and tests it, the policy appears not to work. The analyst can still see the column. The natural next move is to revoke <code>IAMAllowedPrincipals</code> so the grants take effect.</p>



<p class="wp-block-paragraph">That revoke is the moment everything relying on IAM-only access stops working. Glue jobs. Notebooks. Scheduled dbt runs. The QuickSight dataset nobody has looked at since the person who built it left. None of them had a Lake Formation grant, because none of them ever needed one.</p>



<p class="wp-block-paragraph">It is invisible in advance because nothing warns you. Lake Formation cannot know which principals are silently depending on the compatibility group. There is no dry run.</p>



<h3 class="wp-block-heading">Hybrid access mode is the way through</h3>



<p class="wp-block-paragraph">Hybrid access mode exists exactly for this. It supports two permission pathways to the same catalog objects at once. Principals you explicitly opt in are governed by Lake Formation grants. Everyone else keeps reaching the data through their IAM and S3 policies, untouched.</p>



<p class="wp-block-paragraph">The detail worth internalising: when an S3 location is registered in hybrid access mode, Lake Formation enforces only <code>CREATE_TABLE</code>, <code>CREATE_PARTITION</code> and <code>UPDATE_TABLE</code> by default. The write path gets governed first. The read path stays on IAM until you opt a principal in. That asymmetry is what lets you migrate one team at a time instead of one account at a time.</p>



<p class="wp-block-paragraph">The safe sequence: register the location in hybrid mode, opt in one low-stakes principal, verify its queries respect the fine-grained grants, verify a non-opted-in principal still works, repeat. Only after every consumer is opted in and verified do you consider revoking the compatibility group. Plenty of teams never need that last step.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Where the lakehouse genuinely wins</h2>



<ul class="wp-block-list">
<li><strong>One permission surface across engines.</strong> If governance today is &#8220;Lake Formation for the lake, Redshift GRANTs for the warehouse, IAM for DynamoDB&#8221;, you maintain three access models that disagree with each other. Catalogs let you express column and row-level policy once and have Athena, Redshift, EMR and Spark honour it.</li>



<li><strong>Tag-based access control across federated sources.</strong> LF-TBAC extends to federated catalogs covering S3 Tables, Redshift and external sources. If you hand-maintain named-resource grants across dozens of databases, tag expressions are a real operational win.</li>



<li><strong>Querying across boundaries without copying.</strong> A federated catalog lets one statement touch a Redshift table and an S3 Iceberg table, with no ETL job whose only purpose is moving rows so they can sit next to each other. Every pipeline you delete is a pipeline that cannot break at 3am.</li>



<li><strong>The Iceberg REST interface.</strong> Exposing the catalog through Iceberg REST APIs lets engines outside AWS read your tables over a standard protocol. If you run Trino, Spark or DuckDB elsewhere, this is what makes the catalog genuinely portable rather than nominally open.</li>



<li><strong>Cross-account sharing without replication.</strong> For data mesh setups, sharing a catalog instead of shipping a copy per consumer removes a whole category of staleness bug. Cross-account grants on federated catalog objects require the cross-account data sharing version set to 4 or higher.</li>
</ul>



<p class="wp-block-paragraph">Notice what these share. Every one is a governance or interoperability problem. None is a query performance problem, and none is a cost problem.</p>



<h2 class="wp-block-heading">Where Glue plus Athena is still the right answer</h2>



<ul class="wp-block-list">
<li><strong>Single account, S3 only, one team.</strong> Catalogs solve a problem you do not have. You would be adding a governance layer to govern one group.</li>



<li><strong>Your access control is coarse and that is correct.</strong> Not every dataset needs column masking. If bucket-level IAM boundaries match your actual risk model, Lake Formation enforcement adds a failure mode without adding safety.</li>



<li><strong>You have heavy IAM-only automation.</strong> Every Glue job, Lambda, Airflow DAG and third-party tool touching the catalog is a principal you must eventually opt in and re-verify. That inventory is the real cost of migration, and it is always larger than the first estimate.</li>



<li><strong>Your bottleneck is query cost, not governance.</strong> Partitioning, file sizes, compaction and columnar layout drive Athena spend. A lakehouse migration touches none of them. Fix the small-files problem first; it pays back faster and it is reversible.</li>



<li><strong>You depend on a specific engine behaviour today.</strong> Fine-grained access control support varies by engine and table format. Confirm your exact combination is supported rather than assuming parity across Athena, Redshift, EMR and Glue Spark.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The cost levers that move, and the ones that don&#8217;t</h2>



<p class="wp-block-paragraph">Pricing pages change, so here are the billing mechanisms rather than numbers.</p>



<p class="wp-block-paragraph"><strong>Unchanged:</strong> metadata storage and API requests follow existing Glue Data Catalog pricing, including free tier allowances. Reorganising into catalogs does not put you on a new metadata meter. Athena still bills on data scanned. S3 storage and request charges are what they were.</p>



<p class="wp-block-paragraph"><strong>New line items:</strong></p>



<ul class="wp-block-list">
<li><strong>Lambda invocations behind federated catalogs.</strong> When a query hits a federated table, Lake Formation vends credentials that invoke a Lambda function defined in the Glue connection to retrieve metadata from the source. That is compute you did not previously pay for, and it is latency on the metadata path. Chatty BI tools that re-resolve schemas on every refresh are the ones to watch.</li>



<li><strong>Service-managed Redshift Serverless compute.</strong> Queries against a federated catalog pointing at an existing Redshift warehouse execute through a service-managed Redshift Serverless workgroup, which stages results before returning them. That compute is billed, and it does not appear where you expect it.</li>



<li><strong>Redshift Managed Storage.</strong> Managed catalogs backed by RMS rather than S3 put you on Redshift storage and Redshift Serverless compute economics, which behave nothing like S3 plus Athena.</li>
</ul>



<p class="wp-block-paragraph">The service-managed workgroup is the one that catches people, because the spend lands under Redshift Serverless with no obvious link to the analyst who ran the query. AWS publishes a generated cost allocation tag for exactly this, which you must activate in the billing console before it produces anything:</p>



<pre class="wp-block-code"><code># Cost allocation tag on lakehouse-managed Redshift Serverless compute.
# Activate under Billing -&gt; Cost allocation tags before it appears in reports.

Key:   aws:redshift-serverless:LakehouseManagedWorkgroup
Value: "True"</code></pre>



<p class="wp-block-paragraph">Activate it on day one, not after the first surprising invoice. Tags only apply to usage recorded after activation, so a late start leaves a blind period you cannot reconstruct. If you already run a cost visibility layer such as Vantage, CloudZero or Datadog&#8217;s cloud cost views, build a view isolating this tag before you create your first federated catalog. If you do not, the AWS Cost and Usage Report grouped by that tag key is enough to start.</p>



<h2 class="wp-block-heading">Is a SageMaker Lakehouse migration worth it? A decision procedure</h2>



<p class="wp-block-paragraph">Run these in order. Stop at the first one that gives a clear answer.</p>



<ol class="wp-block-list">
<li><strong>Count your non-S3 governed sources.</strong> Redshift clusters, DynamoDB tables, operational databases and external warehouses that analysts need and that each carry their own access model today. Zero or one means the federated catalog story does not apply to you yet. Three or more and the unified permission surface starts paying for itself.</li>



<li><strong>Inventory every principal touching the Data Catalog.</strong> Pull CloudTrail for Glue and Lake Formation API calls across a full billing cycle, long enough to catch monthly jobs. Every distinct role in that list is work. This number, not your data volume, is the migration estimate.</li>



<li><strong>Ask whether column or row-level control has actually been requested.</strong> If the answer is a compliance requirement with a name attached, proceed. If it is &#8220;it would be nice&#8221;, you are buying a permission model to solve a hypothetical.</li>



<li><strong>Check engine coverage against your real workloads.</strong> Confirm fine-grained access control support for each production engine in current documentation. One unsupported critical path makes hybrid mode permanent rather than transitional, which is a legitimate outcome but should be a decision, not a discovery.</li>



<li><strong>Price the federated path against a copy.</strong> For your highest-frequency cross-source query, estimate metadata Lambda invocations plus any service-managed compute, then compare against a nightly job that lands a copy in S3. Federation wins on freshness and on one less pipeline. It does not automatically win on cost.</li>



<li><strong>Decide the destination, not just the direction.</strong> &#8220;Hybrid access mode indefinitely, Lake Formation enforcement on the two regulated datasets&#8221; is a good end state. Write it down first, because migrations without a finish line become permanent half-migrations.</li>
</ol>



<h3 class="wp-block-heading">Three things to run before you commit</h3>



<p class="wp-block-paragraph">Enumerate what Athena can see today. This is your before picture, and it exposes a tooling gap worth knowing up front:</p>



<pre class="wp-block-code"><code>aws athena list-data-catalogs

# Returns catalogs registered directly with Athena, for example:
# {
#   "DataCatalogsSummary": [
#     { "CatalogName": "AwsDataCatalog", "Type": "GLUE" },
#     { "CatalogName": "cw_logs_catalog", "Type": "LAMBDA" }
#   ]
# }</code></pre>



<p class="wp-block-paragraph">S3 Tables catalogs use a <code>s3tablescatalog/bucket-name</code> naming convention and are not returned by this call, even though you can query them by naming the catalog explicitly. The console shows them through mechanisms the public API does not expose. Any inventory or drift-detection tooling built on <code>list-data-catalogs</code> will quietly under-report, so enumerate S3 Tables catalogs separately.</p>



<p class="wp-block-paragraph">Next, confirm the role you intend to use as data lake administrator can create catalogs at all. If it is not already a data lake admin, it needs an explicit grant:</p>



<pre class="wp-block-code"><code>aws lakeformation grant-permissions 
  --cli-input-json 
  '{
     "Principal": {
       "DataLakePrincipalIdentifier": "arn:aws:iam::123456789012:role/Admin"
     },
     "Resource": { "Catalog": {} },
     "Permissions": [ "CREATE_CATALOG", "DESCRIBE" ]
   }'</code></pre>



<p class="wp-block-paragraph">This avoids the common false start where catalog creation works in the console for an admin user and fails for the automation role you actually meant to use.</p>



<p class="wp-block-paragraph">Finally, if you read lakehouse tables from a Glue Spark job, fine-grained access control is not on by default. The job parameters that enable it:</p>



<pre class="wp-block-code"><code>--datalake-formats                          iceberg
--enable-lakeformation-fine-grained-access  true</code></pre>



<p class="wp-block-paragraph">Without the second one, the job reads through its IAM role and your column-level grants are not applied. That is a governance gap that looks exactly like a working pipeline, which is the worst kind.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<ul class="wp-block-list">
<li><strong>&#8220;It&#8217;s the same Glue Data Catalog, so migrating is low risk.&#8221;</strong> True about the metadata, false about enforcement. The catalog is the same object. The thing deciding whether a query returns rows is not.</li>



<li><strong>&#8220;We&#8217;ll migrate to avoid falling behind.&#8221;</strong> AWS states plainly that you can keep using Athena, EMR, Glue and MWAA as you do today. Nothing here is a deprecation. Migrating on an invented schedule is how you end up owning a permission model nobody needed.</li>



<li><strong>&#8220;Federated catalogs mean we can retire our ETL.&#8221;</strong> Some of it, sometimes. Federation removes copies that existed only for co-location. It does not remove transformation, deduplication, late-arriving data handling or historical snapshots. Those jobs stay.</li>



<li><strong>&#8220;Iceberg REST support means no lock-in.&#8221;</strong> The table format is portable and that is real. The Lake Formation grants, LF-Tags, hybrid mode opt-ins and catalog hierarchy are not, and they become the hardest part of your platform to reproduce elsewhere. Portability moved down a layer; it did not appear.</li>



<li><strong>&#8220;We can test this in a sandbox.&#8221;</strong> You can test the mechanics. You cannot test the thing that actually breaks, which is the set of production principals depending on IAM-only access. That inventory does not exist in a sandbox.</li>
</ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<p class="wp-block-paragraph">S3-only, one account, small team: I would not migrate. The same effort spent on partitioning and compaction returns more, and it keeps returning.</p>



<p class="wp-block-paragraph">Multiple governed sources, several consuming teams, a named compliance requirement for fine-grained control: migrate, and treat hybrid access mode as the destination rather than a waypoint. Enforce Lake Formation on the datasets that need it, leave the rest on IAM. The pressure to reach a &#8220;clean&#8221; fully-enforced state is aesthetic, not operational.</p>



<p class="wp-block-paragraph">Either way, start with a federated catalog over one source that is currently painful to reach. Prove the query path and the cost, and let the rest follow evidence rather than a plan drawn before anyone had numbers.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does SageMaker Lakehouse replace AWS Glue and Athena?</h3>



<p class="wp-block-paragraph">No. It is built on the Glue Data Catalog and Lake Formation, and Athena remains one of the engines you query it with. AWS documentation states you can continue using Athena, EMR, Glue and MWAA without migrating. It is a governance and connectivity layer, not a replacement stack.</p>



<h3 class="wp-block-heading">Will a SageMaker Lakehouse migration break my existing Athena queries?</h3>



<p class="wp-block-paragraph">Reorganising into catalogs does not break queries on its own. What breaks them is revoking <code>Super</code> from <code>IAMAllowedPrincipals</code> before every consuming principal has a Lake Formation grant. Use hybrid access mode, opt principals in one at a time, and existing queries keep working throughout.</p>



<h3 class="wp-block-heading">Do I have to move data into Redshift Managed Storage?</h3>



<p class="wp-block-paragraph">No. Managed catalogs can be backed by S3, including S3 Tables, and existing S3 data lakes integrate directly. RMS is a storage choice with different cost and performance behaviour, not a requirement of the architecture.</p>



<h3 class="wp-block-heading">How much extra does it cost?</h3>



<p class="wp-block-paragraph">The catalog layer itself is not a new meter: metadata storage and API requests follow existing Glue Data Catalog pricing including free tier allowances. New costs come from what you build on it, mainly Lambda invocations for federated metadata retrieval, service-managed Redshift Serverless compute for federated Redshift queries, and RMS storage if you choose it.</p>



<h3 class="wp-block-heading">Is SageMaker Unified Studio required?</h3>



<p class="wp-block-paragraph">No. Unified Studio is the integrated console experience. The catalogs are addressable from Athena, Redshift, EMR, Glue and Iceberg-compatible engines directly. Adopting the catalog structure without adopting the studio is a valid and common choice.</p>



<h3 class="wp-block-heading">What is the smallest useful first step?</h3>



<p class="wp-block-paragraph">Create one federated catalog over a single non-S3 source that analysts currently reach through an awkward workaround. Grant read access to one team. Measure query latency and the resulting Lambda and compute charges. That gives you real numbers on your own workload, and it is trivially reversible.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">A SageMaker Lakehouse migration is not a data migration. The Glue Data Catalog underneath is the same catalog, the tables are the same tables, Athena is still Athena. What changes is which system decides whether a query returns rows.</p>



<p class="wp-block-paragraph">That makes the call simpler than the marketing implies. Several governed sources, multiple consuming teams and a real fine-grained control requirement, and the unified permission surface is worth the work. Without them, you are taking on a permission model and a Lambda-backed metadata path in exchange for a nicer console.</p>



<p class="wp-block-paragraph">Whichever way you go: register in hybrid access mode, opt principals in one at a time, and do not revoke <code>IAMAllowedPrincipals</code> until you can name every principal that will notice.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help deciding, or unpicking a migration already underway?</h2>



<p class="wp-block-paragraph">An outside read on the actual inventory usually saves more time than any amount of architecture discussion. Things I help with:</p>



<ul class="wp-block-list">
<li>Building the principal inventory from CloudTrail so you know exactly who depends on IAM-only Data Catalog access before anything is revoked</li>



<li>Designing a hybrid access mode rollout that opts teams in incrementally, with a verification step and a rollback position at every stage</li>



<li>Modelling federated catalog costs against the copy-the-data alternative for your query patterns, including the service-managed compute nobody budgets for</li>



<li>Diagnosing Lake Formation access denied errors across Athena, Glue Spark and Redshift, including the fine-grained access flags that silently bypass your grants</li>



<li>Cost allocation tagging so lakehouse spend is attributable per catalog and per team from the first query rather than retrofitted later</li>



<li>Honest second opinions on whether a migration is worth doing at all, including the answer that it isn&#8217;t</li>
</ul>



<p class="wp-block-paragraph">Send me a failing query, a Lake Formation grant listing or a CloudTrail export and I&#8217;ll tell you what I&#8217;d do with it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>

<p>The post <a href="https://john-nessime.com/blog/cloud-computing/sagemaker-lakehouse-migration/">SageMaker Lakehouse vs Glue + Athena: Is Migration Worth It?</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>SAP Data Integration with AWS Glue: The Failure Modes Nobody Demos</title>
		<link>https://john-nessime.com/blog/devops/sap-data-integration-aws-glue/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 18 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Change Data Capture]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Delta Token]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[ODP]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[SAP]]></category>
		<category><![CDATA[SAP OData]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[VPC]]></category>
		<category><![CDATA[Zero-ETL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=213</guid>

					<description><![CDATA[<p>The AWS Glue SAP OData connector demos beautifully and then drifts. A practical look at delta token invalidation, deletes that never delete, partitioning limits on ODP entities, and the licensing line between ODP-RFC and ODP-OData.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/sap-data-integration-aws-glue/">SAP Data Integration with AWS Glue: The Failure Modes Nobody Demos</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The Glue job had been green for weeks. Every run finished, every run wrote Parquet, nobody had a reason to look at it. Then someone from finance mentioned that a report was short by a few hundred rows, and the reconciliation that followed turned up the real problem: the job had not actually been reading changes for most of that time. It had been reading nothing, successfully.</p>



<p class="wp-block-paragraph">That is the shape of most trouble with SAP data integration with AWS Glue. The connector does not usually crash. It returns an empty or partial result set, the job exits zero, and your data lake quietly diverges from the source system. By the time anyone notices, you have no idea which run was the last correct one.</p>



<p class="wp-block-paragraph">This post covers the parts that bite after the proof of concept: how delta tokens actually behave, why deleted records keep showing up in your reports, where partitioning stops working, and the licensing distinction between two things that both get called &#8220;ODP&#8221; and are not the same thing at all. There is a troubleshooting section at the end for when a job is already broken.</p>



<h2 class="wp-block-heading">First, get the ODP terminology right, because licensing depends on it</h2>



<p class="wp-block-paragraph">Operational Data Provisioning is SAP&#8217;s framework for exposing extraction interfaces in a delta-aware way. A provider (a BW extractor, an ABAP CDS view, SLT, a HANA information view) writes changes into an Operational Delta Queue, and subscribers pull from that queue. This part is uncontroversial.</p>



<p class="wp-block-paragraph">What matters is that there are two ways to talk to ODP, and SAP treats them very differently. SAP Note 3255746 restricts third-party and customer applications from using the RFC modules of the ODP Data Replication API against ABAP sources. That note has been sharpened over several revisions, from &#8220;unsupported&#8221; to &#8220;not permitted&#8221;, and SAP has stated it reserves the right to introduce technical measures and audit unpermitted use. A number of established extraction tools were built directly on ODP-RFC and have had to move.</p>



<p class="wp-block-paragraph">The AWS Glue SAP OData connector takes the other path. It talks OData over HTTPS to an activated SAP Gateway service, and uses the ODP framework through that OData layer rather than through RFC. That is the route SAP points customers at. If you are being asked in an architecture review whether your pipeline is on the wrong side of Note 3255746, this distinction is the whole answer, and it is worth having the note number written down before the meeting.</p>



<p class="wp-block-paragraph">Two honest caveats. First, notes get revised, so read the current version rather than a vendor blog summarising it, including this one. Second, &#8220;permitted&#8221; is not the same as &#8220;free&#8221;: you still need the OData service activated, a service user with the right authorisations, and a Basis team willing to own the queue growth on their side.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: the delta token is shared state you do not own</h2>



<p class="wp-block-paragraph">This is the one that produces silent drift, and it is the reason the finance report was short.</p>



<p class="wp-block-paragraph">When you read an ODP-enabled entity with change data capture switched on, the connector asks SAP to initialise a delta queue subscription. SAP hands back a delta token. That token is your bookmark. On the next run you pass it back, and you get only what changed since.</p>



<pre class="wp-block-code"><code>sapodata_df = glueContext.create_dynamic_frame.from_options(
    connection_type="SAPOData",
    connection_options={
        "connectionName": "sap-prod-odata",
        "ENTITY_NAME": "entityName",
        "ENABLE_CDC": "true"
    },
    transformation_ctx="initial_load"
)</code></pre>



<p class="wp-block-paragraph"><code>ENABLE_CDC</code> defaults to false. Leave it off and you get a full load every single run, which is expensive but at least obvious. Turn it on and the token appears in the <code>DELTA_TOKEN</code> column, populated only on the last row of the frame. That last row is not a business record. It is a carrier for the token, and if you write your frame straight to S3 without stripping it you have just added a junk row to your table.</p>



<p class="wp-block-paragraph">The subsequent run passes the token back:</p>



<pre class="wp-block-code"><code>sapodata_df = glueContext.create_dynamic_frame.from_options(
    connection_type="SAPOData",
    connection_options={
        "connectionName": "sap-prod-odata",
        "ENTITY_NAME": "entityName",
        "DELTA_TOKEN": last_known_token
    },
    transformation_ctx="incremental_load"
)</code></pre>



<h3 class="wp-block-heading">Why this goes wrong</h3>



<p class="wp-block-paragraph">A delta token is bound to a service collection <em>and a user</em>. If a new initialisation runs with <code>ENABLE_CDC</code> set true for that same collection and user, SAP invalidates every token previously issued from the earlier initialisation. Not &#8220;expires eventually&#8221;. Invalidates, immediately.</p>



<p class="wp-block-paragraph">So picture two Glue jobs sharing one service account, both reading the same extractor. Someone redeploys job B with the CDC flag still set from testing. Job B re-initialises. Job A&#8217;s stored token is now dead, and the next time job A runs you get:</p>



<pre class="wp-block-code"><code>Could not open data access via extraction API RODPS_REPL_ODP_OPEN</code></pre>



<p class="wp-block-paragraph">An error is the good outcome. The bad outcome is code that catches the exception, falls back to a full load, and writes it over the top of an incremental table with no dedupe. Or code that re-initialises on failure, which fixes job A and breaks job B, and now the two jobs take turns invalidating each other forever while both keep exiting zero.</p>



<p class="wp-block-paragraph">Practical rules that come out of this:</p>



<ul class="wp-block-list">
<li>One SAP service user per consuming pipeline. Sharing a user is what couples independent jobs together.</li>

<li>Persist the token somewhere durable and versioned. DynamoDB, Parameter Store, S3 with versioning enabled, all fine. Job bookmarks are not a substitute, because the token is source state, not read position.</li>

<li>Never auto re-initialise on error. Fail loudly, page someone, and make the reinitialisation a deliberate human action with a documented backfill.</li>

<li>Alert on zero-change runs. A pipeline that returns no changes for several consecutive runs against a busy table is telling you something, and it is rarely &#8220;business is quiet&#8221;.</li>
</ul>



<h3 class="wp-block-heading">Retention on the SAP side is a hard deadline</h3>



<p class="wp-block-paragraph">The delta queue is not infinite. In transaction <code>ODQMON</code> the Basis team configures retention, and a reorganisation job cleans up data past that window. If your pipeline is down longer than the retention period, the changes you missed are gone from the queue and no token will bring them back. You are into a full reload and a reconciliation.</p>



<p class="wp-block-paragraph">Two things to find out before you go live: what the retention setting is, and whether the cleanup job is actually scheduled. Then make sure your pipeline&#8217;s alerting escalates well inside that window. A weekend outage on a queue with a short retention is a very expensive Monday.</p>



<p class="wp-block-paragraph">Related: if anyone has been testing extraction with <code>RODPS_REPL_TEST</code>, that program registers as a real additional subscriber rather than simulating one. Because the queue is shared and data is only cleaned once every subscriber has collected it, forgotten test subscriptions hold delta data indefinitely and the underlying queue tables grow. Ask whoever ran the tests to reset those subscriptions in ODQMON.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family two: deletes arrive as a column, not as a deletion</h2>



<p class="wp-block-paragraph">CDC extraction from ODP entities returns a couple of extra fields alongside your business columns. <code>DML_STATUS</code> carries <code>UPDATED</code> for inserts and updates, and <code>DELETED</code> for records removed at source. <code>GLUE_FETCH_SQ</code> is a per-record sequence derived from an epoch timestamp, in receipt order, and it is your tie-breaker when the same key changes twice inside one delta.</p>



<p class="wp-block-paragraph">Nothing acts on those fields for you. If your job appends the frame to S3 and calls it done, a deleted sales order is now a row in your lake with <code>DML_STATUS = DELETED</code> and every other column populated exactly as before. Any query that does not filter on that column counts it. Cancelled orders keep appearing in revenue. Closed accounts stay open. The pipeline is green, the numbers are wrong, and the discrepancy grows slowly enough that nobody catches it early.</p>



<p class="wp-block-paragraph">You have to apply the changes rather than accumulate them. Broadly two options:</p>



<ol class="wp-block-list">
<li><strong>Merge into a table format that supports it.</strong> Apache Iceberg, which Glue supports natively, gives you a real merge with delete handling. Land the raw delta, then merge on the business key with <code>GLUE_FETCH_SQ</code> deciding the winner within a batch, deleting where the status says deleted.</li>

<li><strong>Keep an append-only history and build a current-state view.</strong> Cheaper to write, and it preserves an audit trail. It also means every downstream consumer has to use the view, and someone eventually will not.</li>
</ol>



<p class="wp-block-paragraph">I reach for Iceberg first on anything that feeds reporting, because the correctness lives in the table rather than in the discipline of whoever writes the next query. Append-only is the right call when you genuinely need history, or when the consumer is a downstream warehouse like Amazon Redshift that will do its own merge anyway.</p>



<p class="wp-block-paragraph">Worth knowing: not every ODP provider emits deletes reliably. Some older extractors express them as reversal records rather than a delete flag. Confirm the behaviour for your specific entity with the functional owner before you build merge logic on an assumption.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family three: pagination and partitioning do not compose</h2>



<p class="wp-block-paragraph">The connector&#8217;s default page size is 50,000 records, and you can raise it up to 500,000. Anything above that is ignored and the maximum is used instead. Raising the page size cuts the number of round trips, which is usually the dominant cost on a wide table across a slow link.</p>



<p class="wp-block-paragraph">The catch is what your SAP Gateway will tolerate. A large page means a large response the ABAP stack has to assemble and hold, and Gateway or an intervening load balancer may cut it off before it completes. Symptoms are timeouts, connection resets, or a job that appears to paginate without ever finishing. Tune the page size with your Basis team, not in isolation, and treat it as a negotiation between round trips and response size rather than a value you set once.</p>



<p class="wp-block-paragraph">On partitioning, there is a specific documented limit that catches people out: ODP entities are not compatible with record-based partitioning, because pagination is already handled through skip and delta tokens. For record-based partitioning, the default concurrency is forced to null regardless of what you asked for. So the obvious lever for making a slow ODP extraction faster is not available.</p>



<p class="wp-block-paragraph">What you can still do on non-ODP OData entities is field-based partitioning, splitting a read across a bounded range:</p>



<ul class="wp-block-list">
<li><code>PARTITION_FIELD</code> is the field to split on, typically a date.</li>

<li><code>LOWER_BOUND</code> is inclusive, <code>UPPER_BOUND</code> is exclusive. Getting that backwards is how you duplicate or drop a boundary day.</li>

<li><code>NUM_PARTITIONS</code> defaults to 1, so leaving it out means you did not partition at all.</li>

<li>Where a limit and a partition are both applied, the limit wins. That is the usual cause of &#8220;I partitioned it and got fewer rows&#8221;.</li>
</ul>



<p class="wp-block-paragraph">For non-ODP entities you can also do timestamp-based incremental reads with <code>filteringExpression</code>, pointing at whatever last-changed field the entity exposes. Note the casing: that option is camelCase while most of the others are upper snake case, which trips people up when a filter appears to be ignored.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family four: connection, auth and network</h2>



<p class="wp-block-paragraph">These fail early and loudly, which makes them the easy ones. The trap is that they fail during setup, when you have three people on a call and someone suggests loosening something to move on.</p>



<p class="wp-block-paragraph">The connector supports two authentication types: CUSTOM, which is SAP basic authentication, and OAUTH2. The custom secret in AWS Secrets Manager takes this shape:</p>



<pre class="wp-block-code"><code>{
  "basicAuthUsername": "&lt;your SAP username&gt;",
  "basicAuthPassword": "&lt;your SAP password&gt;",
  "basicAuthDisableSSO": "&lt;True/False&gt;",
  "customAuthenticationType": "CustomBasicAuth"
}</code></pre>



<p class="wp-block-paragraph">For OAuth 2.0 the client secret goes into Secrets Manager under <code>USER_MANAGED_CLIENT_APPLICATION_CLIENT_SECRET</code>, and the client ID, authorisation code URL, token URL and scopes are supplied on the connection itself.</p>



<p class="wp-block-paragraph">Basic auth is genuinely easier for a scheduled pipeline: no interactive redirect, no refresh token to keep alive, and rotation is a Secrets Manager operation. OAuth gets you short-lived tokens and revocation you can actually reason about, which is what a security review will want. Both are defensible. What is not defensible is one shared service user across every pipeline, for the delta token reasons above as much as the audit ones.</p>



<p class="wp-block-paragraph">A few things that reliably waste a morning:</p>



<ul class="wp-block-list">
<li>The service path on the connection is the <em>catalog</em> service path, not the path to your object. The entity path goes in <code>ENTITY_NAME</code> on the job.</li>

<li>The OData service has to be activated in the SAP system. Until it is, nothing you do on the AWS side matters.</li>

<li>For a non-VPC connection, the application host URL must be reachable from the public internet. If your SAP system sits in a private network, configure VPC connectivity properly rather than opening it up.</li>

<li>The Glue role needs Secrets Manager read plus the EC2 network interface permissions when a VPC is involved. A missing <code>ec2:CreateNetworkInterface</code> shows up as a connection failure that looks like a network problem.</li>

<li>Private certificate authorities need the chain in a form Glue accepts. If you are wiring this up yourself, expect it to take a pass or two.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Choosing your path: Glue jobs, zero-ETL, or something else</h2>



<p class="wp-block-paragraph">The same connector is reachable through more than one product surface, and the choice is really about how much state you want to own.</p>



<ul class="wp-block-list">
<li><strong>Glue Studio or script jobs.</strong> You control page size, filters, output format, transformation and scheduling, and you own delta token persistence. Pick this when you need real transformation in flight, or when the merge logic has to live next to the extraction.</li>

<li><strong>Glue zero-ETL integrations.</strong> AWS manages token state internally, runs the initial full load and then the incremental cycle, and preserves the token on failure so retries do not lose data. Less code, less control. Worth noting that once an incremental update field is chosen for a connection and entity, a later integration on the same pair cannot pick a different one.</li>

<li><strong>Amazon AppFlow.</strong> Simpler, flow-shaped, well suited to modest volumes and quick prototypes. Less appropriate once you need Spark-scale processing.</li>

<li><strong>Commercial replication tools.</strong> Fivetran, Airbyte and SAP-specialist vendors all play here. They cost money and add a vendor, and in exchange they absorb exactly the state management this post is about. If your team has no SAP depth, that trade is often correct.</li>
</ul>



<p class="wp-block-paragraph">On cost, Glue bills for the resources a job consumes for as long as it runs, so the levers are the same as any Spark workload: fewer round trips, less data pulled, fewer idle executors, and not re-reading history you already have. Rates change and vary by region, so model it against current pricing rather than a figure from a blog post. If your incremental run is pulling the same volume as your initial load, the delta mechanism is not working and the bill is the symptom.</p>



<p class="wp-block-paragraph">Also worth writing back. The connector supports write operations to SAP OData, using <code>INSERT</code> and <code>UPDATE</code>, with <code>ID_FIELD_NAMES</code> required on update to identify the external ID field. Useful for enrichment loops, and something to treat with a good deal more caution than a read.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">&#8220;Could not open data access via extraction API RODPS_REPL_ODP_OPEN&#8221;</h3>



<p class="wp-block-paragraph">Your token is invalid or expired. Find out why before you re-initialise, because re-initialising destroys the evidence. Check ODQMON for a recent initialisation on that collection and user, check whether another job or a test program shares the credentials, and check whether the request you wanted has aged out of retention. Then re-initialise deliberately and reconcile the gap.</p>



<h3 class="wp-block-heading">The job succeeds but returns no rows</h3>



<p class="wp-block-paragraph">Most often a stale or wrong token, a filter predicate that matches nothing, or CDC never having been initialised so the connector has nothing to hand back. Run once without <code>ENABLE_CDC</code> against a small filtered slice and confirm you get data at all. That separates &#8220;no changes&#8221; from &#8220;not reading&#8221;.</p>



<h3 class="wp-block-heading">Pagination never terminates</h3>



<p class="wp-block-paragraph">Usually the SAP side enforcing its own page limit under your requested size, so the connector keeps requesting more. Reduce the page size, verify the filter is actually being applied server side rather than after retrieval, and have Basis confirm what Gateway is capping.</p>



<h3 class="wp-block-heading">Row counts drift over time</h3>



<p class="wp-block-paragraph">Almost always unhandled <code>DML_STATUS = DELETED</code> records, or a token invalidation that silently turned an incremental into a partial. Count distinct business keys in your target against a direct count in SAP for a bounded date range. Make that comparison a scheduled check, not something you run when someone complains.</p>



<h3 class="wp-block-heading">Duplicates after a rerun</h3>



<p class="wp-block-paragraph">A delta was consumed, the write failed, and the token advanced anyway. Store the token only after the write commits, never before. If you are appending rather than merging, add a dedupe on business key ordered by <code>GLUE_FETCH_SQ</code>.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Sharing one SAP service user across multiple pipelines, which couples their delta tokens together.</li>

<li>Automatic re-initialisation in an exception handler. It converts a loud failure into permanent silent drift.</li>

<li>Writing the token-carrier row into the target table as if it were data.</li>

<li>Appending CDC output and ignoring <code>DML_STATUS</code>.</li>

<li>Persisting the token before the write succeeds.</li>

<li>Assuming an entity is ODP-enabled because it looks analytical. Confirm it, because the incremental mechanism differs.</li>

<li>Leaving test subscriptions from <code>RODPS_REPL_TEST</code> registered against the queue.</li>

<li>Building the pipeline without ever asking Basis what the ODQ retention window is.</li>
</ul>



<h2 class="wp-block-heading">Best practices for SAP data integration with AWS Glue</h2>



<ul class="wp-block-list">
<li>Treat the delta token as production state. Versioned storage, a documented recovery procedure, and a written-down answer to &#8220;how do we backfill&#8221;.</li>

<li>One service user per pipeline, one connection per environment. Isolation is the cheapest fix for the worst failure.</li>

<li>Land raw, merge second. Keep the untouched connector output so you can replay a merge without going back to SAP.</li>

<li>Alert on absence, not just errors. Zero rows for several consecutive runs against a busy entity should page someone. Push Glue metrics into CloudWatch and forward them wherever you already look, whether that is Grafana Cloud, Datadog or your own Prometheus.</li>

<li>Reconcile on a schedule against SAP for a bounded window, and record the result. Drift you can see is a bug; drift you cannot see is an incident.</li>

<li>Start with a filtered slice and a small page size while you are building. Get it correct, then tune throughput.</li>

<li>Write down what each field means while the functional owner is still on the call. SAP field names do not explain themselves later.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does the AWS Glue SAP OData connector fall under SAP Note 3255746?</h3>



<p class="wp-block-paragraph">The note restricts the RFC modules of the ODP Data Replication API. The Glue connector uses OData over HTTPS against an activated Gateway service, which is the interface SAP directs customers toward. Read the current revision of the note yourself and get it confirmed with your SAP account team before you rely on any summary, including this one.</p>



<h3 class="wp-block-heading">Where should I store the delta token?</h3>



<p class="wp-block-paragraph">Anywhere durable and versioned that your job can write to transactionally: DynamoDB, Parameter Store, or S3 with versioning. What matters more than the store is the ordering. Write it only after the target write commits, and keep the previous value so you can roll back after a bad run.</p>



<h3 class="wp-block-heading">Can I run two Glue jobs against the same SAP entity?</h3>



<p class="wp-block-paragraph">Yes, but give them separate SAP service users. A delta token is scoped to the service collection and the user, so two jobs on one user will invalidate each other&#8217;s tokens whenever either re-initialises.</p>



<h3 class="wp-block-heading">Why do deleted SAP records still appear in my data lake?</h3>



<p class="wp-block-paragraph">Because the connector reports them rather than applying them. Deletes come back as rows with <code>DML_STATUS</code> set to <code>DELETED</code>. If your pipeline appends without a merge, those rows sit in the target looking like live data. Merge into Iceberg, or filter through a current-state view that every consumer actually uses.</p>



<h3 class="wp-block-heading">What page size should I use?</h3>



<p class="wp-block-paragraph">The default is 50,000 and the ceiling is 500,000. Larger pages mean fewer round trips but bigger responses for SAP to assemble, and Gateway or a load balancer may cut them short. Start at the default, raise it while watching for timeouts, and agree the ceiling with your Basis team.</p>



<h3 class="wp-block-heading">Can I partition an ODP extraction for speed?</h3>



<p class="wp-block-paragraph">Not with record-based partitioning. ODP entities handle pagination through skip and delta tokens, so concurrency is forced to null there regardless of your input. Field-based partitioning on a bounded range is available for non-ODP OData entities. For ODP, tune page size and narrow the entity instead.</p>



<h3 class="wp-block-heading">Can Glue write data back into SAP?</h3>



<p class="wp-block-paragraph">Yes. The SAP OData connector supports insert and update operations, with the external ID field named through <code>ID_FIELD_NAMES</code> on updates. Test that against a sandbox client with a functional owner watching, because a bad write into an ERP system is a very different class of incident to a bad read.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">SAP data integration with AWS Glue is not hard because the connector is difficult to configure. Configuration takes an afternoon. It is hard because the correctness of your pipeline depends on a token held in a queue on a system your team does not administer, subject to a retention policy you may not have been told about, and shared with every other subscriber using the same credentials.</p>



<p class="wp-block-paragraph">Design for that. Isolate the users, persist the token deliberately, apply deletes rather than collecting them, and alert on silence as loudly as you alert on errors. Do that and the connector is a good piece of engineering that will run for a long time without your attention. Skip it and you get a job that stays green while the numbers quietly stop being true.</p>



<h2 class="wp-block-heading">Need a hand with an SAP to AWS pipeline?</h2>



<p class="wp-block-paragraph">This is the kind of work I take on regularly. If you are standing one up, or you have one that has started drifting, I can help with:</p>



<ul class="wp-block-list">
<li>Designing delta token state management that survives failed runs, redeploys and concurrent pipelines.</li>

<li>Building merge logic that applies <code>DML_STATUS</code> deletes correctly into Iceberg or Redshift instead of accumulating ghost rows.</li>

<li>Diagnosing <code>RODPS_REPL_ODP_OPEN</code> failures, silent zero-row runs and row count drift, and working out how far back the damage goes.</li>

<li>Setting up SAP OData connections properly: Secrets Manager, least-privilege IAM, VPC connectivity and private certificate authorities.</li>

<li>Tuning page size, filters and job sizing so incremental runs cost what an incremental run should cost.</li>

<li>Adding reconciliation and absence alerting so drift shows up in a dashboard rather than in a finance report.</li>
</ul>



<p class="wp-block-paragraph">Send me a job log, a connection config, or the row counts that stopped matching, and I will tell you what I think is going on.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/sap-data-integration-aws-glue/">SAP Data Integration with AWS Glue: The Failure Modes Nobody Demos</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Zendesk Data Integration with AWS Glue Zero-ETL: The Delete Gap That Skews Your Numbers</title>
		<link>https://john-nessime.com/blog/technical-guides/zendesk-aws-glue-zero-etl/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 16 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Change Data Capture]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Customer Support Analytics]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Zendesk]]></category>
		<category><![CDATA[Zero-ETL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=223</guid>

					<description><![CDATA[<p>AWS Glue zero-ETL replicates seven Zendesk entities, but only three of them ever remove a row. Here is how that gap quietly skews CSAT and knowledge base counts, plus the three IAM layers to wire, the two settings you cannot change after creation, and the CloudWatch metrics that make drift visible before someone spots it in a meeting.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/zendesk-aws-glue-zero-etl/">Zendesk Data Integration with AWS Glue Zero-ETL: The Delete Gap That Skews Your Numbers</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The support lead pings you on a Tuesday. The CSAT figure on the exec dashboard is higher than the one in Zendesk Explore, and somebody noticed during the meeting. Nothing failed. The Glue integration is green. CloudWatch shows ingestion succeeding on every interval. And yet the warehouse and the source disagree, and have quietly disagreed for weeks.</p>



<p class="wp-block-paragraph">That divergence is not a bug. It is documented behaviour, sitting in one table in the AWS docs that almost everyone scrolls past. This post covers what Zendesk data integration with AWS Glue zero-ETL actually replicates, the entities where deletes silently never propagate, the configuration choices you cannot undo after creation, how to wire the three IAM roles involved, and how to monitor the thing so drift shows up as an alarm instead of a meeting.</p>



<h2 class="wp-block-heading">What Zendesk AWS Glue zero-ETL actually gives you</h2>



<p class="wp-block-paragraph">Zero-ETL is AWS&#8217;s name for managed replication. You create a Glue connection to Zendesk, pick the entities you want, pick a target, and Glue handles the initial snapshot, the schema mapping, and ongoing change data capture. No Spark job. No bookmark logic. No pagination code hammering the Zendesk API and no retry handling for it.</p>



<p class="wp-block-paragraph">Supported targets are a general purpose Amazon S3 bucket through the lakehouse architecture of Amazon SageMaker, S3 Tables through the same lakehouse, Redshift Managed Storage, or an Amazon Redshift data warehouse directly. Data lands as Apache Iceberg, which is what makes row-level updates and deletes viable on object storage at all.</p>



<p class="wp-block-paragraph">What you give up is control. No transform step, no server-side filter, no column pruning at ingest. You get the entity as the connector sees it and you shape it downstream. For most support analytics that trade is fine. It stops being fine the moment your compliance team asks why deleted records are still queryable.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The delete gap: four of seven entities never remove a row</h2>



<p class="wp-block-paragraph">AWS Glue zero-ETL supports seven Zendesk entities. Only three of them replicate deletes.</p>



<ul class="wp-block-list">
<li><strong>tickets</strong> &#8211; create, update and delete all propagate</li>



<li><strong>users</strong> &#8211; create, update and delete all propagate</li>



<li><strong>organizations</strong> &#8211; create, update and delete all propagate</li>



<li><strong>satisfaction-rating</strong> &#8211; create and update only, <strong>no delete</strong></li>



<li><strong>articles</strong> &#8211; create and update only, <strong>no delete</strong></li>



<li><strong>calls</strong> &#8211; create and update only, <strong>no delete</strong></li>



<li><strong>legs</strong> (call legs) &#8211; create and update only, <strong>no delete</strong></li>
</ul>



<p class="wp-block-paragraph">When a satisfaction rating disappears in Zendesk, the row stays in your lakehouse. Permanently. Nothing errors, <code>IngestionSucceeded</code> fires as usual, and the row count in that table only ever goes up.</p>



<p class="wp-block-paragraph">Now think about what your CSAT query does. It averages a score column across a set of rows. If removed ratings never leave the target, your average drifts further from the source every cycle, and the skew is rarely random because retracted or disputed ratings are exactly the ones most likely to be removed. The same mechanism inflates knowledge base article counts, call volume, and any call leg analysis you build. It stays invisible until two dashboards get compared side by side.</p>



<h3 class="wp-block-heading">How to design around it</h3>



<p class="wp-block-paragraph">You cannot make the connector track deletes it does not track. So stop trying, and split the problem instead.</p>



<ol class="wp-block-list">
<li><strong>Run two integrations, not one.</strong> Put tickets, users and organizations in a continuously synced integration. Put the four append-only entities in a second integration you can tear down and rebuild on a cadence that matches how much drift you can tolerate.</li>



<li><strong>Treat the append-only tables as an event log, not as state.</strong> Query them through a view that reconciles against a periodic authoritative count, rather than trusting row presence to mean the record still exists.</li>



<li><strong>Prefer status fields over row existence.</strong> Where an entity exposes a field describing its state, filter on that field downstream. Check which fields your entities actually return before you build on this, because coverage varies by entity.</li>



<li><strong>Reconcile on a schedule.</strong> Pull counts from the Zendesk API for the four entities and compare against the target. Alert on divergence, not on an absolute number.</li>
</ol>



<p class="wp-block-paragraph">Splitting the integration costs you a second set of IAM wiring and a second thing to monitor. It buys you the ability to rebuild the drifting half without touching the half that is correct. That is a good trade.</p>



<h2 class="wp-block-heading">Entity coverage is narrower than the Zendesk API</h2>



<p class="wp-block-paragraph">Seven entities is not the Zendesk API. Ticket comments, ticket metrics, ticket audits, groups, views, macros and SLA policy definitions are not part of the zero-ETL entity set. If your reporting needs first response time, the audit trail behind a status change, or custom field definitions, zero-ETL alone will not get you there.</p>



<p class="wp-block-paragraph">The useful part is that the Glue connection you create is reusable. The same connection backs a normal Glue ETL job, so you can run zero-ETL for the bulk entities and a scheduled Spark job for the ones it does not cover:</p>



<pre class="wp-block-code"><code># Read a Zendesk entity through the same connection
# used by the zero-ETL integration. ENTITY_NAME takes
# the entity name, not the label, e.g. "tickets".
zendesk_read = glueContext.create_dynamic_frame.from_options(
    connection_type="Zendesk",
    connection_options={
        "connectionName": "my-zendesk-connection",
        "ENTITY_NAME": "tickets",
        "API_VERSION": "v2"
    }
)</code></pre>



<p class="wp-block-paragraph">One schema detail worth knowing before you write a single downstream query: the connector converts struct and list types to strings. Zendesk custom fields arrive as arrays of objects, so they land as serialized text, not as a nested column you can address with dot notation. Partitioning is also not supported on the Zendesk source. Plan for a parse step in your silver layer rather than discovering it in a broken dashboard.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Two decisions you cannot change after creation</h2>



<p class="wp-block-paragraph">Most Glue settings are editable. These two are not, and getting them wrong means deleting the integration and starting over.</p>



<h3 class="wp-block-heading">Continuous sync versus on-demand snapshot</h3>



<p class="wp-block-paragraph">By default the integration syncs continuously. Enable the on-demand snapshot setting and it does a single one-time replication with no ongoing CDC. That setting is locked once the integration exists. If you enabled it for a proof of concept and then decided you wanted CDC, you are recreating the integration and re-running the full load.</p>



<h3 class="wp-block-heading">Refresh interval, and why your target choice locks it</h3>



<p class="wp-block-paragraph">The refresh interval controls how often CDC pulls run. It accepts anything from 15 minutes up to 8640 minutes, which is six days. Here is the part that catches people:</p>



<ul class="wp-block-list">
<li>If the target is <strong>Amazon Redshift</strong>, the refresh interval cannot be modified after creation.</li>



<li>For <strong>other targets</strong>, including the SageMaker lakehouse on S3, you can change it later.</li>
</ul>



<p class="wp-block-paragraph">So the target you pick is not only an architecture decision, it is a flexibility decision. If you are not certain what freshness the business actually needs, land in the lakehouse first and query Redshift over it. You keep the ability to tune the interval once you have real usage data instead of a guess made in week one.</p>



<p class="wp-block-paragraph">On cost: AWS does not charge separately for the integration itself. You pay for everything around it, target storage, Redshift or Athena query compute, S3 requests, Data Catalog usage and CloudWatch logs. A 15 minute interval on seven entities produces far more small-file churn and log volume than a 6 hour one. Set the interval from the freshness a human will actually act on, not from the smallest number the field accepts.</p>



<h2 class="wp-block-heading">Wiring the three roles</h2>



<p class="wp-block-paragraph">There are three separate permission layers here and they fail in different ways, which is why &#8220;check IAM&#8221; is useless advice on its own.</p>



<ul class="wp-block-list">
<li><strong>The Zendesk OAuth credential.</strong> The connector uses the authorization code grant, so you get redirected to Zendesk to log in and approve. You can rely on the Glue-managed client application and supply only your instance URL, or register your own OAuth app in the Zendesk admin center and provide your own client ID and secret. The resulting access token does not expire, which is convenient and also means a revoked app in Zendesk is something you will only discover through failed ingestions.</li>



<li><strong>The source role.</strong> This is what lets the integration read through the connection. It is a prerequisite for SaaS sources and it is the step most people miss, because creating the connection successfully does not mean the integration can use it.</li>



<li><strong>The target role and catalog policy.</strong> This is what lets the integration write. A misconfigured catalog resource policy puts the integration into <code>NEEDS_ATTENTION</code>, not into a clear error at creation time.</li>
</ul>



<p class="wp-block-paragraph">The source role policy looks like this. Note <code>glue:RefreshOAuth2Tokens</code>, which is the one people leave out and then spend an afternoon debugging:</p>



<pre class="wp-block-code"><code>{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "GlueConnections",
            "Effect": "Allow",
            "Action": [
                "glue:GetConnections",
                "glue:GetConnection"
            ],
            "Resource": [
                "arn:aws:glue:*:111122223333:catalog",
                "arn:aws:glue:us-east-1:111122223333:connection/*"
            ]
        },
        {
            "Sid": "GlueActionBasedPermissions",
            "Effect": "Allow",
            "Action": [
                "glue:ListEntities",
                "glue:RefreshOAuth2Tokens"
            ],
            "Resource": ["*"]
        },
        {
            "Sid": "CloudWatchLogging",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": ["*"]
        }
    ]
}</code></pre>



<p class="wp-block-paragraph">The trust policy allows <code>glue.amazonaws.com</code> to assume it. Then you attach the role to the connection, which is a CLI-only step and does not appear in the console flow:</p>



<pre class="wp-block-code"><code>aws glue create-integration-resource-property 
  --resource-arn arn:aws:glue:us-east-1:123456789012:connection/my-zendesk-connection 
  --source-processing-properties "{"RoleArn" : "arn:aws:iam::123456789012:role/zendesk-zetl-source"}" 
  --region us-east-1</code></pre>



<p class="wp-block-paragraph">If you skip this, the connection tests fine and the integration still cannot read. That mismatch is the single most common reason a first attempt sits in <code>NEEDS_ATTENTION</code> with no obvious cause.</p>



<h2 class="wp-block-heading">Setting up the integration</h2>



<ol class="wp-block-list">
<li>If you are using your own OAuth app, register it in the Zendesk admin center under the API settings and note the client ID and secret. Store the secret in AWS Secrets Manager, one secret per Glue connection.</li>



<li>In Glue Studio, create a connection under Data Connections. Choose Zendesk as the connection type, supply your instance URL and environment, and complete the OAuth redirect.</li>



<li>Create the source role with the policy above and attach it to the connection using <code>create-integration-resource-property</code>.</li>



<li>Prepare the target: the Glue database or S3 Table bucket, the target role, the catalog resource policy, and any Lake Formation grants your account&#8217;s permission model requires.</li>



<li>Create the integration. Select entities, set the refresh interval, and decide continuous sync versus on-demand snapshot. Both of those are locked from here.</li>



<li>Watch the first full load complete before you point anything at the tables. Full load and CDC are separate load types in the metrics, and the numbers look very different.</li>
</ol>



<p class="wp-block-paragraph">Once it works, move it into code. Glue zero-ETL integrations are supported by CloudFormation and the AWS CDK, which is the difference between a thing one person built in the console and a thing your team can redeploy into another account.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Monitoring: the metrics that expose drift</h2>



<p class="wp-block-paragraph">Zero-ETL publishes to the <code>AWS/Glue/ZeroETL</code> CloudWatch namespace, with dimensions for <code>integrationArn</code>, <code>loadType</code> and <code>tableName</code>. The metrics are <code>InsertCount</code>, <code>UpdateCount</code>, <code>DeleteCount</code>, <code>IngestionSucceeded</code>, <code>IngestionFailed</code> and <code>LastSyncTimestamp</code>.</p>



<p class="wp-block-paragraph">The obvious alarm is on failure:</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-alarm 
  --alarm-name zendesk-zetl-ingestion-failed 
  --namespace AWS/Glue/ZeroETL 
  --metric-name IngestionFailed 
  --dimensions Name=integrationArn,Value=&lt;your-integration-arn&gt; 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 1 
  --threshold 1 
  --comparison-operator GreaterThanOrEqualToThreshold 
  --treat-missing-data notBreaching</code></pre>



<p class="wp-block-paragraph">That one catches loud failures. It does not catch the quiet one, which is an integration that stops running entirely. For that, alarm on <code>IngestionSucceeded</code> with <code>--treat-missing-data breaching</code> over a window longer than your refresh interval. Absence of success is the signal, not presence of failure.</p>



<p class="wp-block-paragraph">And here is the one specific to this post. Because <code>DeleteCount</code> carries a <code>tableName</code> dimension, you can see the delete gap directly in a graph. Plot <code>DeleteCount</code> per table. Tickets, users and organizations will show a nonzero line. Satisfaction ratings, articles, calls and legs will sit flat at zero forever. That flat line is not a broken metric, it is the behaviour, and having it on a dashboard is the cheapest way to keep the whole team aware of it.</p>



<p class="wp-block-paragraph">Glue also writes a system table into the target database recording the outcome of each full load and CDC run, with per-run record, insert and delete counts. Query that when CloudWatch is not granular enough. If you already run Grafana Cloud or Datadog, pulling the <code>AWS/Glue/ZeroETL</code> namespace in alongside your existing infrastructure dashboards puts pipeline health next to everything else on call already watches.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<p class="wp-block-paragraph">Integrations move through Creating, Active, Modifying, Syncing, Needs attention, Failed and Deleting. What each one means in practice:</p>



<ul class="wp-block-list">
<li><strong>Syncing</strong> means it hit a recoverable error and is re-seeding data. Not an emergency. Let it finish before you touch anything.</li>



<li><strong>Needs attention</strong> means you have to fix something, usually connection credentials, the source role, the target role, or the catalog resource policy. Once you fix it, there is no manual recovery action. Glue retries automatically on an exponential backoff schedule, so the gap between attempts grows over time. If you fixed a policy and nothing happened after two minutes, that is expected. Wait it out.</li>



<li><strong>Failed</strong> is terminal. Delete the integration and recreate it. There is no repair path.</li>
</ul>



<p class="wp-block-paragraph">For anything else, the CloudWatch logs emitted after each full load or CDC run carry the actual root cause. The console status is a summary, and a fairly lossy one. Read the logs first.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Building CSAT, article count or call volume metrics on raw row counts from the four entities that never delete.</li>



<li>Creating the connection and assuming the integration can use it. The source role attachment is a separate CLI step.</li>



<li>Choosing Redshift as the direct target during a proof of concept and locking a refresh interval you picked arbitrarily.</li>



<li>Setting a 15 minute interval because it was available, then being surprised by small-file churn and CloudWatch log volume.</li>



<li>Writing downstream queries that address Zendesk custom fields as nested columns. They arrive as strings.</li>



<li>Alarming only on <code>IngestionFailed</code>, so a stalled integration produces silence and silence looks like health.</li>



<li>Reaching for zero-ETL when the requirement is ticket comments or ticket metrics, which are not in the entity set at all.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Split delete-tracking and append-only entities into separate integrations so you can rebuild one without disturbing the other.</li>



<li>Land in the SageMaker lakehouse rather than straight into Redshift unless you are certain about freshness. It keeps the refresh interval editable.</li>



<li>Put <code>DeleteCount</code> per table on a dashboard. It makes an invisible behaviour visible to everyone, not just whoever read the docs.</li>



<li>Define the integration in CloudFormation or CDK once it works, so the next environment is a deploy rather than a repeat of the console clicking.</li>



<li>Keep a silver layer between the replicated tables and your BI tool. That is where you parse the stringified structs and apply the reconciliation logic, and it means schema surprises break one view instead of every dashboard.</li>



<li>Document which entities do not track deletes somewhere your analysts will actually read, next to the tables, not in a wiki page nobody opens.</li>
</ul>



<h3 class="wp-block-heading">When to use something else</h3>



<p class="wp-block-paragraph">Zero-ETL is a good fit when your target is already AWS, your entities are in the supported set, and you want AWS to own the pipeline. It is the wrong fit if you need entities outside that list, need transformation at ingest, or need the same Zendesk data landing somewhere that is not AWS. Managed connector platforms like Fivetran or Airbyte cover a wider slice of the Zendesk API and will replicate to non-AWS destinations, at the cost of another vendor relationship, another bill, and data leaving your account boundary on the way through. If the seven entities cover your reporting, zero-ETL is cheaper and simpler. If they do not, forcing it is worse than paying for a tool that fits.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Which Zendesk entities does AWS Glue zero-ETL support?</h3>



<p class="wp-block-paragraph">Tickets, users, organizations, satisfaction ratings, articles, calls and call legs. Tickets, users and organizations replicate creates, updates and deletes. The other four replicate creates and updates only.</p>



<h3 class="wp-block-heading">Can I get ticket comments or ticket metrics through zero-ETL?</h3>



<p class="wp-block-paragraph">No. They are not in the entity set. Use a Glue ETL job against the same Zendesk connection, or a connector platform with broader API coverage, and join the result to your replicated tables downstream.</p>



<h3 class="wp-block-heading">How fresh can the data be?</h3>



<p class="wp-block-paragraph">The refresh interval goes down to 15 minutes and up to six days. This is near real time replication, not streaming. If you need sub-minute latency on Zendesk events, zero-ETL is the wrong mechanism and you want webhooks into an event pipeline instead.</p>



<h3 class="wp-block-heading">Can I change the refresh interval later?</h3>



<p class="wp-block-paragraph">Only if the target is not Redshift. With Redshift as the target the interval is fixed at creation. With the SageMaker lakehouse and other targets you can modify it afterwards.</p>



<h3 class="wp-block-heading">Why is my integration stuck in NEEDS_ATTENTION after I fixed the permissions?</h3>



<p class="wp-block-paragraph">Because recovery is automatic but not immediate. Glue retries on exponential backoff, so if the integration has been unhealthy for a while the next retry may be some way off. There is no manual recovery command. Confirm the fix is correct, then wait.</p>



<h3 class="wp-block-heading">Does zero-ETL cost extra on top of Glue?</h3>



<p class="wp-block-paragraph">AWS does not bill a separate charge for the integration itself. You pay for the surrounding services: target storage, Redshift or Athena query compute, S3 requests, Data Catalog usage and CloudWatch logs. Refresh interval is the main lever, since it drives how many small commits and log entries you generate.</p>



<h3 class="wp-block-heading">Why do my Zendesk custom fields look like text?</h3>



<p class="wp-block-paragraph">The connector converts struct and list types to strings. Custom fields come through as serialized values rather than nested columns, so parse them in a downstream layer instead of querying them directly.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">Zendesk AWS Glue zero-ETL removes a genuinely tedious pipeline from your plate. The setup is fiddly in the IAM layer and then it mostly runs itself, which is exactly the point of a managed integration.</p>



<p class="wp-block-paragraph">The one thing worth carrying out of this post: four of the seven supported entities never replicate a delete. Satisfaction ratings, articles, calls and call legs only grow. Nothing warns you, every metric stays green, and your numbers drift a little further from Zendesk with every cycle. Design for that on day one, put <code>DeleteCount</code> per table on a dashboard, and you will not be the person explaining the discrepancy in a meeting six months from now.</p>



<h2 class="wp-block-heading">Need help with your Zendesk to AWS data pipeline?</h2>



<p class="wp-block-paragraph">I work with teams building and fixing replication pipelines on AWS. On this kind of setup, that usually means:</p>



<ul class="wp-block-list">
<li>Auditing an existing Zendesk zero-ETL integration and quantifying how far the target has drifted from the source</li>



<li>Designing the split between continuously synced and append-only entities, including the rebuild cadence and reconciliation queries</li>



<li>Untangling the source role, target role and catalog policy layers when an integration sits in <code>NEEDS_ATTENTION</code> with no clear cause</li>



<li>Building the silver layer that parses stringified Zendesk custom fields into something your BI tool can actually use</li>



<li>Setting up CloudWatch alarms and dashboards that catch a stalled integration, not just a failed one</li>



<li>Moving a console-built integration into CloudFormation or CDK so it can be deployed across accounts</li>
</ul>



<p class="wp-block-paragraph">Send me the integration status, a CloudWatch log excerpt, or the query that is giving you the wrong number, and I will tell you what I think is going on.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/zendesk-aws-glue-zero-etl/">Zendesk Data Integration with AWS Glue Zero-ETL: The Delete Gap That Skews Your Numbers</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Data Lake vs Data Warehouse for CRM Analytics: Volume Is the Wrong Question</title>
		<link>https://john-nessime.com/blog/devops/data-lake-vs-data-warehouse-crm-analytics/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 06 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Business Intelligence]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[CRM Analytics]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Warehouse]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[SQL]]></category>
		<category><![CDATA[Zero-ETL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=153</guid>

					<description><![CDATA[<p>Everyone argues this one on data volume, and volume is the argument that matters least: CRM data is small enough that both architectures handle it comfortably. What actually decides data lake vs data warehouse for CRM analytics is how much point-in-time history you need, how fast the schema churns, what shape your queries are, and who is going to maintain the thing. Includes a decision procedure you can run in an afternoon.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/data-lake-vs-data-warehouse-crm-analytics/">Data Lake vs Data Warehouse for CRM Analytics: Volume Is the Wrong Question</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question never arrives as an architecture question. It arrives as someone in RevOps asking whether they can get pipeline coverage by rep, by week, for the last two years, including the deals that were deleted along the way.</p>



<p class="wp-block-paragraph">You look at the CRM reporting tab. It cannot do that. So now you are picking a platform, and within about ten minutes someone will say &#8220;we should build a data lake,&#8221; and someone else will say &#8220;just put it in the warehouse,&#8221; and the deciding argument will be about how much data there is.</p>



<p class="wp-block-paragraph">That argument is almost always the wrong one. This post covers data lake vs data warehouse specifically for CRM analytics: why volume rarely decides it, what actually does, a decision procedure you can run in an afternoon, and where zero-ETL has changed the maths recently enough that older advice is stale.</p>



<h2 class="wp-block-heading">Start with the number, then stop using it</h2>



<p class="wp-block-paragraph">Go and count your rows. Not estimate. Count.</p>



<p class="wp-block-paragraph">A mid-sized company&#8217;s entire Salesforce org is usually a few million rows across the objects anyone actually reports on: accounts, contacts, opportunities, activities, cases. Add HubSpot, Zendesk and a billing system and you are still typically in the low tens of gigabytes once it is written as compressed Parquet. Activity and event history is the one object that can genuinely run large, and it is worth checking separately.</p>



<p class="wp-block-paragraph">Numbers at that scale do not stress either architecture. A data warehouse will not break a sweat. A lake will not break a sweat. The lake-versus-warehouse debate as it is usually framed, the one about petabytes and elasticity and separating storage from compute, is a debate about a scale problem you do not have.</p>



<p class="wp-block-paragraph">So count the rows, confirm you are small, and then stop using volume as an argument. Everything below is about what decides it instead.</p>



<h2 class="wp-block-heading">Data lake vs data warehouse: the three options, honestly</h2>



<h3 class="wp-block-heading">Warehouse only</h3>



<p class="wp-block-paragraph">Load CRM objects straight into Redshift, Snowflake, BigQuery or Postgres. Model with SQL. Point the BI tool at it.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> it is the fastest path to an analyst being productive, and that is not a small thing. Query performance on joins is better than a lake gives you without tuning. Access control is mature and row-level. Every BI tool connects natively. The people who will maintain this already know SQL and probably do not know Spark.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> semi-structured data is awkward, and CRM payloads are full of it. Custom field bags, JSON blobs, email bodies, call transcripts. You can store JSON in a modern warehouse but you will not enjoy querying it at any depth. Storing many years of daily snapshots gets expensive on warehouse storage in a way it does not on object storage. And you are more locked in: moving between warehouses is a migration project, not a repoint.</p>



<h3 class="wp-block-heading">Lake only</h3>



<p class="wp-block-paragraph">Land raw API responses to S3, catalogue them, query with Athena or Spark. Iceberg or Delta as the table format if you want anything resembling a database.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> raw retention is nearly free, so keeping every version of every record for years is a storage line item rather than a decision. Schema evolution is genuinely easier: when a SaaS admin adds a field, the lake absorbs it. Unstructured content lives alongside structured tables, which matters more now that people want to run models over call notes and support tickets. And an open table format means the data is not hostage to one query engine.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> you have taken on work a warehouse does for you. Small-file compaction, partitioning, table maintenance, catalogue hygiene. Query latency on interactive dashboards is worse unless you invest in layout. Governance is a build, not a setting. And the per-query cost model punishes exactly the workload CRM analytics generates: lots of small, repeated, unpredictable queries from a BI tool. Athena bills on data scanned with a per-query minimum, which means a dashboard refreshing thirty tiles is thirty billable scans whether or not anyone is looking.</p>



<h3 class="wp-block-heading">Lakehouse</h3>



<p class="wp-block-paragraph">Raw and historical data in object storage as Iceberg tables; curated marts in the warehouse or queried in place by a warehouse engine. This is where most serious setups end up, and it is what AWS, Databricks and Snowflake have all converged on.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> you get cheap deep history and expensive fast queries in the right places. Raw payloads stay replayable when a vendor changes a field type. Analysts still get warehouse-shaped tables.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> it is two systems. Two cost models, two sets of permissions, two places a table can be defined and drift. For a five-person company with one CRM, this is architecture as procrastination. The complexity is real and it should be earning something.</p>



<h2 class="wp-block-heading">What actually decides it</h2>



<p class="wp-block-paragraph">Four levers, in the order they usually matter for CRM data.</p>



<h3 class="wp-block-heading">1. How much history you need, and whether deletes matter</h3>



<p class="wp-block-paragraph">This is the big one and it is the one people skip.</p>



<p class="wp-block-paragraph">CRM systems mutate in place. An opportunity&#8217;s stage, amount and close date are overwritten as the deal moves. If you only ever sync current state, you can report on the pipeline as it is today and you cannot report on what it looked like last March. Every question about pipeline movement, forecast accuracy, stage velocity or &#8220;what changed since the QBR&#8221; needs point-in-time history, and the CRM will not give it to you retroactively. Deleted records are worse: they are simply gone.</p>



<p class="wp-block-paragraph">If you need history, you need somewhere cheap to keep daily snapshots or a change feed. That pushes toward object storage, or toward a warehouse with a genuinely cheap storage tier. If a rolling twelve months of current-state data covers every question anyone asks, the warehouse handles it alone.</p>



<h3 class="wp-block-heading">2. How fast the schema churns</h3>



<p class="wp-block-paragraph">Count the custom fields added to your main CRM object in the last six months. If it is two, schema churn is not your problem. If it is forty, and half of them are from a business unit you do not talk to, a rigid warehouse schema becomes a standing maintenance tax and the lake&#8217;s tolerance for drift starts paying for itself.</p>



<h3 class="wp-block-heading">3. The shape of the queries</h3>



<p class="wp-block-paragraph">Scheduled dashboards hitting the same tables repeatedly favour a warehouse, because per-second compute amortises across many queries and results are cached. Redshift Serverless, for example, bills in processing-unit-hours per second with a short minimum and shuts down when idle, which suits bursty BI traffic.</p>



<p class="wp-block-paragraph">Exploratory one-off analysis across wide, rarely-touched history favours a lake, because you only pay when someone asks. Most CRM analytics is overwhelmingly the first kind. That fact alone settles a lot of these arguments.</p>



<h3 class="wp-block-heading">4. Who is going to maintain it</h3>



<p class="wp-block-paragraph">The least technical criterion and often the most decisive. If the answer is &#8220;a RevOps analyst who is very good at SQL and has no platform team behind them,&#8221; build the warehouse. A lake handed to someone who cannot maintain it becomes an unqueryable pile of Parquet in about eight months. That is not a hypothetical failure mode; it is the normal one.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">A decision procedure you can run this afternoon</h2>



<ol class="wp-block-list">
<li><strong>Count rows per object</strong> for everything anyone reports on. Note which objects are over about ten million rows. Usually it is activities and nothing else.</li>

<li><strong>Write down the ten questions</strong> the business actually asks. Not the ones you imagine. Mark each one as needing current state or point-in-time history.</li>

<li><strong>Count schema changes</strong> on your main objects over the last six months.</li>

<li><strong>Estimate query volume:</strong> number of dashboards, tiles per dashboard, refresh frequency, plus rough ad-hoc queries per week. This is the number that drives cost, not storage.</li>

<li><strong>Model both costs against those numbers.</strong> For the lake: bytes scanned per query times queries per month. For the warehouse: compute-hours per month plus storage. Do it on a napkin. At CRM scale the answer is usually obvious and usually favours the warehouse.</li>

<li><strong>Name the maintainer.</strong> If you cannot name a person, pick the option that needs less of them.</li>
</ol>



<p class="wp-block-paragraph">Read together, most CRM-only shops land on: warehouse for the curated layer, cheap object storage for raw history, and no ambition beyond that until something forces it. That is a lakehouse, but the small honest kind, and it is a very different commitment from standing up a full lake platform.</p>



<h2 class="wp-block-heading">Where zero-ETL changed the maths</h2>



<p class="wp-block-paragraph">A lot of writing on this topic predates a genuine shift, so it is worth stating plainly.</p>



<p class="wp-block-paragraph">Managed zero-ETL integrations now replicate CRM and support platforms into AWS targets without you building a pipeline. AWS Glue supports SaaS sources including Salesforce, SAP, ServiceNow, Zendesk and Zoho CRM, landing into Amazon Redshift, the SageMaker lakehouse, or S3 Tables with Iceberg underneath. Salesforce has pushed the other direction with zero-copy federation, letting Data Cloud read Iceberg tables in Databricks, Snowflake, BigQuery or Redshift in place, and letting those platforms read Data Cloud objects back.</p>



<p class="wp-block-paragraph">Two consequences for this decision:</p>



<ul class="wp-block-list">
<li><strong>Ingestion is no longer the differentiator.</strong> &#8220;The lake is easier to load into&#8221; was a real argument when you were hand-rolling connectors. It is much weaker when the same managed integration targets either destination.</li>

<li><strong>Iceberg makes the choice less permanent.</strong> If your raw layer is Iceberg tables in object storage, adding a warehouse engine later is a configuration change rather than a migration. That is a genuine reason to prefer open table formats even if you never build a &#8220;lake&#8221; as such.</li>
</ul>



<p class="wp-block-paragraph">Two honest caveats. Zero-ETL integrations cover a curated list of sources and regions, so check yours before designing around them. And a managed replica is still current-state replication plus change capture, not a modelled history; you still have to decide how snapshots and slowly changing dimensions get built.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<ul class="wp-block-list">
<li><strong>&#8220;We need a lake because we&#8217;ll do AI later.&#8221;</strong> Reasonable in spirit, wrong in sequence. Build the thing that answers this quarter&#8217;s questions, keep raw payloads in object storage so you have not destroyed your options, and revisit. Raw retention is the cheap insurance; a lake platform is not.</li>

<li><strong>&#8220;Storage is cheaper in a lake.&#8221;</strong> True and almost always irrelevant. At CRM volumes, storage is a rounding error next to compute and, more to the point, next to the salary of whoever maintains the thing.</li>

<li><strong>&#8220;The warehouse is a single point of vendor lock-in.&#8221;</strong> Fair, and it is why open table formats matter. But if you replicate raw data to object storage, the lock-in is on your transformation logic, not your data, and that is portable if you keep it in SQL and version control.</li>

<li><strong>&#8220;We&#8217;ll just query the CRM API directly and skip both.&#8221;</strong> Works until a dashboard fans out into rate limits and the whole thing becomes a data pipeline you did not design. If more than one person is asking questions, you need a copy.</li>

<li><strong>&#8220;Let&#8217;s do both properly from day one.&#8221;</strong> This is the expensive mistake. Two systems, two cost models, and tables that quietly disagree, in service of a workload that fits comfortably in one.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is a data lake overkill for one CRM?</h3>



<p class="wp-block-paragraph">For a single CRM with current-state reporting needs, usually yes. The point at which it stops being overkill is when you need years of point-in-time history, when unstructured content like call transcripts becomes part of the analysis, or when you are integrating enough systems that a common raw layer is cheaper than a dozen warehouse loaders.</p>



<h3 class="wp-block-heading">How do I get historical CRM snapshots if I only have current state?</h3>



<p class="wp-block-paragraph">You mostly cannot get it retroactively, which is why this decision is urgent rather than academic. Start capturing now: either daily full snapshots to partitioned object storage, or a change feed if the platform offers one, then build slowly changing dimension tables from that. Field history tracking inside the CRM covers a limited number of fields and is not a substitute.</p>



<h3 class="wp-block-heading">Does the lakehouse make this comparison obsolete?</h3>



<p class="wp-block-paragraph">It blurs the storage layer, not the operational reality. You still choose a query engine, a cost model, a governance model and a maintainer. The useful question has shifted from &#8220;lake or warehouse&#8221; to &#8220;how much of this do I want to operate myself,&#8221; which is a better question anyway.</p>



<h3 class="wp-block-heading">Where does reverse ETL fit?</h3>



<p class="wp-block-paragraph">If you plan to push computed values back into the CRM, such as a lead score or account health field, that pushes toward the warehouse. Reverse ETL tools connect to warehouses far more readily than to query engines over object storage, and the write path wants a stable, modelled table rather than a raw one.</p>



<h3 class="wp-block-heading">Can I start with one and move later?</h3>



<p class="wp-block-paragraph">Moving from lake to warehouse is straightforward: you are loading modelled tables into a new engine. Moving from warehouse to lake is harder, because the history only exists inside the warehouse and the transformations are written in its dialect. The asymmetric fix is to land raw data in object storage regardless of which you query from. That single habit preserves the option cheaply.</p>



<h3 class="wp-block-heading">What about just using the CRM&#8217;s own analytics product?</h3>



<p class="wp-block-paragraph">Worth pricing seriously before building anything. It is genuinely the right answer when all your questions are about one system, and it removes an entire pipeline from your life. It stops being the right answer the moment a question spans CRM plus billing plus product usage, because you will be paying to ingest the other systems into a platform priced for CRM data.</p>



<h2 class="wp-block-heading">The short version</h2>



<p class="wp-block-paragraph">In the data lake vs data warehouse question for CRM analytics, volume is the argument everyone reaches for and it is the one that matters least. Your CRM data is small. Both options handle it.</p>



<p class="wp-block-paragraph">What decides it is how much history you need, how fast the schema moves, what shape the queries are, and who is going to keep it running. Answer those four and the architecture falls out on its own. And whatever you pick, land the raw payloads in object storage from day one, because that is the cheap decision that keeps every other decision reversible.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second opinion on your CRM analytics stack?</h2>



<p class="wp-block-paragraph">I help teams make this call without over-building, and unpick it when it has already been over-built. Things I can help with:</p>



<ul class="wp-block-list">
<li>Running the sizing and cost model above against your real query volume, not a vendor calculator.</li>

<li>Designing point-in-time history capture for CRM objects, including deletes and slowly changing dimensions.</li>

<li>Setting up zero-ETL or connector-based ingestion from Salesforce, HubSpot, Zendesk or Zoho into Redshift, Snowflake or an Iceberg lake.</li>

<li>Migrating an unmaintainable pile of Parquet into something an analyst can actually query.</li>

<li>Cutting query costs on an existing setup: partitioning, table layout, dashboard refresh patterns.</li>

<li>Building the curated CRM models that BI tools sit on, in SQL and in version control.</li>
</ul>



<p class="wp-block-paragraph">If you are mid-argument about this, send me your row counts and the ten questions the business wants answered. That is usually enough to settle it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/data-lake-vs-data-warehouse-crm-analytics/">Data Lake vs Data Warehouse for CRM Analytics: Volume Is the Wrong Question</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
