<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Change Data Capture | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/change-data-capture/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/change-data-capture/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sun, 08 Nov 2026 07:05:47 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Change Data Capture | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/change-data-capture/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Oracle to Aurora PostgreSQL: A Migration Plan That Survives Contact</title>
		<link>https://john-nessime.com/blog/technical-guides/oracle-to-aurora-postgresql-migration-plan/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 08 Nov 2026 07:05:47 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Databases]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Aurora]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS DMS]]></category>
		<category><![CDATA[AWS SCT]]></category>
		<category><![CDATA[Change Data Capture]]></category>
		<category><![CDATA[Cutover Planning]]></category>
		<category><![CDATA[Database Migration]]></category>
		<category><![CDATA[Migration Assessment]]></category>
		<category><![CDATA[ora2pg]]></category>
		<category><![CDATA[Oracle]]></category>
		<category><![CDATA[orafce]]></category>
		<category><![CDATA[PL/SQL]]></category>
		<category><![CDATA[PostgreSQL]]></category>
		<category><![CDATA[Query Plan Management]]></category>
		<category><![CDATA[Rollback Plan]]></category>
		<category><![CDATA[Schema Conversion]]></category>
		<category><![CDATA[Supplemental Logging]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=498</guid>

					<description><![CDATA[<p>The assessment report says most objects convert automatically, and that number becomes a date on a roadmap. It measures objects, not work. Here are the six decisions that actually set the timeline of an Oracle to Aurora PostgreSQL migration, a phase sequence gated on artefacts rather than dates, and the places estimates quietly go soft.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/oracle-to-aurora-postgresql-migration-plan/">Oracle to Aurora PostgreSQL: A Migration Plan That Survives Contact</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The date was on a slide before anyone had opened the converter.</p>



<p class="wp-block-paragraph">That is usually how it happens. A director asks how long it takes to get off Oracle, somebody runs the assessment, the report says most objects convert automatically, and that number becomes a quarter on a roadmap. Nobody lied. The number is real. It just does not measure the thing the schedule depends on.</p>



<p class="wp-block-paragraph">Here is the part that bites, and it stays invisible until about the second month of testing: the conversion percentage counts objects, not work. The objects that fail to convert are the safe ones. They get flagged, scoped and estimated. The objects that convert cleanly are where the risk lives, because a converted function compiles, runs, returns a value, and can still be quietly wrong in a way nobody notices until a finance user queries a total.</p>



<p class="wp-block-paragraph">This post is about planning around that. It covers the six decisions that set the timeline, a phase sequence that reflects how the work really lands, and the specific places an Oracle to Aurora PostgreSQL migration plan goes soft. It is not a console walkthrough. AWS documents the clicking well enough.</p>



<h2 class="wp-block-heading">Start from what the assessment report does not tell you</h2>



<p class="wp-block-paragraph">There are two supported ways to convert an Oracle schema for a PostgreSQL target. DMS Schema Conversion runs in the AWS console and CLI as a managed feature. AWS SCT is the older Java desktop application you install yourself. They are two front ends over the same conversion engine, and AWS now points most OLTP conversions at the managed one. SCT still earns its place when you want a local tool, want to script the conversion into a pipeline, or hit a path the managed feature does not cover.</p>



<p class="wp-block-paragraph">One planning input that used to exist no longer does. AWS ended support for DMS Fleet Advisor on 20 May 2026. If your last migration playbook opened with &#8220;run Fleet Advisor across the estate to find candidates,&#8221; that step is gone and the discovery work goes back to you, your CMDB, or a script that walks the data dictionary. Worth knowing before you copy an old plan forward.</p>



<p class="wp-block-paragraph">DMS Schema Conversion also has a generative AI option that proposes conversions for code objects it could not handle on its own, and it genuinely shrinks the manual pile. Be honest in the plan about what it changes though: it moves effort from writing PL/pgSQL to reviewing PL/pgSQL. Review is cheaper than authoring. It is not free, and it still needs someone who knows what the original code was meant to do.</p>



<p class="wp-block-paragraph">What no assessment report measures:</p>



<ul class="wp-block-list">
<li>Behavioural divergence in the objects that converted successfully</li>



<li>Query performance on the target, which is a different optimiser making different choices</li>



<li>SQL that lives outside the database, in application code, report definitions and shell scripts</li>



<li>Whether your source tables can support change data capture at all</li>



<li>How long it takes to get a production-shaped dataset into a test environment</li>
</ul>



<h2 class="wp-block-heading">Decision one: what moves, what waits, and what should never move</h2>



<p class="wp-block-paragraph">Not every Oracle schema is a migration candidate, and finding that out in month four is expensive.</p>



<p class="wp-block-paragraph">Vendor applications are the clearest case. If the vendor certifies Oracle and only Oracle, you are not planning a database migration, you are planning an application replacement, and that is a different project with a different budget. Decide that first and take those schemas off the board.</p>



<p class="wp-block-paragraph">The second filter is mechanical. DMS needs a primary key or unique constraint on a table to replicate changes to it. The premigration assessment flags tables that lack one, along with LOB columns that are not nullable on the target and Oracle-specific structures that complicate capture: index-organized tables, function-based indexes, global temporary tables, anything leaning on ROWID. Run it early, not as a pre-flight check the week before the load.</p>



<p class="wp-block-paragraph">You can find the primary key problem yourself in a minute, and it is worth doing before anyone commits to a date:</p>



<pre class="wp-block-code"><code>-- Tables in scope with no primary key and no unique constraint.
-- These cannot participate in DMS change data capture as-is.
SELECT t.owner, t.table_name
FROM   all_tables t
WHERE  t.owner IN ('HR', 'SALES', 'BILLING')
AND    NOT EXISTS (
         SELECT 1
         FROM   all_constraints c
         WHERE  c.owner = t.owner
         AND    c.table_name = t.table_name
         AND    c.constraint_type IN ('P', 'U'))
ORDER  BY t.owner, t.table_name;</code></pre>



<p class="wp-block-paragraph">Every row that comes back is a decision. Add a key to the source table, which is a production change on a table that may be very large. Migrate it full load only and freeze writes to it during cutover. Or leave it behind. All three are acceptable. Discovering the list after you have promised a cutover weekend is not.</p>



<h2 class="wp-block-heading">Decision two: how much Oracle you keep inside PostgreSQL</h2>



<p class="wp-block-paragraph">PostgreSQL has an extension called orafce that implements a subset of Oracle functions and packages. It is available on Aurora PostgreSQL and RDS for PostgreSQL, and the conversion tooling knows about it. There are also conversion settings that emulate Oracle behaviour directly, such as standing in for the ROWID pseudocolumn with a bigint or character column, and emulating format parameters that PostgreSQL&#8217;s own <code>TO_CHAR</code> and <code>TO_DATE</code> do not accept.</p>



<p class="wp-block-paragraph">The case for turning it all on is strong. Compatibility shims delete whole categories of rewrite. Oracle&#8217;s <code>SUBSTR</code> behaves differently from PostgreSQL&#8217;s on certain inputs, and orafce gives you the Oracle behaviour without auditing every call site. If the goal is to be off Oracle licensing by a fixed date, that trade is often correct.</p>



<p class="wp-block-paragraph">The cost is that every shim is one more thing your team has to remember is not stock PostgreSQL. Two years later somebody writes a query on a laptop against community Postgres, it behaves differently in production, and nobody can explain why. You did not leave Oracle. You brought a piece of it along and stopped labelling it.</p>



<p class="wp-block-paragraph">How I would decide: turn on compatibility where it removes an entire class of rewrite you would otherwise do by hand, and turn it off where it preserves an Oracle idiom you were specifically trying to escape. Package-level session state is the obvious one to let go of. String semantics are the obvious one to keep.</p>



<p class="wp-block-paragraph">The single divergence with the widest blast radius does not have a clean shim, and it deserves its own line item on the plan. Oracle treats an empty string as NULL. PostgreSQL does not.</p>



<pre class="wp-block-code"><code>-- Oracle: these two return the same rows.
SELECT * FROM customers WHERE notes IS NULL;
SELECT * FROM customers WHERE notes = '';

-- PostgreSQL: the second returns nothing.
-- '' is a zero-length string here, and NULL is absence of a value.
SELECT * FROM customers WHERE notes IS NULL;
SELECT * FROM customers WHERE notes = '';</code></pre>



<p class="wp-block-paragraph">That affects WHERE clauses, concatenation, unique constraints and any application code that inserts <code>''</code> expecting it to land as NULL. It is not a bug you fix once. It is an audit you schedule.</p>



<h2 class="wp-block-heading">Decision three: who rewrites the PL/SQL and how you prove it behaves</h2>



<p class="wp-block-paragraph">This is the decision that dominates the schedule, and it is usually the one with the least detail on the plan.</p>



<p class="wp-block-paragraph">Converted PL/pgSQL compiles. That is the trap. Compiling proves the syntax is legal, not that the logic survived. The cleanest example is <code>SELECT INTO</code>, which is used in nearly every Oracle procedure ever written.</p>



<pre class="wp-block-code"><code>-- PL/pgSQL: if no row matches, v_name is simply left NULL.
-- If several match, you get one of them. No exception either way.
SELECT name INTO v_name FROM customers WHERE id = p_id;

-- With STRICT, you get NO_DATA_FOUND and TOO_MANY_ROWS
-- the way Oracle raises them implicitly.
SELECT name INTO STRICT v_name FROM customers WHERE id = p_id;</code></pre>



<p class="wp-block-paragraph">An Oracle procedure that relied on <code>NO_DATA_FOUND</code> to branch now takes the happy path with a NULL in hand. It will not error. It will post a record with a missing field, skip a validation, or return zero where it should have raised. Multiply that by a few hundred procedures.</p>



<p class="wp-block-paragraph">Transaction handling is the other place assumptions break. Oracle commits DDL implicitly; PostgreSQL runs DDL inside transactions and lets you roll it back. Autonomous transactions, which Oracle code often uses to write an audit row that survives a rollback, have no direct equivalent. Both are fine once you know about them, and both silently change behaviour if you do not.</p>



<p class="wp-block-paragraph">The only test that settles this is differential: same inputs, both engines, compare outputs. Not &#8220;does the procedure run.&#8221; Capture real inputs from production before the rewrite starts, because reconstructing them from memory afterwards is how you end up testing the happy path only.</p>



<p class="wp-block-paragraph">The staffing reality underneath all of this: the people who know what those packages are supposed to do are usually Oracle developers who have maintained them for a decade, and the people fluent in PostgreSQL are usually somebody else. That gap is the real cost of the migration, and no tool closes it.</p>



<h2 class="wp-block-heading">Decision four: what you are asking the source database to give up</h2>



<p class="wp-block-paragraph">Change data capture is not something you switch on in DMS. It is a change to production Oracle that has to happen weeks before anyone migrates anything, and it needs its own milestone and its own change window.</p>



<p class="wp-block-paragraph">The source has to be in ARCHIVELOG mode. It needs supplemental logging, minimally at the database level, plus identification key logging so that updates carry enough of the row to be replayed. AWS notes that retaining archive logs for around 24 hours is usually enough when a task starts capturing immediately, and longer if you plan to start from a point in the past.</p>



<pre class="wp-block-code"><code>-- Minimal supplemental logging at the database level.
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;

-- All-column logging for a single replicated table, if you
-- would rather not carry the overhead database-wide.
ALTER TABLE hr.employees ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;</code></pre>



<p class="wp-block-paragraph">You can also let DMS add table-level supplemental logging itself with the <code>addSupplementalLogging=Y</code> extra connection attribute, which requires the DMS user to hold ALTER on the replicated tables. That is convenient and it is also a DBA conversation, because it means a migration tool is issuing DDL against production.</p>



<p class="wp-block-paragraph">Two planning consequences fall out of this. Supplemental logging increases redo volume, which lands on backup windows and archive storage on a system that was sized before anyone mentioned Aurora. And LogMiner, which DMS uses to read redo by default, does not support connecting to a pluggable database. If the source is multitenant, you are on Binary Reader, which reads the log files directly, needs different privileges, and behaves differently around ASM and around backups that clear archive logs early.</p>



<p class="wp-block-paragraph">None of that is hard. All of it takes calendar time you have to ask another team for.</p>



<h2 class="wp-block-heading">Decision five: what &#8220;fast enough&#8221; means, decided before cutover</h2>



<p class="wp-block-paragraph">Aurora PostgreSQL&#8217;s planner is not Oracle&#8217;s optimiser. A query that ran in single-digit milliseconds for years can pick a different join order and take minutes, and it will do that on the queries you did not think to test.</p>



<p class="wp-block-paragraph">The first thing to get right is trivial and gets missed constantly. After a DMS full load, the target tables have no statistics, and plans built on no statistics are close to random. Run <code>ANALYZE</code> before you measure anything, or you will spend a week tuning a problem that does not exist.</p>



<pre class="wp-block-code"><code>-- After the full load completes, before any performance testing.
ANALYZE VERBOSE;

-- Query plan management is enabled at the cluster level first.
SHOW rds.enable_plan_management;</code></pre>



<p class="wp-block-paragraph">Aurora has a feature Oracle people will recognise: query plan management, packaged as the <code>apg_plan_mgmt</code> extension. It captures plans as the optimiser generates them, stores them in <code>apg_plan_mgmt.dba_plans</code>, and lets you approve, reject or force specific ones. New plans arrive as unapproved, and <code>apg_plan_mgmt.evolve_plan_baselines</code> compares them on real execution time so you can promote the good ones. Set <code>apg_plan_mgmt.capture_plan_baselines</code> to collect, and <code>apg_plan_mgmt.use_plan_baselines</code> to enforce.</p>



<p class="wp-block-paragraph">It is the closest thing to Oracle&#8217;s plan stability and it is genuinely useful, but be clear about what it does. It stabilises plans you have already decided are acceptable. It does not find the slow queries for you. For that you need a workload replay against the target and a baseline captured from Oracle before you started.</p>



<p class="wp-block-paragraph">Whatever you already use for database monitoring works here, whether that is CloudWatch and Performance Insights, Grafana against Prometheus exporters, Percona Monitoring and Management, or Datadog. The requirement is not a particular tool. It is that you took the Oracle baseline before you touched anything, because you cannot go back and capture it later.</p>



<h2 class="wp-block-heading">Decision six: the cutover and the way back</h2>



<p class="wp-block-paragraph">Decide the rollback story during planning, because one of the two options takes weeks to build.</p>



<p class="wp-block-paragraph">A one-way door is cheaper and perfectly legitimate for a system where a few hours of degraded service is survivable. You cut over, freeze the source, and fix forward. It just requires saying out loud that there is no going back once the first write lands on Aurora.</p>



<p class="wp-block-paragraph">Reverse replication is the alternative: a second DMS task running Aurora back to Oracle so the old database stays current and you can repoint the application. It is the only real rollback once production writes are on the new engine, and it is also a full migration in the opposite direction with its own type mappings, key requirements and testing. Budget it as a work stream, not a checkbox. The failure mode I see most often is a plan that assumes reverse replication and funds a one-way door.</p>



<h2 class="wp-block-heading">A realistic Oracle to Aurora PostgreSQL migration plan, phase by phase</h2>



<p class="wp-block-paragraph">The ordering matters more than the durations, which vary enormously with how much PL/SQL you have. Gate each phase on an artefact rather than a date.</p>



<ol class="wp-block-list">
<li><strong>Inventory and disposition.</strong> Every schema gets a decision: migrate, replace, retire, or leave. Gate: a signed list.</li>



<li><strong>Target shape and compatibility.</strong> Aurora or RDS, which shims are on, which Oracle idioms you are deliberately keeping. Gate: a written decision with reasons, because someone will ask in month five.</li>



<li><strong>Convert and apply to a scratch cluster.</strong> First contact with reality. Gate: the schema applies without manual patching.</li>



<li><strong>PL/SQL rewrite and differential testing.</strong> The longest phase by a wide margin. It runs in parallel with everything below it. Gate: input and output pairs matching across both engines.</li>



<li><strong>Source enablement.</strong> ARCHIVELOG, supplemental logging, DMS user privileges, log retention. This is a production change request. Gate: capture proven working on a single table.</li>



<li><strong>Timed full load rehearsal.</strong> You need the real duration, not an estimate, because it sets the shape of the cutover window. Gate: a measured number.</li>



<li><strong>CDC steady state and validation.</strong> Let it run for days, not hours. Gate: lag stable under production write volume.</li>



<li><strong>Performance parity.</strong> Replay the workload, fix what is slow, capture plans. Gate: the slow list is empty or explicitly accepted.</li>



<li><strong>Cutover rehearsal, twice.</strong> Once to find the gaps, once to prove you fixed them. Gate: a runbook someone else could execute.</li>



<li><strong>Cutover.</strong> Boring, if the nine steps above were not.</li>
</ol>



<p class="wp-block-paragraph">Phase four is the one that decides your date. Everything else is reasonably predictable once you have measured it. Rewriting and proving business logic is not, because the estimate depends on code nobody has read in years.</p>



<h2 class="wp-block-heading">Where these plans actually slip</h2>



<ul class="wp-block-list">
<li><strong>SQL outside the database.</strong> Query strings embedded in application code, report definitions in a BI tool, ad hoc scripts on a scheduler. None of it appears in a schema assessment and all of it has to be found. Routinely the biggest underestimate on the plan.</li>



<li><strong>Adding keys to source tables.</strong> Putting a primary key on a very large production table so CDC can replicate it is its own change, its own maintenance window and its own risk.</li>



<li><strong>Test data.</strong> Getting production-shaped volume into a non-production environment is slow, and if the data is regulated you have to build masking first. A project inside the project.</li>



<li><strong>Running two databases at once.</strong> For the whole parallel period you pay for Oracle, for Aurora, and for the replication capacity between them. Model that before it becomes a surprise in a finance review.</li>



<li><strong>Waiting on other teams.</strong> The supplemental logging change window, the network path, the security review of the DMS user&#8217;s privileges. Each one is short and none of them start when you ask.</li>



<li><strong>Environment sprawl.</strong> The conversion tooling, the replay harness and the comparison scripts all want somewhere persistent to live, and a laptop is not it. A modest dedicated box from a provider like InterServer or Contabo, or a small always-on EC2 instance, saves a lot of &#8220;it worked on mine&#8221; over a long project.</li>
</ul>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<h3 class="wp-block-heading">&#8220;The tool converted almost everything, so we&#8217;re almost done&#8221;</h3>



<p class="wp-block-paragraph">Object counts and effort are not proportional. A thousand trivially converted tables and one 4,000-line billing package are not a 1000-to-1 ratio of work. The leftovers are also the objects most tangled up in business rules, which is exactly why the tool could not convert them.</p>



<h3 class="wp-block-heading">&#8220;We&#8217;ll tune the slow queries after go-live&#8221;</h3>



<p class="wp-block-paragraph">You can, and sometimes that is the right call. Just understand the trade: after the first production write lands on Aurora, the rollback window is closed unless you built reverse replication. Deferring performance work means accepting whatever you find with no way back.</p>



<h3 class="wp-block-heading">&#8220;We can skip a rehearsal because the load is repeatable&#8221;</h3>



<p class="wp-block-paragraph">The full load is repeatable. The cutover is not. The rehearsal tests the sequence around the load: stopping application writes, draining lag to zero, resetting sequences, re-enabling triggers and foreign keys, repointing connection strings, confirming the first write landed. That sequence is where the time goes and where the mistakes happen.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How long does an Oracle to Aurora PostgreSQL migration take?</h3>



<p class="wp-block-paragraph">The honest answer is that the data movement is the predictable part and the code is not. A schema with few stored procedures and a well-understood application can move in weeks. A system with hundreds of thousands of lines of PL/SQL encoding two decades of business rules is a multi-quarter programme. Estimate from the volume of procedural code and the availability of people who understand it, not from the object count in the assessment report.</p>



<h3 class="wp-block-heading">Should I target Aurora PostgreSQL or RDS for PostgreSQL?</h3>



<p class="wp-block-paragraph">Aurora separates compute from a shared storage layer, which gives faster replica creation and failover, and it carries AWS-specific extensions including query plan management. RDS for PostgreSQL is closer to community PostgreSQL and simpler to reason about. For a workload coming off Oracle with plan-stability concerns and high availability requirements, Aurora usually fits better. For a small, steady workload where portability matters more than failover speed, RDS is a reasonable answer and easier to leave later.</p>



<h3 class="wp-block-heading">Do I still need AWS SCT, or is DMS Schema Conversion enough?</h3>



<p class="wp-block-paragraph">For a standard Oracle to PostgreSQL OLTP conversion, the managed DMS Schema Conversion feature is the default path and what AWS points you at. Reach for the SCT desktop application when you want the conversion scripted into a repeatable pipeline, need a feature the managed version does not expose, or hit a conversion path it does not cover.</p>



<h3 class="wp-block-heading">What replaced DMS Fleet Advisor?</h3>



<p class="wp-block-paragraph">Nothing directly. Support ended on 20 May 2026, and AWS did not ship a like-for-like successor inside DMS. Fleet-level discovery now comes from whatever you already have: an existing CMDB, AWS Application Discovery Service for the server layer, or scripts that query each database&#8217;s data dictionary for object counts and feature usage. If you are migrating one or two databases rather than an estate, you were probably not using it anyway.</p>



<h3 class="wp-block-heading">Can I migrate from Oracle to Aurora with no downtime at all?</h3>



<p class="wp-block-paragraph">Near-zero rather than zero. Full load plus CDC keeps the target current while the source stays live, so the outage shrinks to the moment you stop writes, let lag reach zero and repoint the application. That is typically minutes. What stretches it is sequence resets, re-enabling constraints and triggers, and connection pools holding the old endpoint longer than expected.</p>



<h3 class="wp-block-heading">Is ora2pg a real alternative to the AWS tooling?</h3>



<p class="wp-block-paragraph">Yes, and it is worth running even if you plan to use AWS tools. ora2pg is a mature open source converter that handles schema and PL/SQL and produces its own migration assessment. Teams that run both tend to find each catches things the other missed, and the second assessment costs an afternoon. Where AWS tooling wins is integration with DMS for the data movement itself.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A conversion percentage measures objects. Your schedule is set by behaviour.</p>



<p class="wp-block-paragraph">The code that failed to convert is already on someone&#8217;s list. The code that converted cleanly is not, and that is where the empty strings, the missing exceptions and the different plans are hiding. An Oracle to Aurora PostgreSQL migration plan that budgets for proving behaviour, rather than for converting objects, is the one that lands on its date.</p>



<p class="wp-block-paragraph">If you take one action from this post, run the primary key query above against your real schemas this week. It takes a minute and it will tell you more about your cutover shape than the assessment report will.</p>



<h2 class="wp-block-heading">Working on an Oracle to Aurora PostgreSQL migration?</h2>



<p class="wp-block-paragraph">This is the kind of work I do as a freelance DevOps and cloud engineer. Specifically, on migrations like this one:</p>



<ul class="wp-block-list">
<li>Turning an assessment report into a phased plan with realistic gates, including the schemas that should not move at all</li>



<li>Setting up DMS full load and CDC properly: supplemental logging, LogMiner versus Binary Reader, LOB handling, and premigration assessments run early rather than late</li>



<li>Building the differential test harness that proves converted PL/pgSQL behaves like the original, instead of just compiling</li>



<li>Performance parity work on Aurora PostgreSQL: statistics, plan capture with query plan management, and finding the queries the optimiser now handles differently</li>



<li>Writing and rehearsing the cutover runbook, including the sequence resets and constraint re-enablement that get forgotten</li>



<li>Deciding whether reverse replication is worth building for your risk profile, and building it if it is</li>
</ul>



<p class="wp-block-paragraph">If you are in the middle of one of these, send me something concrete: your assessment report summary, a DMS task log, or the output of the primary key query above. I would rather look at your actual situation than talk in general terms.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/oracle-to-aurora-postgresql-migration-plan/">Oracle to Aurora PostgreSQL: A Migration Plan That Survives Contact</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a Hotel Data Lake on AWS That Agrees With the Night Audit</title>
		<link>https://john-nessime.com/blog/cloud-computing/hotel-data-lake-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 24 Sep 2026 06:00:06 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Hospitality Technology]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Bitemporal Modeling]]></category>
		<category><![CDATA[Booking Systems]]></category>
		<category><![CDATA[Change Data Capture]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Modeling]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Hotel PMS]]></category>
		<category><![CDATA[Lake Formation]]></category>
		<category><![CDATA[Late-Arriving Data]]></category>
		<category><![CDATA[Medallion Architecture]]></category>
		<category><![CDATA[OPERA Cloud]]></category>
		<category><![CDATA[PCI DSS]]></category>
		<category><![CDATA[PII Redaction]]></category>
		<category><![CDATA[Reconciliation]]></category>
		<category><![CDATA[RevPAR]]></category>
		<category><![CDATA[Table Compaction]]></category>
		<category><![CDATA[Travel Technology]]></category>
		<category><![CDATA[USALI]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=461</guid>

					<description><![CDATA[<p>Hotel source data is mutable in the past, so an append-only pipeline drifts away from the PMS without anyone noticing until month close. A practical guide to room-night grain, bitemporal modeling with Apache Iceberg, PMS ingestion, guest data scope, and file physics at hotel volumes.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/hotel-data-lake-aws/">Building a Hotel Data Lake on AWS That Agrees With the Night Audit</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone opens the manager&#8217;s flash report on one screen and the shiny new dashboard on the other. The RevPAR figures don&#8217;t match. Not by a rounding error either, by enough that the revenue meeting stops being about revenue and turns into an argument about which system is lying.</p>



<p class="wp-block-paragraph">The dashboard usually isn&#8217;t lying. It&#8217;s answering a slightly different question, and the cause is almost always the same one: the pipeline recorded what the property management system said on the day it read it, and the PMS has since changed its mind. A reservation got shortened. A rate was corrected. A rebate landed against a folio that closed two weeks ago. The lake never went back to look.</p>



<p class="wp-block-paragraph">This post is about building a <strong>hotel data lake on AWS</strong> that survives that. Not a service tour, because the AWS documentation already covers S3, Glue and Athena better than a blog post can. What it covers instead is the handful of modelling and ingestion decisions that determine whether your numbers reconcile with the night audit six months from now, and what each of those decisions costs you.</p>



<h2 class="wp-block-heading">The failure that shows up three weeks late</h2>



<p class="wp-block-paragraph">Most first attempts at hotel analytics are append-only. Pull yesterday&#8217;s reservations, write Parquet under a folder keyed by the day you pulled it, register the partition, done. Yesterday reconciles perfectly. So does the day after. Confidence goes up.</p>



<p class="wp-block-paragraph">The problem is that hotel source data is mutable in the past, and it stays mutable for a long time. Group blocks get released. Stays get extended at the desk. Adjustments and rebates get posted against nights that already closed. Commission corrections come back from an OTA weeks after checkout. Night audit itself is a rewrite operation: it closes the business date and posts room and tax charges for every occupied room, and if the auditor reruns it or corrects a posting, the figures for that date change after you already stored them.</p>



<p class="wp-block-paragraph">An append-only lake captures a diagonal. For each business date you keep the first thing you ever saw about it, and nothing after. Every individual day looks right on the day you check it. The month-to-date total drifts. Nobody notices until close, and by then you have no idea which nights moved or why.</p>



<p class="wp-block-paragraph">That is the whole reason to be careful about grain and table format. Everything else in a hospitality pipeline is ordinary data engineering.</p>



<h2 class="wp-block-heading">Fix the grain before you fix the pipeline</h2>



<p class="wp-block-paragraph">The core hotel metrics are per-night metrics. Occupancy is rooms sold divided by rooms available. ADR is room revenue divided by rooms sold. RevPAR is room revenue divided by rooms available, which is the same as ADR multiplied by occupancy. All three are defined against a single night.</p>



<p class="wp-block-paragraph">What the PMS hands you is reservations. A reservation spans nights, carries one confirmation number, and may have different rates on different nights of the same stay. If you store reservations and try to compute occupancy from them, you end up writing date-range arithmetic into every query, and two analysts will write it differently.</p>



<h3 class="wp-block-heading">Explode to room-nights once, in the pipeline</h3>



<p class="wp-block-paragraph">The fact table that makes everything else easy has one row per room, per night, per reservation. Stay dates, rate code, market segment, source of business, channel, room type, rate amount and the taxes as separate columns. Every downstream metric becomes a filter and a sum instead of a range join.</p>



<p class="wp-block-paragraph">Do the explosion in the transform layer, not in the BI tool. If a range join lives in a Power BI model or a QuickSight dataset, it exists once per report and drifts once per report.</p>



<h3 class="wp-block-heading">The denominator is a decision, not a fact</h3>



<p class="wp-block-paragraph">&#8220;Rooms available&#8221; sounds like a constant. It isn&#8217;t. Rooms out of order for renovation, rooms out of service for a night, house-use rooms and comp rooms all get treated differently by different properties, and the PMS report you&#8217;re reconciling against has already made a choice.</p>



<p class="wp-block-paragraph">Same question on the numerator. Whether no-show revenue counts as room revenue, whether cancellation fees count, whether package components get split out of the room rate, whether ADR is quoted net or gross of tax. Hotel accounting conventions follow the Uniform System of Accounts for the Lodging Industry, and most PMS vendors document how their own reports treat each case. Read that documentation before you write a single SQL definition.</p>



<p class="wp-block-paragraph">Write the definitions down in the repository next to the transform code. When the lake and the PMS disagree later, the first question is always whether it&#8217;s a bug or a definition, and without written definitions you cannot answer that question quickly.</p>



<h2 class="wp-block-heading">Model restatements with two dates, not one</h2>



<p class="wp-block-paragraph">Every row needs a business date, which is the night the revenue belongs to, and an as-of timestamp, which is when you learned it. Two time axes. That&#8217;s what makes the difference between &#8220;the numbers changed&#8221; and &#8220;here is exactly what changed, when, and by how much&#8221;.</p>



<p class="wp-block-paragraph">There are two ways to store that. Keep every version of every row and let the reader pick a version, or keep the current truth in a table you update in place and keep the history separately. The second is easier to query and easier to explain, and it&#8217;s the one I reach for first. It needs a table format that supports updates and deletes, which in practice on AWS means Apache Iceberg.</p>



<p class="wp-block-paragraph">Iceberg gives you three things that matter here: row-level MERGE, snapshots you can query as of a point in time, and schema evolution without rewriting the dataset. Athena, Glue ETL, EMR and Redshift all read Iceberg tables through the Glue Data Catalog, so you&#8217;re not tying the storage to one engine.</p>



<p class="wp-block-paragraph">A curated room-night table in Athena looks roughly like this. Note the partitioning, which I&#8217;ll come back to.</p>



<pre class="wp-block-code"><code>CREATE TABLE curated.room_night (
  property_code      string,
  business_date      date,
  confirmation_id    string,
  room_number        string,
  room_type_code     string,
  rate_code          string,
  market_segment     string,
  source_of_business string,
  room_revenue       decimal(12,2),
  room_tax           decimal(12,2),
  reservation_status string,
  guest_key          string,
  as_of_ts           timestamp,
  source_system      string
)
PARTITIONED BY (property_code, month(business_date))
LOCATION 's3://example-lake-curated/room_night/'
TBLPROPERTIES ('table_type' = 'ICEBERG');</code></pre>



<p class="wp-block-paragraph">Loading is a MERGE from a staging table holding the latest extract. Matched rows get overwritten with the newer version, unmatched rows get inserted, and rows that vanished from the source get marked rather than deleted.</p>



<pre class="wp-block-code"><code>MERGE INTO curated.room_night AS t
USING staging.room_night_batch AS s
  ON  t.property_code   = s.property_code
  AND t.business_date   = s.business_date
  AND t.confirmation_id = s.confirmation_id
  AND t.room_number     = s.room_number
WHEN MATCHED AND s.as_of_ts &gt; t.as_of_ts THEN
  UPDATE SET
    room_revenue       = s.room_revenue,
    room_tax           = s.room_tax,
    rate_code          = s.rate_code,
    reservation_status = s.reservation_status,
    as_of_ts           = s.as_of_ts
WHEN NOT MATCHED THEN
  INSERT (property_code, business_date, confirmation_id, room_number,
          room_revenue, room_tax, rate_code, reservation_status, as_of_ts)
  VALUES (s.property_code, s.business_date, s.confirmation_id, s.room_number,
          s.room_revenue, s.room_tax, s.rate_code, s.reservation_status, s.as_of_ts);</code></pre>



<p class="wp-block-paragraph">The <code>as_of_ts</code> comparison in the MATCHED clause is doing real work. Extracts arrive out of order more often than you&#8217;d like, particularly when a backfill runs alongside the scheduled job, and without that guard a late-arriving older snapshot overwrites newer truth. It&#8217;s a silent corruption, which is the worst kind.</p>



<h3 class="wp-block-heading">Keep a restatement log, not just the current state</h3>



<p class="wp-block-paragraph">Before the MERGE runs, write the delta out to an append-only table: key, field changed, old value, new value, as-of timestamp. It costs almost nothing and it turns &#8220;the March numbers moved&#8221; into a query that returns rows.</p>



<p class="wp-block-paragraph">You can also lean on Iceberg snapshots and time travel for this, and they&#8217;re useful for spot checks. I still prefer an explicit log, because snapshot retention will eventually expire the snapshot you want, and finance will eventually ask about a night that&#8217;s older than your retention window.</p>



<h2 class="wp-block-heading">Landing the sources without building a point-to-point mess</h2>



<p class="wp-block-paragraph">A property runs more systems than people expect. PMS, POS, channel manager, revenue management, spa and golf, parking, loyalty, and whatever the marketing team bought last year. Land each of them raw and immutable first, then transform. The raw zone is your audit trail and your ability to reprocess without going back to a vendor API.</p>



<h3 class="wp-block-heading">The PMS is the hard one</h3>



<p class="wp-block-paragraph">For OPERA Cloud, the Oracle Hospitality Integration Platform is the supported route. It exposes REST APIs over OAuth, and registration is self-service rather than going through the older partner validation process. Two things to plan for: the API surface is large and reservation, financial and configuration data live in different areas, and access depends on the property&#8217;s licensing, so confirm what the property actually has before you scope the work.</p>



<p class="wp-block-paragraph">Legacy on-premise OPERA is a different world. Integration there goes through the older interface stack and the validation programme, and scheduled report exports to a file drop are frequently the pragmatic answer. Cloud-native systems like Cloudbeds and Mews expose REST APIs directly and are far less work.</p>



<p class="wp-block-paragraph">Reading the PMS database directly, where you can even reach it, is tempting and usually a bad idea. Schema changes arrive with vendor upgrades and nothing tells you in advance, and depending on the contract it can put support out of scope. Check the agreement before you build anything on top of direct database access.</p>



<h3 class="wp-block-heading">The relay box, if you need one</h3>



<p class="wp-block-paragraph">Properties with on-premise systems usually need something that picks up scheduled exports and pushes them to S3. Keep it boring: a small always-on Linux host, a scheduled job, checksum on write, and an SNS or CloudWatch alarm when a file doesn&#8217;t turn up. A cheap VPS from somewhere like Contabo or InterServer does this fine, and it is much easier to hand over at the end of an engagement than a machine sitting under the front desk.</p>



<p class="wp-block-paragraph">For reaching that host without opening inbound ports on a hotel network, an outbound-only tunnel such as Cloudflare Tunnel is a clean fit. If staff are hitting admin interfaces from guest or public Wi-Fi, a consumer VPN client like NordVPN or Surfshark protects that transport, though it is not a substitute for a real network path into the property.</p>



<h3 class="wp-block-heading">Time zones will bite you</h3>



<p class="wp-block-paragraph">The hotel business date is not a calendar day and it is not UTC. It&#8217;s a property-local concept that rolls at night audit, which might be at 2am or 4am depending on the property, and it can be held open when the audit runs late.</p>



<p class="wp-block-paragraph">Store the source system&#8217;s own business date as the partition key. Store event timestamps in UTC alongside it. Never derive the business date by truncating a UTC timestamp, because for a group with properties in several time zones that quietly shifts revenue between nights, and the error is small enough to look like noise.</p>



<h2 class="wp-block-heading">Keep card data out and plan for erasure on day one</h2>



<p class="wp-block-paragraph">Guest records are personal data and hotels hold a lot of it: names, addresses, passport and ID details in some jurisdictions, loyalty identifiers, stay history. Payment card data sits under a separate regime again.</p>



<p class="wp-block-paragraph">The single highest-value decision is negative. Do not bring cardholder data into the analytics platform at all. Nothing in occupancy, ADR or segment analysis needs a card number, and pulling one in drags the whole lake, its query engines and everyone with access into PCI DSS scope. Take the payment token or a masked reference from the PMS and stop there.</p>



<p class="wp-block-paragraph">For guest identity, pseudonymise at ingestion. Keep a surrogate guest key in the analytics tables and hold the mapping to real identity in one small, separately governed place with its own access controls. Amazon Macie can scan the raw zone for personal data that shouldn&#8217;t be there, which is useful precisely because someone will eventually land a CSV export nobody reviewed.</p>



<p class="wp-block-paragraph">Erasure needs thinking about before you have to do it. Iceberg makes row-level deletes straightforward, but a deleted row is still present in older snapshots, and time travel will happily return it. Deletion isn&#8217;t complete until snapshot expiration has run past the snapshots containing that row and the underlying files have been removed. Set retention deliberately, know what your window is, and be able to state it. Crypto-shredding, where the identity mapping is encrypted with a per-subject key you can destroy, is the cleaner design if the requirement is strict.</p>



<p class="wp-block-paragraph">The same discipline applies to workstations. Analysts pull extracts, extracts contain guest names, and laptops get replaced. A secure erasure tool such as O&amp;O SafeErase covers that end of it, and it belongs in the same policy as the lake.</p>



<p class="wp-block-paragraph">For access control, Lake Formation is the right layer. Define column and row filters once and they&#8217;re enforced across Athena, EMR and Redshift rather than reimplemented per engine. A group operator can then give each property&#8217;s management team row access to its own data from the same tables, which is far less work than building a schema per property.</p>



<h2 class="wp-block-heading">File physics at hotel scale</h2>



<p class="wp-block-paragraph">Hotel data is small. A single property produces a few hundred room-nights a day. Even a large group is nowhere near the volumes that data lake tutorials assume, and that changes the advice in one specific direction: your problem is too many tiny files, not too much data.</p>



<p class="wp-block-paragraph">Two habits create it. Partitioning by day when a property generates kilobytes per day, and polling the PMS every fifteen minutes so each run writes another sliver. Both feel careful and both make queries slower and more expensive, because the engine spends its time opening files rather than reading them.</p>



<ul class="wp-block-list">
<li>Partition curated tables by property and month, not by day. Iceberg prunes from its own metadata, so you get file-level skipping without the partition explosion.</li>

<li>Batch the writes. Buffer frequent polls and MERGE on a schedule that matches how the numbers are actually consumed, which for most operational reporting means hourly at most.</li>

<li>Run compaction. The Glue Data Catalog offers managed table optimizers for Iceberg tables covering compaction, snapshot retention and orphan file deletion, configurable from the console, CLI or API. Check the current format and engine support against the AWS docs before you rely on it.</li>

<li>Or drive it from Athena directly with <code>OPTIMIZE</code> to rewrite data files and <code>VACUUM</code> to expire snapshots and remove orphans.</li>
</ul>



<p class="wp-block-paragraph">For the raw zone, where tables are usually plain Hive-style Parquet rather than Iceberg, Athena partition projection is worth knowing about. It computes partition values from a configured pattern instead of listing them from the catalog, which removes a metadata round trip on tables with many partitions. It does not apply to Iceberg tables, which handle pruning through their own manifests.</p>



<p class="wp-block-paragraph">Athena bills on data scanned and Glue on job runtime, so cost tracks file layout and query shape rather than how much data you own. That&#8217;s a good thing at this scale, but it does mean a single badly written dashboard query on a schedule can cost more than the entire storage bill. Tools like Vantage or CloudZero make that visible per workload if the AWS cost console isn&#8217;t giving you enough granularity.</p>



<h2 class="wp-block-heading">Troubleshooting a hotel data lake on AWS</h2>



<p class="wp-block-paragraph"><strong>The lake and the PMS report disagree by a small, consistent amount.</strong> Almost always a definition, not a bug. Compare the room count first: if rooms sold matches and revenue doesn&#8217;t, you&#8217;re including or excluding tax, packages or fees differently. If rooms sold doesn&#8217;t match, check house-use and comp rooms.</p>



<p class="wp-block-paragraph"><strong>Yesterday reconciles, last month doesn&#8217;t.</strong> The append-only failure. Query your restatement log for that period. If you don&#8217;t have one, compare the current curated table against an older Iceberg snapshot for the same business dates.</p>



<p class="wp-block-paragraph"><strong>Numbers went backwards after a backfill.</strong> An older extract overwrote a newer one. Check that the as-of guard is present in the MERGE and that the staging table is deduplicated to one row per key before the merge runs, keeping the highest as-of value.</p>



<p class="wp-block-paragraph"><strong>Occupancy exceeds one hundred percent.</strong> Usually duplicate room-nights from a reservation that was modified into a new record while the old one stayed active, or a room moved mid-stay and counted in both rooms for the same night. Add a uniqueness check on property, business date and room number to the pipeline and fail the load rather than publishing it.</p>



<p class="wp-block-paragraph"><strong>Queries got slow and nothing changed.</strong> Look at file counts before anything else. Compaction that stopped running, or a new high-frequency job appending small files, explains most of these.</p>



<p class="wp-block-paragraph"><strong>A daily load silently produced nothing.</strong> This is the one that hurts, because an empty successful run looks like a healthy pipeline. Alert on the absence of expected rows per property per business date, not just on job failure. A deadman check in CloudWatch, or in Grafana Cloud if you already run dashboards there, catches it the same morning instead of at month end.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Storing reservations instead of room-nights, then rebuilding the date logic in every report.</li>

<li>Treating the ingestion date as the business date. They diverge the first time a load fails and gets rerun the next morning.</li>

<li>Building the reporting layer before agreeing the metric definitions with the finance and revenue teams.</li>

<li>Pulling the full guest profile because the API returns it, rather than the fields the analysis needs.</li>

<li>Skipping the raw zone and transforming on ingest, which means a logic bug is unrecoverable without re-extracting from the vendor.</li>

<li>Partitioning by day out of habit, then wondering why a small dataset queries slowly.</li>

<li>Relying on time travel as the audit trail, and discovering retention expired the snapshot finance asked about.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>One row per room, per night, per reservation. Everything else derives from it.</li>

<li>Business date and as-of timestamp on every fact row, always.</li>

<li>Raw zone immutable, curated zone merged, consumption zone shaped for the tool that reads it.</li>

<li>Metric definitions in version control, next to the SQL that implements them.</li>

<li>A daily reconciliation query against one PMS report, checked automatically, with a tolerance and an alert.</li>

<li>Pseudonymise guest identity at ingestion and keep the mapping somewhere separate.</li>

<li>Compaction and snapshot retention configured before the table grows, not after someone complains.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need Iceberg, or is plain Parquet enough?</h3>



<p class="wp-block-paragraph">Plain Parquet is fine for anything genuinely append-only, such as raw extracts and event logs. It stops being fine the moment you need to correct history, because updating a row means rewriting whole partitions and coordinating that yourself. Since correcting history is the defining characteristic of hotel data, the curated layer wants a table format with row-level updates.</p>



<h3 class="wp-block-heading">Should the data lake match the PMS exactly?</h3>



<p class="wp-block-paragraph">It should match one named PMS report, on one named metric, within a tolerance you&#8217;ve agreed. Trying to match every report is a trap, because vendor reports use different denominators and different treatment of taxes and fees, and several of them disagree with each other. Pick the report finance already trusts and reconcile to that one.</p>



<h3 class="wp-block-heading">How often should the pipeline pull from the PMS?</h3>



<p class="wp-block-paragraph">Match the decision it feeds. Financial reporting is a daily cycle anchored on night audit and needs nothing faster. Operational views like arrivals and room status benefit from intraday refresh, but that&#8217;s a different table with a different retention policy, not a reason to run the whole lake hourly.</p>



<h3 class="wp-block-heading">Can guest names go in the data lake?</h3>



<p class="wp-block-paragraph">Sometimes, with a lawful basis, retention limits and access controls. But most of the analysis people actually want, occupancy, rate, segment, channel mix, pace, works perfectly on a pseudonymous key. Start without names and add them back only for a use case that genuinely requires them. Card data is a different matter and the answer there is simply no.</p>



<h3 class="wp-block-heading">Athena or Redshift for the reporting layer?</h3>



<p class="wp-block-paragraph">At single-property and small-group volumes, Athena over Iceberg is usually enough, and there&#8217;s no cluster to size or keep warm. Redshift earns its place when you have high-concurrency BI users, heavy joins across many domains, or an existing warehouse investment. Either way, keep the storage open so the choice stays reversible.</p>



<h3 class="wp-block-heading">How do I handle a group with many properties?</h3>



<p class="wp-block-paragraph">One set of tables with property code as the leading partition key, and Lake Formation row filters for per-property access. Separate schemas per property look tidy at three properties and become unmanageable at thirty, because every schema change and every metric fix has to be applied thirty times.</p>



<h3 class="wp-block-heading">Where does benchmarking data fit?</h3>



<p class="wp-block-paragraph">Land it in the same lake as a separate domain, at whatever grain the provider supplies, usually a market aggregate rather than a room-night. Keep it in its own tables and join at report time. Mixing external benchmark figures into your own fact table is how index numbers end up in a revenue total.</p>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">A hotel data lake on AWS is not hard because of the volume. It&#8217;s hard because the past keeps changing, and an architecture that assumes otherwise fails quietly and slowly rather than loudly and fast.</p>



<p class="wp-block-paragraph">Get the room-night grain right, carry a business date and an as-of timestamp on every row, merge instead of appending, and write down what each metric means before anyone builds a dashboard on it. The S3, Glue, Iceberg and Athena assembly is the easy part, and it&#8217;s well documented. Reconciling with the night audit six months later is the part that decides whether anyone trusts the thing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Working on this with me</h2>



<p class="wp-block-paragraph">Most of the hospitality data work I get asked about is one of a small number of shapes. Here&#8217;s where I&#8217;d be useful:</p>



<ul class="wp-block-list">
<li>Designing the room-night model and the bitemporal load, including the MERGE logic and the restatement log, so your numbers reconcile with the night audit and you can show why they moved.</li>

<li>Getting data out of the PMS reliably, whether that&#8217;s OHIP against OPERA Cloud, a REST API on a cloud-native system, or a scheduled export relay for an on-premise property.</li>

<li>Auditing an existing lake that has quietly drifted, finding which business dates restated and which metric definitions diverged from the reports finance uses.</li>

<li>Cutting Athena and Glue spend on a small dataset that behaves like a big one, usually through partition layout, compaction and killing scheduled queries nobody reads.</li>

<li>Setting up Lake Formation row and column controls for a multi-property group so each property sees its own data from shared tables.</li>

<li>Building the boring safety net: freshness and row-count checks per property per business date, deadman alerts, and a daily automated reconciliation against a named PMS report.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;ve got a discrepancy you can&#8217;t explain, send me the two figures and the query behind one of them, and I&#8217;ll tell you which side is wrong.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/hotel-data-lake-aws/">Building a Hotel Data Lake on AWS That Agrees With the Night Audit</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Salesforce AWS Integration Architecture: The Four Paths and What Breaks Silently</title>
		<link>https://john-nessime.com/blog/devops/salesforce-aws-integration-architecture/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 19 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AppFlow]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Change Data Capture]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Event Relay]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[Named Credentials]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Platform Events]]></category>
		<category><![CDATA[PrivateLink]]></category>
		<category><![CDATA[Pub/Sub API]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[Serverless]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=209</guid>

					<description><![CDATA[<p>Most Salesforce AWS integrations don't fail loudly, they drift. This guide covers the four integration paths between Salesforce and AWS, when to use each, the silent failure mode hiding in every one of them, and the reconciliation job you have to build yourself.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-aws-integration-architecture/">Salesforce AWS Integration Architecture: The Four Paths and What Breaks Silently</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone in the ops channel asks why the opportunity count on the QuickSight dashboard doesn&#8217;t match the count in the Salesforce report. It&#8217;s off by a few hundred. Nobody can answer that, and worse, nobody can say when the two numbers stopped agreeing. Every pipeline is green. The scheduled flow reports successful runs. The EventBridge rule shows invocations. Everything is healthy except the data.</p>



<p class="wp-block-paragraph">That&#8217;s the shape of most Salesforce AWS integration problems. They&#8217;re rarely outages. They&#8217;re divergence, accumulating quietly while every status page stays green, and you find out about it when someone in finance runs a number past a customer.</p>



<p class="wp-block-paragraph">This guide covers the four integration paths between Salesforce and AWS, which one to reach for and when, the specific failure each one hides, and the reconciliation layer you&#8217;ll have to build yourself because neither vendor ships it.</p>



<h2 class="wp-block-heading">Why a Salesforce AWS integration fails silently</h2>



<p class="wp-block-paragraph">Start here, because it changes how you design everything downstream. There are three mechanisms that lose data without raising an error anywhere you&#8217;re looking.</p>



<p class="wp-block-paragraph"><strong>Events sent to an unassociated partner event bus are dropped on the floor.</strong> When you create an Event Relay, Salesforce creates a partner event source in your AWS account. Until you associate that source with an event bus, EventBridge discards anything the partner publishes to it. AWS is explicit that those events are not persisted at rest. There&#8217;s no queue holding them for you, no backlog to drain once you finish the setup. If the relay is running and the bus isn&#8217;t associated, you are streaming into a void.</p>



<p class="wp-block-paragraph"><strong>The Salesforce event bus retains high-volume events for 72 hours.</strong> Platform events and change data capture events sit on the event bus for three days, and each one carries a replay ID so a durable subscriber can resume from where it left off. Past that window, they&#8217;re purged. Salesforce doesn&#8217;t guarantee availability beyond it. So a relay that goes into an error state on a Friday afternoon is a data loss event by Monday morning, and the only symptom during those three days is an absence of messages, which looks identical to a quiet weekend.</p>



<p class="wp-block-paragraph"><strong>Deletes don&#8217;t show up in query-based syncs.</strong> A scheduled pull selects records that exist. Records that stopped existing between runs are simply not in the result set, so your S3 prefix or Redshift table keeps a row that Salesforce no longer has. Nothing errors. The row count on the AWS side just creeps upward relative to the CRM, forever, until someone notices.</p>



<p class="wp-block-paragraph">Every architecture decision below is really a decision about which of these three you&#8217;re accepting and how you&#8217;re going to detect it.</p>



<h2 class="wp-block-heading">A Salesforce AWS integration starts with a direction, not a service</h2>



<p class="wp-block-paragraph">The most common mistake I see is picking a service first, usually whichever one appeared in a conference demo, and then bending the requirement to fit it. Two questions decide the path, and they take about a minute to answer.</p>



<ol class="wp-block-list">
<li><strong>Which way does the data move?</strong> Salesforce to AWS, or AWS to Salesforce. If the answer is &#8220;both&#8221;, you have two integrations, not one, and they will have different failure modes and different owners.</li>

<li><strong>What shape is the workload?</strong> Batch, where you want a consistent set of records on a schedule, or event-driven, where you want a reaction within seconds of a change.</li>
</ol>



<p class="wp-block-paragraph">That gives you four cells, and each cell has a default answer.</p>



<h3 class="wp-block-heading">Salesforce to AWS, batch: Amazon AppFlow</h3>



<p class="wp-block-paragraph">Amazon AppFlow is the managed answer for pulling Salesforce objects into S3, Redshift, or the Glue Data Catalog on a schedule or on demand. It handles the OAuth connection, field mapping, and incremental pulls without you writing or hosting anything. For analytics loads it&#8217;s usually the right default, and it&#8217;s the path of least resistance if the destination is a data lake you&#8217;re already cataloguing with Glue and querying with Athena.</p>



<p class="wp-block-paragraph">The trade-off is that it&#8217;s a polling client, and it spends your Salesforce API allocation to do it. AWS documents this plainly with an example: an hourly flow pulling five pages of data makes 120 API calls a day, and all of them count against the org&#8217;s 24-hour API request limit. Multiply that by every object you&#8217;re syncing and every environment you&#8217;re syncing from, and AppFlow can become the largest single consumer of an allocation that Apex callouts, integration users, and third-party tools are also drawing on.</p>



<p class="wp-block-paragraph">The failure mode that catches people: if a field referenced in the flow&#8217;s mapping is deleted in Salesforce, the flow run fails. Someone cleans up an unused custom field on a Tuesday and your nightly load breaks on Wednesday, in a service that nobody on the Salesforce side knows exists. AppFlow will automatically pick up newly created Salesforce fields when the destination is S3, but the reverse case, a removed field, needs a human to edit the mapping. Put the flow names somewhere the Salesforce admins will see them before they run a field cleanup.</p>



<h3 class="wp-block-heading">Salesforce to AWS, event-driven: Event Relay and Amazon EventBridge</h3>



<p class="wp-block-paragraph">Event Relay subscribes to a channel on the Salesforce event bus and streams platform events and change data capture events into Amazon EventBridge as a partner event source. No middleware, no subscriber you have to keep alive, no polling and therefore no API allocation burn on the read path. From EventBridge you fan out to Lambda, Step Functions, SQS, Firehose, whatever the downstream is.</p>



<p class="wp-block-paragraph">Setup runs across both consoles, and the order matters. On the Salesforce side you create a named credential that points at your AWS account and region. It uses no authentication protocol, because trust is established by the partner event source relationship rather than by a credential in the request. The region goes in uppercase, which is the single most common reason a first attempt doesn&#8217;t work.</p>



<pre class="wp-block-code"><code>POST /services/data/vXX.X/tooling/sobjects/NamedCredential/

{
  "FullName": "AwsRelayCredential",
  "Metadata": {
    "label": "AwsRelayCredential",
    "endpoint": "arn:aws:US-EAST-1:111122223333",
    "principalType": "NamedUser",
    "protocol": "NoAuthentication",
    "generateAuthorizationHeader": true
  }
}</code></pre>



<p class="wp-block-paragraph">The endpoint is not a URL. It&#8217;s an ARN-shaped string carrying the AWS region and the twelve-digit account ID, and that&#8217;s the whole addressing scheme. Get the account wrong and the partner event source appears in a stranger&#8217;s console, not yours.</p>



<p class="wp-block-paragraph">You then create an event channel, add the platform events or change events you want as channel members, and create the relay configuration against that channel. A relay is created in a stopped state deliberately, so you get a chance to verify the AWS account and region before anything flows. That pause is the useful part of the design, not an inconvenience to click through.</p>



<p class="wp-block-paragraph">Now switch to AWS and associate the source. This is the step that, skipped, produces the silent drop described earlier.</p>



<pre class="wp-block-code"><code># Find the partner event source Salesforce created in your account
aws events list-event-sources --region us-east-1

# Create the matching bus. Both values must be the source name, character for character.
aws events create-event-bus 
  --name "aws.partner/salesforce.com/&lt;org-and-channel-suffix&gt;" 
  --event-source-name "aws.partner/salesforce.com/&lt;org-and-channel-suffix&gt;" 
  --region us-east-1

# Confirm the source moved from PENDING to ACTIVE
aws events describe-event-source 
  --name "aws.partner/salesforce.com/&lt;org-and-channel-suffix&gt;" 
  --region us-east-1</code></pre>



<p class="wp-block-paragraph"><code>describe-event-source</code> returning <code>ACTIVE</code> is your proof that a matching bus exists and is enabled. <code>PENDING</code> means either you haven&#8217;t created the bus or it&#8217;s deactivated, and every event published in that state is gone. This is the single check worth wiring into a synthetic monitor, because it&#8217;s cheap to run and it catches the most expensive mistake.</p>



<p class="wp-block-paragraph">Only once the source is active do you set the relay state to run. From the Salesforce side, the feedback object is where relay errors surface rather than in any log you&#8217;d normally tail:</p>



<pre class="wp-block-code"><code>SELECT Id, RemoteResource, Status, ErrorMessage, ErrorTime, ErrorIdentifier
FROM EventRelayFeedback
WHERE EventRelayConfigId = '7k2XXXXXXXXXXXXXXX'</code></pre>



<p class="wp-block-paragraph">If you&#8217;d rather not use Event Relay at all, the Pub/Sub API is the direct route: a gRPC interface to the same event bus, delivering Avro-encoded payloads, which you subscribe to from your own consumer. That&#8217;s more code and a process to keep running, on ECS or a VPS from a provider like InterServer or Hetzner if you&#8217;re keeping it outside AWS, but you get explicit control over replay ID checkpointing. Teams already running Kafka often go this way and land the stream through a managed connector from Confluent instead. Event Relay is the lower-operational-cost option; the Pub/Sub API is the one to reach for when you need the checkpoint under your own control.</p>



<h3 class="wp-block-heading">AWS to Salesforce, event-driven: EventBridge API Destinations</h3>



<p class="wp-block-paragraph">Going the other way, an API destination lets EventBridge call the Salesforce REST API directly, with no Lambda in the middle. You create a connection holding OAuth client credentials from a Salesforce connected app, point the destination at your org&#8217;s My Domain endpoint, and route matched events to it. Publishing back into an inbound platform event is the common pattern, so the write lands on the Salesforce event bus and Apex or Flow picks it up from there.</p>



<p class="wp-block-paragraph">Two constraints shape the design, and both are documented:</p>



<ul class="wp-block-list">
<li><strong>Five second client execution timeout.</strong> If Salesforce takes longer than that to respond, EventBridge times out the request and treats it as a retryable failure. Anything you make synchronous behind that endpoint, a trigger doing a callout, a validation chain, a busy org at peak, will eat into it.</li>

<li><strong>The default retry policy is generous, which is the problem.</strong> For event buses, EventBridge will retry a failed delivery for up to 24 hours and up to 185 times by default. If the endpoint is genuinely broken, that&#8217;s a day of retries against your org, consuming API calls, before the event is discarded with nowhere to go.</li>
</ul>



<p class="wp-block-paragraph">Tighten the policy and always attach a dead letter queue. Failed events land somewhere you can inspect and replay rather than evaporating at the end of the retry window:</p>



<pre class="wp-block-code"><code>{
  "Id": "salesforce-writeback",
  "Arn": "arn:aws:events:us-east-1:111122223333:api-destination/sf-platform-event/&lt;id&gt;",
  "RoleArn": "arn:aws:iam::111122223333:role/EventBridgeApiDestinationRole",
  "RetryPolicy": {
    "MaximumRetryAttempts": 20,
    "MaximumEventAgeInSeconds": 3600
  },
  "DeadLetterConfig": {
    "Arn": "arn:aws:sqs:us-east-1:111122223333:salesforce-writeback-dlq"
  }
}</code></pre>



<p class="wp-block-paragraph">Set the invocation rate limit on the destination to something your org can absorb. It exists specifically so you can align delivery with a third-party API&#8217;s limits, and Salesforce is very much a third-party API with limits.</p>



<p class="wp-block-paragraph">One thing worth knowing before you design around it: API destinations don&#8217;t support mutual TLS. If your security review requires mTLS on outbound calls, you&#8217;re putting a Lambda or a proxy in the path regardless.</p>



<h3 class="wp-block-heading">AWS to Salesforce, batch: Lambda and the Bulk API</h3>



<p class="wp-block-paragraph">For loading volume back into the org, nothing managed does this well and you write it yourself. A Lambda or a container job authenticates with the JWT bearer flow against a connected app, submits a Bulk API job, and polls for results. The REST API is fine for small writes; past a few thousand records the per-call overhead and the API allocation make the Bulk API the only sensible choice.</p>



<p class="wp-block-paragraph">This is the cell where you own everything: retries, idempotency, partial-failure handling, and backoff when the org throttles you. Use external ID fields and upsert semantics rather than insert, so a retried batch converges instead of duplicating. Store the job ID before you submit, not after, so a Lambda timeout mid-flight leaves you something to reconcile against. And keep the Salesforce credential in Secrets Manager with rotation wired up, because a hardcoded connected app secret in an environment variable is the finding that shows up in every audit.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The network layer: public internet, PrivateLink, or Private Connect</h2>



<p class="wp-block-paragraph">All four paths above work over the public internet with TLS, and for most workloads that&#8217;s genuinely fine. The question is whether your compliance posture or your risk appetite says otherwise.</p>



<p class="wp-block-paragraph">Salesforce Private Connect is the managed private path. Salesforce operates a transit VPC in each supported AWS region, and you create a PrivateLink connection between that transit VPC and yours. Traffic between the two never traverses the public internet. The terminology is worth getting right because it trips people up in design reviews:</p>



<ul class="wp-block-list">
<li><strong>Inbound</strong> means traffic flowing <em>into</em> Salesforce. AWS to Salesforce, or on-premises to Salesforce.</li>

<li><strong>Outbound</strong> means traffic flowing <em>out of</em> Salesforce. Salesforce calling a service in your VPC.</li>
</ul>



<p class="wp-block-paragraph">The direction is named from Salesforce&#8217;s point of view, not yours, and getting it backwards means provisioning the wrong connection.</p>



<p class="wp-block-paragraph">AppFlow can ride on this too. Choosing the PrivateLink option when you create the Salesforce connection has AppFlow route through Private Connect and manage the endpoint lifecycle for you, which is a meaningful simplification over building it by hand.</p>



<p class="wp-block-paragraph">Where I&#8217;d push back: Private Connect is a licensed add-on, and each VPC needs its own connection, so a multi-account AWS estate with separate VPCs per environment multiplies the cost. If the driver is a checkbox on a questionnaire rather than a real threat model, TLS with strict IP allowlisting on the Salesforce side and a well-scoped connected app gets you most of the way for nothing. If the driver is a regulator who wants to see that customer data never touches a public network path, Private Connect is the clean answer and the audit conversation ends quickly. Decide which one you&#8217;re in before you start pricing it.</p>



<h2 class="wp-block-heading">The reconciliation job nobody builds</h2>



<p class="wp-block-paragraph">This is the part that separates an integration that survives from one that quietly rots, and it&#8217;s about thirty lines of code plus a schedule.</p>



<p class="wp-block-paragraph">Run a count comparison on a window that both sides can express. Nightly is plenty:</p>



<pre class="wp-block-code"><code>-- Salesforce side, via the REST query endpoint
SELECT COUNT(Id) FROM Opportunity
WHERE LastModifiedDate = LAST_N_DAYS:1

-- AWS side, same window, in Athena
SELECT COUNT(DISTINCT id) FROM crm.opportunity
WHERE last_modified_date &gt;= date_add('day', -1, current_date)</code></pre>



<p class="wp-block-paragraph">If the two numbers differ by more than a small tolerance for clock skew at the window edges, alert. That single check catches a stopped relay, a failed flow, a mapping that silently dropped a field, and a partner event source sitting in <code>PENDING</code>. It catches all of them without you having to instrument each path separately, which is exactly why it&#8217;s worth building first.</p>



<p class="wp-block-paragraph">For the delete problem, Salesforce exposes a dedicated endpoint. Ask it for records removed in a window and apply those deletions on the AWS side:</p>



<pre class="wp-block-code"><code>GET /services/data/vXX.X/sobjects/Opportunity/deleted/
    ?start=&lt;ISO-8601 timestamp, URL encoded&gt;
    &amp;end=&lt;ISO-8601 timestamp, URL encoded&gt;</code></pre>



<p class="wp-block-paragraph">The window you can ask about is bounded by how long deleted records stay recoverable, so run it more often than that bound, not less. Change data capture delivers delete events too, which is the better answer when it&#8217;s available, but the endpoint is what you use to backfill after a gap, and after a 72-hour gap it&#8217;s the <em>only</em> thing that will tell you what you missed.</p>



<p class="wp-block-paragraph">Put both checks on one dashboard alongside your AppFlow run status and EventBridge failed-invocation metrics. CloudWatch will do it, and if you&#8217;re already running Grafana Cloud or Datadog for the rest of the estate, pull the Salesforce-side numbers in as a custom metric so one panel answers &#8220;are the two systems agreeing right now&#8221;.</p>



<h2 class="wp-block-heading">Troubleshooting a Salesforce AWS integration</h2>



<ul class="wp-block-list">
<li><strong>Relay is running, nothing arrives in EventBridge.</strong> Run <code>describe-event-source</code>. If the state is <code>PENDING</code>, the bus was never created or has been deactivated, and everything published so far is gone. Create the bus, then treat the gap as a backfill job.</li>

<li><strong>Relay won&#8217;t start, or flips to an error state.</strong> Query <code>EventRelayFeedback</code> for the config. The usual causes are a wrong AWS account ID in the named credential, a region in lowercase, or the channel having no members.</li>

<li><strong>Events arrive but no rule matches.</strong> Check the rule is attached to the partner event bus, not the default bus. This one costs people an afternoon. Match on the source prefix and the detail type, and test with an event pattern before wiring a real target.</li>

<li><strong>An AppFlow flow that ran for months suddenly fails.</strong> Look for a field in the mapping that no longer exists in the source object. Also check for a bad gateway error, which points at a connection configured with the wrong instance URL and is fixed by deleting the connection and recreating it rather than editing it.</li>

<li><strong>API destination returns 401 intermittently.</strong> The connection refreshes its OAuth token when it sees a 401 or 407. Persistent 401s usually mean the connected app has no usable scopes for the client credentials flow, or the run-as user was changed. Check the connected app&#8217;s scopes before you touch anything in AWS.</li>

<li><strong>Everything times out at almost exactly five seconds.</strong> That&#8217;s the API destination timeout, not Salesforce being slow. Move the work behind an inbound platform event so the HTTP call returns immediately and the processing happens asynchronously in the org.</li>

<li><strong>Unexplained API limit exhaustion.</strong> Sum the scheduled AppFlow runs against page counts before blaming a rogue integration user. A frequent flow across several objects adds up faster than most people expect.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Starting the relay before associating the partner event bus, and assuming the events queued somewhere.</li>

<li>Treating the 72-hour retention as a comfortable buffer. It&#8217;s three days including a weekend, and an unmonitored relay burns through it without a single alert.</li>

<li>Using change data capture as a replacement for an initial load. It streams changes from the moment you enable it; it doesn&#8217;t backfill history.</li>

<li>Building the whole integration bidirectionally through one mechanism, then discovering you&#8217;ve created an echo loop where a write from AWS fires a change event that triggers another write.</li>

<li>Sizing the AppFlow schedule for freshness without checking what it does to the org&#8217;s API allocation.</li>

<li>Leaving the default 24-hour EventBridge retry policy on an API destination with no dead letter queue.</li>

<li>Provisioning Private Connect for every VPC because it&#8217;s the secure-sounding option, without modelling the per-connection cost across environments.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Build the reconciliation check before you build the second integration path. It&#8217;s the only thing that tells you the whole system is working.</li>

<li>Monitor the partner event source state, not just rule invocations. Invocations go to zero on a quiet day and on a broken day, and the state field distinguishes them.</li>

<li>Give every integration its own connected app and its own integration user. Shared credentials make it impossible to attribute API consumption or revoke one path without breaking three.</li>

<li>Make writes idempotent with external IDs and upsert. Every path here retries, and some retry aggressively.</li>

<li>Version your event payloads. Platform events are immutable once published, and adding a required field later means a new event definition, not an edit.</li>

<li>Keep the AWS-side resource names discoverable from the Salesforce side. Put the flow name or event bus name in the named credential label so a Salesforce admin can see what&#8217;s attached before they change something.</li>

<li>Define the whole thing in Terraform or CloudFormation on the AWS side and in metadata on the Salesforce side, so a sandbox refresh doesn&#8217;t leave you clicking through two consoles from memory.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need MuleSoft to integrate Salesforce with AWS?</h3>



<p class="wp-block-paragraph">No. For the four patterns above, the native services cover it without an integration platform. MuleSoft, Boomi, and similar tools earn their place when you&#8217;re orchestrating across many systems, need transformation logic that lives outside both platforms, or have a governance requirement for a single integration layer. For a two-system integration, they&#8217;re usually more machinery than the problem needs.</p>



<h3 class="wp-block-heading">Does Event Relay use my Salesforce API allocation?</h3>



<p class="wp-block-paragraph">Event delivery counts against your platform event delivery allocation, which is a separate budget from the 24-hour API request limit that REST calls and AppFlow polling draw on. That separation is one of the strongest arguments for the event-driven path when API consumption is already tight in the org.</p>



<h3 class="wp-block-heading">Can I replay Salesforce events that AWS missed?</h3>



<p class="wp-block-paragraph">Within the 72-hour retention window, yes, using replay IDs from a durable subscriber. Beyond it, no. The events are purged and there&#8217;s no recovery path through the event bus. Backfilling from that point means querying the objects directly and using the deleted-records endpoint to catch removals, which is exactly why the reconciliation job matters.</p>



<h3 class="wp-block-heading">Should I use AppFlow or write my own Lambda extractor?</h3>



<p class="wp-block-paragraph">Start with AppFlow. It removes the auth handling, the pagination, and the scheduling, and for standard object pulls into S3 or Redshift it&#8217;s hard to justify writing that yourself. Move to a custom extractor when you need transformation logic AppFlow can&#8217;t express, when you&#8217;re hitting connector limitations on a specific object, or when you want tighter control over API consumption than a fixed schedule allows.</p>



<h3 class="wp-block-heading">How do I send data from AWS back into Salesforce without a Lambda?</h3>



<p class="wp-block-paragraph">EventBridge API destinations call the Salesforce REST API directly using OAuth client credentials from a connected app. Point them at an inbound platform event endpoint so the write returns fast and processing happens asynchronously in the org. You still need a Lambda if you require mTLS or payload transformation beyond what the input transformer handles.</p>



<h3 class="wp-block-heading">Is Private Connect required for a secure Salesforce AWS integration?</h3>



<p class="wp-block-paragraph">Not required. Everything runs over TLS by default. Private Connect removes the public internet from the path entirely, which matters when a regulator or a security review asks you to demonstrate it. It&#8217;s a licensed add-on with a per-connection cost, so model it against your actual VPC count before committing.</p>



<h3 class="wp-block-heading">Why does my EventBridge rule never fire even though events are arriving?</h3>



<p class="wp-block-paragraph">Almost always because the rule is on the default event bus rather than the partner event bus. Partner events land on their own bus and rules are bus-scoped, so a correct pattern on the wrong bus matches nothing and reports nothing.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A Salesforce AWS integration doesn&#8217;t usually break loudly. It drifts. A relay stops, a mapping goes stale, a delete never propagates, and the two systems keep answering questions confidently with different numbers. The services themselves are well built and the setup is genuinely not hard once you know the region goes in uppercase and the bus has to be associated first.</p>



<p class="wp-block-paragraph">So pick the path by direction and shape, wire the failure modes you&#8217;ve accepted into alerts, and build the nightly count comparison before you build anything clever. That one job is worth more than any amount of pipeline polish, because it&#8217;s the only thing in the architecture that can tell you the whole thing is still true.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your Salesforce and AWS setup?</h2>



<p class="wp-block-paragraph">I work with teams on the infrastructure side of CRM integrations, usually where the data is disagreeing and nobody can prove why. Things I can help with:</p>



<ul class="wp-block-list">
<li>Reviewing an existing Salesforce to AWS pipeline and finding where records are being lost between the two.</li>

<li>Standing up Event Relay into EventBridge end to end, including the partner bus association and the monitoring that proves it&#8217;s live.</li>

<li>Designing the AppFlow schedule and object set so it doesn&#8217;t eat the org&#8217;s API allocation.</li>

<li>Building the reconciliation and delete-propagation jobs, with alerting in CloudWatch, Grafana, or whatever you already run.</li>

<li>Writing the write-back path properly: Bulk API jobs, idempotent upserts, dead letter queues, and credential rotation through Secrets Manager.</li>

<li>Working out whether Private Connect is worth the licence in your specific environment, or whether TLS and tighter scoping gets you there.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;ve got a flow that&#8217;s failing, a relay stuck in an error state, or two record counts that won&#8217;t agree, send me the config or the error and I&#8217;ll tell you what I&#8217;d check first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-aws-integration-architecture/">Salesforce AWS Integration Architecture: The Four Paths and What Breaks Silently</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>SAP Data Integration with AWS Glue: The Failure Modes Nobody Demos</title>
		<link>https://john-nessime.com/blog/devops/sap-data-integration-aws-glue/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 18 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Change Data Capture]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Delta Token]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[ODP]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[SAP]]></category>
		<category><![CDATA[SAP OData]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[VPC]]></category>
		<category><![CDATA[Zero-ETL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=213</guid>

					<description><![CDATA[<p>The AWS Glue SAP OData connector demos beautifully and then drifts. A practical look at delta token invalidation, deletes that never delete, partitioning limits on ODP entities, and the licensing line between ODP-RFC and ODP-OData.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/sap-data-integration-aws-glue/">SAP Data Integration with AWS Glue: The Failure Modes Nobody Demos</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The Glue job had been green for weeks. Every run finished, every run wrote Parquet, nobody had a reason to look at it. Then someone from finance mentioned that a report was short by a few hundred rows, and the reconciliation that followed turned up the real problem: the job had not actually been reading changes for most of that time. It had been reading nothing, successfully.</p>



<p class="wp-block-paragraph">That is the shape of most trouble with SAP data integration with AWS Glue. The connector does not usually crash. It returns an empty or partial result set, the job exits zero, and your data lake quietly diverges from the source system. By the time anyone notices, you have no idea which run was the last correct one.</p>



<p class="wp-block-paragraph">This post covers the parts that bite after the proof of concept: how delta tokens actually behave, why deleted records keep showing up in your reports, where partitioning stops working, and the licensing distinction between two things that both get called &#8220;ODP&#8221; and are not the same thing at all. There is a troubleshooting section at the end for when a job is already broken.</p>



<h2 class="wp-block-heading">First, get the ODP terminology right, because licensing depends on it</h2>



<p class="wp-block-paragraph">Operational Data Provisioning is SAP&#8217;s framework for exposing extraction interfaces in a delta-aware way. A provider (a BW extractor, an ABAP CDS view, SLT, a HANA information view) writes changes into an Operational Delta Queue, and subscribers pull from that queue. This part is uncontroversial.</p>



<p class="wp-block-paragraph">What matters is that there are two ways to talk to ODP, and SAP treats them very differently. SAP Note 3255746 restricts third-party and customer applications from using the RFC modules of the ODP Data Replication API against ABAP sources. That note has been sharpened over several revisions, from &#8220;unsupported&#8221; to &#8220;not permitted&#8221;, and SAP has stated it reserves the right to introduce technical measures and audit unpermitted use. A number of established extraction tools were built directly on ODP-RFC and have had to move.</p>



<p class="wp-block-paragraph">The AWS Glue SAP OData connector takes the other path. It talks OData over HTTPS to an activated SAP Gateway service, and uses the ODP framework through that OData layer rather than through RFC. That is the route SAP points customers at. If you are being asked in an architecture review whether your pipeline is on the wrong side of Note 3255746, this distinction is the whole answer, and it is worth having the note number written down before the meeting.</p>



<p class="wp-block-paragraph">Two honest caveats. First, notes get revised, so read the current version rather than a vendor blog summarising it, including this one. Second, &#8220;permitted&#8221; is not the same as &#8220;free&#8221;: you still need the OData service activated, a service user with the right authorisations, and a Basis team willing to own the queue growth on their side.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: the delta token is shared state you do not own</h2>



<p class="wp-block-paragraph">This is the one that produces silent drift, and it is the reason the finance report was short.</p>



<p class="wp-block-paragraph">When you read an ODP-enabled entity with change data capture switched on, the connector asks SAP to initialise a delta queue subscription. SAP hands back a delta token. That token is your bookmark. On the next run you pass it back, and you get only what changed since.</p>



<pre class="wp-block-code"><code>sapodata_df = glueContext.create_dynamic_frame.from_options(
    connection_type="SAPOData",
    connection_options={
        "connectionName": "sap-prod-odata",
        "ENTITY_NAME": "entityName",
        "ENABLE_CDC": "true"
    },
    transformation_ctx="initial_load"
)</code></pre>



<p class="wp-block-paragraph"><code>ENABLE_CDC</code> defaults to false. Leave it off and you get a full load every single run, which is expensive but at least obvious. Turn it on and the token appears in the <code>DELTA_TOKEN</code> column, populated only on the last row of the frame. That last row is not a business record. It is a carrier for the token, and if you write your frame straight to S3 without stripping it you have just added a junk row to your table.</p>



<p class="wp-block-paragraph">The subsequent run passes the token back:</p>



<pre class="wp-block-code"><code>sapodata_df = glueContext.create_dynamic_frame.from_options(
    connection_type="SAPOData",
    connection_options={
        "connectionName": "sap-prod-odata",
        "ENTITY_NAME": "entityName",
        "DELTA_TOKEN": last_known_token
    },
    transformation_ctx="incremental_load"
)</code></pre>



<h3 class="wp-block-heading">Why this goes wrong</h3>



<p class="wp-block-paragraph">A delta token is bound to a service collection <em>and a user</em>. If a new initialisation runs with <code>ENABLE_CDC</code> set true for that same collection and user, SAP invalidates every token previously issued from the earlier initialisation. Not &#8220;expires eventually&#8221;. Invalidates, immediately.</p>



<p class="wp-block-paragraph">So picture two Glue jobs sharing one service account, both reading the same extractor. Someone redeploys job B with the CDC flag still set from testing. Job B re-initialises. Job A&#8217;s stored token is now dead, and the next time job A runs you get:</p>



<pre class="wp-block-code"><code>Could not open data access via extraction API RODPS_REPL_ODP_OPEN</code></pre>



<p class="wp-block-paragraph">An error is the good outcome. The bad outcome is code that catches the exception, falls back to a full load, and writes it over the top of an incremental table with no dedupe. Or code that re-initialises on failure, which fixes job A and breaks job B, and now the two jobs take turns invalidating each other forever while both keep exiting zero.</p>



<p class="wp-block-paragraph">Practical rules that come out of this:</p>



<ul class="wp-block-list">
<li>One SAP service user per consuming pipeline. Sharing a user is what couples independent jobs together.</li>

<li>Persist the token somewhere durable and versioned. DynamoDB, Parameter Store, S3 with versioning enabled, all fine. Job bookmarks are not a substitute, because the token is source state, not read position.</li>

<li>Never auto re-initialise on error. Fail loudly, page someone, and make the reinitialisation a deliberate human action with a documented backfill.</li>

<li>Alert on zero-change runs. A pipeline that returns no changes for several consecutive runs against a busy table is telling you something, and it is rarely &#8220;business is quiet&#8221;.</li>
</ul>



<h3 class="wp-block-heading">Retention on the SAP side is a hard deadline</h3>



<p class="wp-block-paragraph">The delta queue is not infinite. In transaction <code>ODQMON</code> the Basis team configures retention, and a reorganisation job cleans up data past that window. If your pipeline is down longer than the retention period, the changes you missed are gone from the queue and no token will bring them back. You are into a full reload and a reconciliation.</p>



<p class="wp-block-paragraph">Two things to find out before you go live: what the retention setting is, and whether the cleanup job is actually scheduled. Then make sure your pipeline&#8217;s alerting escalates well inside that window. A weekend outage on a queue with a short retention is a very expensive Monday.</p>



<p class="wp-block-paragraph">Related: if anyone has been testing extraction with <code>RODPS_REPL_TEST</code>, that program registers as a real additional subscriber rather than simulating one. Because the queue is shared and data is only cleaned once every subscriber has collected it, forgotten test subscriptions hold delta data indefinitely and the underlying queue tables grow. Ask whoever ran the tests to reset those subscriptions in ODQMON.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family two: deletes arrive as a column, not as a deletion</h2>



<p class="wp-block-paragraph">CDC extraction from ODP entities returns a couple of extra fields alongside your business columns. <code>DML_STATUS</code> carries <code>UPDATED</code> for inserts and updates, and <code>DELETED</code> for records removed at source. <code>GLUE_FETCH_SQ</code> is a per-record sequence derived from an epoch timestamp, in receipt order, and it is your tie-breaker when the same key changes twice inside one delta.</p>



<p class="wp-block-paragraph">Nothing acts on those fields for you. If your job appends the frame to S3 and calls it done, a deleted sales order is now a row in your lake with <code>DML_STATUS = DELETED</code> and every other column populated exactly as before. Any query that does not filter on that column counts it. Cancelled orders keep appearing in revenue. Closed accounts stay open. The pipeline is green, the numbers are wrong, and the discrepancy grows slowly enough that nobody catches it early.</p>



<p class="wp-block-paragraph">You have to apply the changes rather than accumulate them. Broadly two options:</p>



<ol class="wp-block-list">
<li><strong>Merge into a table format that supports it.</strong> Apache Iceberg, which Glue supports natively, gives you a real merge with delete handling. Land the raw delta, then merge on the business key with <code>GLUE_FETCH_SQ</code> deciding the winner within a batch, deleting where the status says deleted.</li>

<li><strong>Keep an append-only history and build a current-state view.</strong> Cheaper to write, and it preserves an audit trail. It also means every downstream consumer has to use the view, and someone eventually will not.</li>
</ol>



<p class="wp-block-paragraph">I reach for Iceberg first on anything that feeds reporting, because the correctness lives in the table rather than in the discipline of whoever writes the next query. Append-only is the right call when you genuinely need history, or when the consumer is a downstream warehouse like Amazon Redshift that will do its own merge anyway.</p>



<p class="wp-block-paragraph">Worth knowing: not every ODP provider emits deletes reliably. Some older extractors express them as reversal records rather than a delete flag. Confirm the behaviour for your specific entity with the functional owner before you build merge logic on an assumption.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family three: pagination and partitioning do not compose</h2>



<p class="wp-block-paragraph">The connector&#8217;s default page size is 50,000 records, and you can raise it up to 500,000. Anything above that is ignored and the maximum is used instead. Raising the page size cuts the number of round trips, which is usually the dominant cost on a wide table across a slow link.</p>



<p class="wp-block-paragraph">The catch is what your SAP Gateway will tolerate. A large page means a large response the ABAP stack has to assemble and hold, and Gateway or an intervening load balancer may cut it off before it completes. Symptoms are timeouts, connection resets, or a job that appears to paginate without ever finishing. Tune the page size with your Basis team, not in isolation, and treat it as a negotiation between round trips and response size rather than a value you set once.</p>



<p class="wp-block-paragraph">On partitioning, there is a specific documented limit that catches people out: ODP entities are not compatible with record-based partitioning, because pagination is already handled through skip and delta tokens. For record-based partitioning, the default concurrency is forced to null regardless of what you asked for. So the obvious lever for making a slow ODP extraction faster is not available.</p>



<p class="wp-block-paragraph">What you can still do on non-ODP OData entities is field-based partitioning, splitting a read across a bounded range:</p>



<ul class="wp-block-list">
<li><code>PARTITION_FIELD</code> is the field to split on, typically a date.</li>

<li><code>LOWER_BOUND</code> is inclusive, <code>UPPER_BOUND</code> is exclusive. Getting that backwards is how you duplicate or drop a boundary day.</li>

<li><code>NUM_PARTITIONS</code> defaults to 1, so leaving it out means you did not partition at all.</li>

<li>Where a limit and a partition are both applied, the limit wins. That is the usual cause of &#8220;I partitioned it and got fewer rows&#8221;.</li>
</ul>



<p class="wp-block-paragraph">For non-ODP entities you can also do timestamp-based incremental reads with <code>filteringExpression</code>, pointing at whatever last-changed field the entity exposes. Note the casing: that option is camelCase while most of the others are upper snake case, which trips people up when a filter appears to be ignored.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family four: connection, auth and network</h2>



<p class="wp-block-paragraph">These fail early and loudly, which makes them the easy ones. The trap is that they fail during setup, when you have three people on a call and someone suggests loosening something to move on.</p>



<p class="wp-block-paragraph">The connector supports two authentication types: CUSTOM, which is SAP basic authentication, and OAUTH2. The custom secret in AWS Secrets Manager takes this shape:</p>



<pre class="wp-block-code"><code>{
  "basicAuthUsername": "&lt;your SAP username&gt;",
  "basicAuthPassword": "&lt;your SAP password&gt;",
  "basicAuthDisableSSO": "&lt;True/False&gt;",
  "customAuthenticationType": "CustomBasicAuth"
}</code></pre>



<p class="wp-block-paragraph">For OAuth 2.0 the client secret goes into Secrets Manager under <code>USER_MANAGED_CLIENT_APPLICATION_CLIENT_SECRET</code>, and the client ID, authorisation code URL, token URL and scopes are supplied on the connection itself.</p>



<p class="wp-block-paragraph">Basic auth is genuinely easier for a scheduled pipeline: no interactive redirect, no refresh token to keep alive, and rotation is a Secrets Manager operation. OAuth gets you short-lived tokens and revocation you can actually reason about, which is what a security review will want. Both are defensible. What is not defensible is one shared service user across every pipeline, for the delta token reasons above as much as the audit ones.</p>



<p class="wp-block-paragraph">A few things that reliably waste a morning:</p>



<ul class="wp-block-list">
<li>The service path on the connection is the <em>catalog</em> service path, not the path to your object. The entity path goes in <code>ENTITY_NAME</code> on the job.</li>

<li>The OData service has to be activated in the SAP system. Until it is, nothing you do on the AWS side matters.</li>

<li>For a non-VPC connection, the application host URL must be reachable from the public internet. If your SAP system sits in a private network, configure VPC connectivity properly rather than opening it up.</li>

<li>The Glue role needs Secrets Manager read plus the EC2 network interface permissions when a VPC is involved. A missing <code>ec2:CreateNetworkInterface</code> shows up as a connection failure that looks like a network problem.</li>

<li>Private certificate authorities need the chain in a form Glue accepts. If you are wiring this up yourself, expect it to take a pass or two.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Choosing your path: Glue jobs, zero-ETL, or something else</h2>



<p class="wp-block-paragraph">The same connector is reachable through more than one product surface, and the choice is really about how much state you want to own.</p>



<ul class="wp-block-list">
<li><strong>Glue Studio or script jobs.</strong> You control page size, filters, output format, transformation and scheduling, and you own delta token persistence. Pick this when you need real transformation in flight, or when the merge logic has to live next to the extraction.</li>

<li><strong>Glue zero-ETL integrations.</strong> AWS manages token state internally, runs the initial full load and then the incremental cycle, and preserves the token on failure so retries do not lose data. Less code, less control. Worth noting that once an incremental update field is chosen for a connection and entity, a later integration on the same pair cannot pick a different one.</li>

<li><strong>Amazon AppFlow.</strong> Simpler, flow-shaped, well suited to modest volumes and quick prototypes. Less appropriate once you need Spark-scale processing.</li>

<li><strong>Commercial replication tools.</strong> Fivetran, Airbyte and SAP-specialist vendors all play here. They cost money and add a vendor, and in exchange they absorb exactly the state management this post is about. If your team has no SAP depth, that trade is often correct.</li>
</ul>



<p class="wp-block-paragraph">On cost, Glue bills for the resources a job consumes for as long as it runs, so the levers are the same as any Spark workload: fewer round trips, less data pulled, fewer idle executors, and not re-reading history you already have. Rates change and vary by region, so model it against current pricing rather than a figure from a blog post. If your incremental run is pulling the same volume as your initial load, the delta mechanism is not working and the bill is the symptom.</p>



<p class="wp-block-paragraph">Also worth writing back. The connector supports write operations to SAP OData, using <code>INSERT</code> and <code>UPDATE</code>, with <code>ID_FIELD_NAMES</code> required on update to identify the external ID field. Useful for enrichment loops, and something to treat with a good deal more caution than a read.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">&#8220;Could not open data access via extraction API RODPS_REPL_ODP_OPEN&#8221;</h3>



<p class="wp-block-paragraph">Your token is invalid or expired. Find out why before you re-initialise, because re-initialising destroys the evidence. Check ODQMON for a recent initialisation on that collection and user, check whether another job or a test program shares the credentials, and check whether the request you wanted has aged out of retention. Then re-initialise deliberately and reconcile the gap.</p>



<h3 class="wp-block-heading">The job succeeds but returns no rows</h3>



<p class="wp-block-paragraph">Most often a stale or wrong token, a filter predicate that matches nothing, or CDC never having been initialised so the connector has nothing to hand back. Run once without <code>ENABLE_CDC</code> against a small filtered slice and confirm you get data at all. That separates &#8220;no changes&#8221; from &#8220;not reading&#8221;.</p>



<h3 class="wp-block-heading">Pagination never terminates</h3>



<p class="wp-block-paragraph">Usually the SAP side enforcing its own page limit under your requested size, so the connector keeps requesting more. Reduce the page size, verify the filter is actually being applied server side rather than after retrieval, and have Basis confirm what Gateway is capping.</p>



<h3 class="wp-block-heading">Row counts drift over time</h3>



<p class="wp-block-paragraph">Almost always unhandled <code>DML_STATUS = DELETED</code> records, or a token invalidation that silently turned an incremental into a partial. Count distinct business keys in your target against a direct count in SAP for a bounded date range. Make that comparison a scheduled check, not something you run when someone complains.</p>



<h3 class="wp-block-heading">Duplicates after a rerun</h3>



<p class="wp-block-paragraph">A delta was consumed, the write failed, and the token advanced anyway. Store the token only after the write commits, never before. If you are appending rather than merging, add a dedupe on business key ordered by <code>GLUE_FETCH_SQ</code>.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Sharing one SAP service user across multiple pipelines, which couples their delta tokens together.</li>

<li>Automatic re-initialisation in an exception handler. It converts a loud failure into permanent silent drift.</li>

<li>Writing the token-carrier row into the target table as if it were data.</li>

<li>Appending CDC output and ignoring <code>DML_STATUS</code>.</li>

<li>Persisting the token before the write succeeds.</li>

<li>Assuming an entity is ODP-enabled because it looks analytical. Confirm it, because the incremental mechanism differs.</li>

<li>Leaving test subscriptions from <code>RODPS_REPL_TEST</code> registered against the queue.</li>

<li>Building the pipeline without ever asking Basis what the ODQ retention window is.</li>
</ul>



<h2 class="wp-block-heading">Best practices for SAP data integration with AWS Glue</h2>



<ul class="wp-block-list">
<li>Treat the delta token as production state. Versioned storage, a documented recovery procedure, and a written-down answer to &#8220;how do we backfill&#8221;.</li>

<li>One service user per pipeline, one connection per environment. Isolation is the cheapest fix for the worst failure.</li>

<li>Land raw, merge second. Keep the untouched connector output so you can replay a merge without going back to SAP.</li>

<li>Alert on absence, not just errors. Zero rows for several consecutive runs against a busy entity should page someone. Push Glue metrics into CloudWatch and forward them wherever you already look, whether that is Grafana Cloud, Datadog or your own Prometheus.</li>

<li>Reconcile on a schedule against SAP for a bounded window, and record the result. Drift you can see is a bug; drift you cannot see is an incident.</li>

<li>Start with a filtered slice and a small page size while you are building. Get it correct, then tune throughput.</li>

<li>Write down what each field means while the functional owner is still on the call. SAP field names do not explain themselves later.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does the AWS Glue SAP OData connector fall under SAP Note 3255746?</h3>



<p class="wp-block-paragraph">The note restricts the RFC modules of the ODP Data Replication API. The Glue connector uses OData over HTTPS against an activated Gateway service, which is the interface SAP directs customers toward. Read the current revision of the note yourself and get it confirmed with your SAP account team before you rely on any summary, including this one.</p>



<h3 class="wp-block-heading">Where should I store the delta token?</h3>



<p class="wp-block-paragraph">Anywhere durable and versioned that your job can write to transactionally: DynamoDB, Parameter Store, or S3 with versioning. What matters more than the store is the ordering. Write it only after the target write commits, and keep the previous value so you can roll back after a bad run.</p>



<h3 class="wp-block-heading">Can I run two Glue jobs against the same SAP entity?</h3>



<p class="wp-block-paragraph">Yes, but give them separate SAP service users. A delta token is scoped to the service collection and the user, so two jobs on one user will invalidate each other&#8217;s tokens whenever either re-initialises.</p>



<h3 class="wp-block-heading">Why do deleted SAP records still appear in my data lake?</h3>



<p class="wp-block-paragraph">Because the connector reports them rather than applying them. Deletes come back as rows with <code>DML_STATUS</code> set to <code>DELETED</code>. If your pipeline appends without a merge, those rows sit in the target looking like live data. Merge into Iceberg, or filter through a current-state view that every consumer actually uses.</p>



<h3 class="wp-block-heading">What page size should I use?</h3>



<p class="wp-block-paragraph">The default is 50,000 and the ceiling is 500,000. Larger pages mean fewer round trips but bigger responses for SAP to assemble, and Gateway or a load balancer may cut them short. Start at the default, raise it while watching for timeouts, and agree the ceiling with your Basis team.</p>



<h3 class="wp-block-heading">Can I partition an ODP extraction for speed?</h3>



<p class="wp-block-paragraph">Not with record-based partitioning. ODP entities handle pagination through skip and delta tokens, so concurrency is forced to null there regardless of your input. Field-based partitioning on a bounded range is available for non-ODP OData entities. For ODP, tune page size and narrow the entity instead.</p>



<h3 class="wp-block-heading">Can Glue write data back into SAP?</h3>



<p class="wp-block-paragraph">Yes. The SAP OData connector supports insert and update operations, with the external ID field named through <code>ID_FIELD_NAMES</code> on updates. Test that against a sandbox client with a functional owner watching, because a bad write into an ERP system is a very different class of incident to a bad read.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">SAP data integration with AWS Glue is not hard because the connector is difficult to configure. Configuration takes an afternoon. It is hard because the correctness of your pipeline depends on a token held in a queue on a system your team does not administer, subject to a retention policy you may not have been told about, and shared with every other subscriber using the same credentials.</p>



<p class="wp-block-paragraph">Design for that. Isolate the users, persist the token deliberately, apply deletes rather than collecting them, and alert on silence as loudly as you alert on errors. Do that and the connector is a good piece of engineering that will run for a long time without your attention. Skip it and you get a job that stays green while the numbers quietly stop being true.</p>



<h2 class="wp-block-heading">Need a hand with an SAP to AWS pipeline?</h2>



<p class="wp-block-paragraph">This is the kind of work I take on regularly. If you are standing one up, or you have one that has started drifting, I can help with:</p>



<ul class="wp-block-list">
<li>Designing delta token state management that survives failed runs, redeploys and concurrent pipelines.</li>

<li>Building merge logic that applies <code>DML_STATUS</code> deletes correctly into Iceberg or Redshift instead of accumulating ghost rows.</li>

<li>Diagnosing <code>RODPS_REPL_ODP_OPEN</code> failures, silent zero-row runs and row count drift, and working out how far back the damage goes.</li>

<li>Setting up SAP OData connections properly: Secrets Manager, least-privilege IAM, VPC connectivity and private certificate authorities.</li>

<li>Tuning page size, filters and job sizing so incremental runs cost what an incremental run should cost.</li>

<li>Adding reconciliation and absence alerting so drift shows up in a dashboard rather than in a finance report.</li>
</ul>



<p class="wp-block-paragraph">Send me a job log, a connection config, or the row counts that stopped matching, and I will tell you what I think is going on.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/sap-data-integration-aws-glue/">SAP Data Integration with AWS Glue: The Failure Modes Nobody Demos</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Oracle to Aurora Migration: The Failures Nobody Catches Until Cutover</title>
		<link>https://john-nessime.com/blog/devops/oracle-to-aurora-migration/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 17 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Aurora]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS DMS]]></category>
		<category><![CDATA[Change Data Capture]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Cutover Planning]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Database Migration]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Oracle]]></category>
		<category><![CDATA[PostgreSQL]]></category>
		<category><![CDATA[Schema Conversion]]></category>
		<category><![CDATA[SQL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=218</guid>

					<description><![CDATA[<p>Most Oracle to Aurora migrations don't fail loudly. They finish green, pass validation, and quietly hand you truncated LOBs, sequences stuck at 1, and empty strings that used to be NULL. Here are the four failure families to check before you cut over, with the SQL and task settings that catch them.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/oracle-to-aurora-migration/">Oracle to Aurora Migration: The Failures Nobody Catches Until Cutover</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Three weeks after cutover, someone in finance sends you a spreadsheet. Forty-one invoice records have a description field that stops mid-sentence. Not blank, not garbled. Just shorter than it used to be.</p>



<p class="wp-block-paragraph">So you open the AWS DMS console. The task is green. Full load completed, change data capture running, zero errors. Data validation passed on every table in scope.</p>



<p class="wp-block-paragraph">Both of those things are true at the same time, and that is the part that catches people out. An Oracle to Aurora migration almost never fails with a stack trace and a rollback. It fails with a green dashboard and a handful of rows that are subtly, permanently wrong.</p>



<p class="wp-block-paragraph">This post is not a walkthrough of the console wizard. AWS documents that well enough. It covers the four families of failure I would go looking for before signing off on an Oracle to Aurora migration: large objects that arrive truncated, change capture that records less than you assume, the semantic gaps between Oracle and PostgreSQL that compile cleanly and behave differently, and the database state that DMS simply does not move for you. Then how to validate properly, and what to do when a task stalls.</p>



<h2 class="wp-block-heading">Pick the Aurora engine before you pick the migration tool</h2>



<p class="wp-block-paragraph">Amazon Aurora comes in a PostgreSQL-compatible edition and a MySQL-compatible edition, and the choice shapes everything downstream. Do not let it get decided by whoever on the team happens to like MySQL.</p>



<p class="wp-block-paragraph">Aurora PostgreSQL is the realistic default coming from Oracle, and the reason is procedural code. PL/SQL and PL/pgSQL are close enough that automated conversion gets you a long way: packages, cursors, exceptions, custom types, and analytic functions all have recognisable counterparts. PostgreSQL also has the <code>orafce</code> extension, which reimplements a chunk of Oracle&#8217;s built-in functions and packages inside a compatibility schema, so a lot of converted code can keep calling things it already calls.</p>



<p class="wp-block-paragraph">The honest case for Aurora MySQL: if the Oracle database is essentially a table store with the business logic living in the application tier, and your team already runs MySQL in production, you will move faster on the engine you can operate at 3am. Migration difficulty is real, but so is the cost of running a database nobody on the rota understands. If your schema has no packages, no triggers worth keeping and no PL/SQL, that argument holds.</p>



<p class="wp-block-paragraph">Where it falls apart is the moment you find twelve thousand lines of PL/SQL that someone will have to rewrite by hand. At that point the engine choice has stopped being an operational preference and become a rewrite project.</p>



<h3 class="wp-block-heading">Schema conversion: SCT or the DMS console</h3>



<p class="wp-block-paragraph">There are two ways to convert the schema. The AWS Schema Conversion Tool is a desktop Java application that connects to both databases, reads the source objects, and emits target DDL plus an assessment report of everything it could not convert. DMS Schema Conversion is the same idea run from the DMS console using instance profiles and data providers, with no local install and no JDBC drivers to manage.</p>



<p class="wp-block-paragraph">Start with the console version. It is less setup and the assessment report is the thing you actually want early: a list of objects with action items, so you can size the manual work before committing to a date. Fall back to the desktop tool when you need something the console flow does not cover, such as converting application SQL files or SQL*Plus scripts.</p>



<p class="wp-block-paragraph">Either way, treat the assessment report as a project plan, not a progress bar. The red items are the schedule.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: LOB columns that arrive shorter than they left</h2>



<p class="wp-block-paragraph">This is the invoice description problem, and it is the single most common way an Oracle to Aurora migration loses data without telling anyone loudly.</p>



<p class="wp-block-paragraph">DMS gives you three ways to handle CLOB, NCLOB and BLOB columns:</p>



<ul class="wp-block-list">
<li><strong>Full LOB mode</strong> moves every LOB regardless of size. DMS has no idea how big they are, so it moves them one at a time, piece by piece. Correct, and slow.</li>



<li><strong>Limited LOB mode</strong> asks you for a maximum size up front. DMS pre-allocates memory and loads LOBs in bulk, which is dramatically faster. Anything larger than the limit is <em>truncated to the limit</em>, and a warning goes into the task log.</li>



<li><strong>Inline LOB mode</strong> sets a threshold. LOBs under it go inline with the row; larger ones fall back to the full LOB path with a lookup against the source. Good when most of your LOBs are small and a few are not.</li>
</ul>



<p class="wp-block-paragraph">Read the middle one again. Limited LOB mode is the default answer to &#8220;the full load is too slow&#8221;, it is the mode most tutorials use, and it silently clips your data. The warning lands in the task log, mixed in with everything else, and nobody is reading task logs line by line on a table with four million rows.</p>



<p class="wp-block-paragraph">Worse, the task still reports success. Truncation is not an error condition in DMS. It is expected behaviour for the mode you asked for.</p>



<h3 class="wp-block-heading">Measure before you set the limit</h3>



<p class="wp-block-paragraph">The fix is boring: go and find out how big your LOBs actually are, per column, on the source. Run this against Oracle for every LOB column in scope.</p>



<pre class="wp-block-code"><code>-- Largest LOB values in a single column, biggest first.
-- dbms_lob.getlength returns length in characters for CLOB/NCLOB,
-- bytes for BLOB. Check the top rows, not the average.
SELECT dbms_lob.getlength(description) AS lob_length
FROM   app.invoices
ORDER  BY dbms_lob.getlength(description) DESC
FETCH  FIRST 10 ROWS ONLY;

-- Same thing expressed in KB, which is the unit the DMS
-- "Max LOB size (K)" setting expects.
SELECT MAX(dbms_lob.getlength(description)) / 1024 AS max_kb
FROM   app.invoices;</code></pre>



<p class="wp-block-paragraph">Two things to know once you have the numbers.</p>



<p class="wp-block-paragraph">First, with Oracle as a source, DMS treats LOBs as VARCHAR data wherever it can, because bulk-fetching them is much faster than the LOB API. Oracle&#8217;s VARCHAR ceiling is 32K, so a limited LOB size under 32K is where the mode performs best. If your real maximum is comfortably under that, limited LOB mode is genuinely the right call.</p>



<p class="wp-block-paragraph">Second, memory. During full load, DMS pre-allocates roughly the max LOB size multiplied by the commit rate multiplied by the number of LOB columns. Push the limit up and that product grows fast. When the replication instance cannot allocate it, it starts swapping, and your &#8220;faster&#8221; mode becomes slower than full LOB mode. If limited LOB mode is dragging, drop the commit rate before you drop the size limit.</p>



<h3 class="wp-block-heading">Split the LOB tables into their own task</h3>



<p class="wp-block-paragraph">The pattern I reach for first: one task with limited LOB mode for the bulk of the schema, a second task for the handful of tables with genuinely large LOBs, using full or inline mode. You can also override task-level LOB settings per table inside the table mapping rules, which keeps it to a single task if you prefer.</p>



<pre class="wp-block-code"><code>{
  "TargetMetadata": {
    "SupportLobs": true,
    "FullLobMode": false,
    "LimitedSizeLobMode": true,
    "LobMaxSize": 16,
    "InlineLobMaxSize": 0,
    "LobChunkSize": 64,
    "BatchApplyEnabled": false
  }
}</code></pre>



<p class="wp-block-paragraph">That is limited mode with a 16 KB ceiling. To use inline mode instead, set <code>FullLobMode</code> to true and give <code>InlineLobMaxSize</code> a non-zero value, which is the threshold below which LOBs travel inline.</p>



<p class="wp-block-paragraph">One requirement that bites during ongoing replication: for CDC, tables with LOB columns need a primary key, because DMS looks the LOB value up in the source rather than reading it out of the redo stream. Full load does not care. CDC does. A LOB table with no primary key will load fine and then fail to replicate changes.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family two: CDC that captures less than you think</h2>



<p class="wp-block-paragraph">Ongoing replication is what buys you a short cutover window. It is also where the assumptions hide.</p>



<p class="wp-block-paragraph">DMS reads Oracle&#8217;s redo logs. For that to work at all, the source has to be in ARCHIVELOG mode with enough retention that DMS can still find the logs it needs after a pause, a network blip or an overnight stall. Retention is the one people get wrong: a task that stops for six hours against a two-hour retention window does not resume, it fails, and you restart from a full load.</p>



<h3 class="wp-block-heading">Supplemental logging is not optional</h3>



<p class="wp-block-paragraph">Oracle&#8217;s redo logs, by default, contain enough information for Oracle to recover the database. That is not the same as enough information to reconstruct a row-level change for a different database. Supplemental logging is what closes that gap.</p>



<pre class="wp-block-code"><code>-- Minimal supplemental logging at the database level.
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;

-- Identification key logging: writes every column of a row's
-- primary key into redo on UPDATE, even when the key didn't change.
-- Without this, DMS cannot reliably match the row on the target.
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS;

-- Per-table alternative, if you only need a subset of tables
-- and want to keep the source overhead down.
ALTER TABLE app.invoices ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS;</code></pre>



<p class="wp-block-paragraph">The case that actually bites is a table with no primary key and no unique index. For those, key-level logging gives DMS nothing to work with, and it needs supplemental logging on <em>all</em> columns so the before-and-after image of the row lands in redo. Miss that, and DMS cannot build a safe UPDATE or DELETE statement for the target. You will not notice during full load. You will notice when an update on the source produces no change on the target, or changes the wrong row.</p>



<p class="wp-block-paragraph">DMS ships premigration assessments that check exactly this. Run them. There are individual checks for supplemental logging on keyed tables, supplemental logging on all columns for unkeyed tables, ARCHIVELOG configuration, the CDC method in use, target permissions, and triggers enabled on the target. It takes minutes and it replaces a category of guesswork.</p>



<h3 class="wp-block-heading">LogMiner or Binary Reader</h3>



<p class="wp-block-paragraph">DMS can read redo two ways. Oracle LogMiner is an Oracle-side API; DMS calls it and Oracle does the parsing. Binary Reader is DMS&#8217;s own parser reading the redo files directly.</p>



<p class="wp-block-paragraph">If you specify nothing, you get LogMiner. That is usually correct. It needs fewer privileges, it handles ASM without extra plumbing, and it supports Oracle features like encryption and compression that the direct parser does not cover. The grants are small:</p>



<pre class="wp-block-code"><code>GRANT EXECUTE ON DBMS_LOGMNR TO dms_user;
GRANT SELECT ON V_$LOGMNR_LOGS TO dms_user;
GRANT SELECT ON V_$LOGMNR_CONTENTS TO dms_user;
GRANT LOGMINING TO dms_user;</code></pre>



<p class="wp-block-paragraph">Binary Reader earns its keep in one situation: a source generating redo faster than LogMiner can chew through it, or several concurrent migration tasks against the same database. LogMiner runs inside the source instance, so its CPU cost lands on the database you are trying not to disturb. Binary Reader shifts that work to the replication instance.</p>



<p class="wp-block-paragraph">The trade-off is setup. Binary Reader needs additional privileges and file-level access to the redo logs, which means either a directory the replication instance can reach or a copy step. If your source is a busy production Oracle instance and CDC latency is climbing steadily rather than spiking, that is the signal to switch. Otherwise stay on LogMiner and spend the complexity budget elsewhere.</p>



<p class="wp-block-paragraph">Whichever you pick, put CDC latency on a dashboard for the whole replication window, not just cutover day. The DMS task metrics publish to CloudWatch, and pulling them into whatever you already run for alerting, whether that is Grafana Cloud, Datadog or New Relic, is worth the twenty minutes. Source latency climbing while target latency stays flat means you are not reading redo fast enough. Both climbing together usually means the target cannot apply fast enough.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family three: the semantics Oracle and PostgreSQL do not share</h2>



<p class="wp-block-paragraph">Schema conversion produces code that compiles. Compiling is not the same as behaving identically, and this is where a migration goes wrong in ways no tool flags.</p>



<h3 class="wp-block-heading">Empty string is not NULL</h3>



<p class="wp-block-paragraph">Oracle treats the empty string and NULL as the same thing. PostgreSQL does not. This one difference propagates into comparisons, concatenation, unique constraints, and any procedural code that checks whether a value is &#8220;empty&#8221;.</p>



<pre class="wp-block-code"><code>-- PostgreSQL
SELECT '' IS NULL;        -- false
SELECT NULL || 'abc';     -- NULL
SELECT '' || 'abc';       -- abc

-- Oracle
SELECT CASE WHEN '' IS NULL THEN 'yes' ELSE 'no' END FROM dual;  -- yes
SELECT NULL || 'abc' FROM dual;                                  -- abc
SELECT '' || 'abc' FROM dual;                                    -- abc</code></pre>



<p class="wp-block-paragraph">Every one of those lines is a place a report can quietly change its numbers. Unique constraints are worse. Oracle stores an empty string as NULL, and NULLs are not compared for uniqueness, so it will happily accept many rows with an empty value in a uniquely constrained column. PostgreSQL sees zero-length strings, which are equal to each other, so the second one is rejected. Same schema, same data, different outcome, and the failure surfaces as an application error nobody can reproduce locally.</p>



<p class="wp-block-paragraph">There is no switch for this. Either the application changes, or you wrap the affected functions, or you normalise on write with a trigger or a check constraint. Decide which, per column, before cutover. The orafce extension helps with function behaviour but it is not a global compatibility mode, and enabling the parts that change NULL semantics has its own consequences.</p>



<h3 class="wp-block-heading">The rest of the list</h3>



<ul class="wp-block-list">
<li><strong>ROWID</strong> has no PostgreSQL equivalent. DMS Schema Conversion can emulate it with a bigint or a character varying column, but if application code depends on ROWID ordering or reuse semantics, emulation will not save you.</li>



<li><strong>SYSDATE and time zones.</strong> Conversion settings let you choose whether to emulate Oracle&#8217;s time zone handling or use native PostgreSQL behaviour. Native is faster. It is only safe if the database and the application genuinely run in the same time zone.</li>



<li><strong>TO_CHAR, TO_DATE, TO_NUMBER format masks.</strong> Oracle accepts parameters PostgreSQL does not. The converter emulates them by default, which is correct and slower. Turning emulation off is a per-codebase decision, not a global one.</li>



<li><strong>DUAL</strong> does not exist. PostgreSQL does not need a FROM clause, so most references just get dropped, but any dynamic SQL that builds a query string around DUAL needs finding.</li>



<li><strong>Synonyms</strong> have no direct equivalent. Search path adjustments and views cover most cases.</li>



<li><strong>Materialized views</strong> can be converted to real materialized views or to plain tables. The refresh semantics differ, so check what the reporting layer expects.</li>



<li><strong>NUMBER without a scale.</strong> An Oracle NUMBER column with no scale specified converts to something with a default, and if the source holds values wider than that default you get rounding or overflow. There is a premigration assessment specifically for this.</li>
</ul>



<p class="wp-block-paragraph">One data-level quirk worth knowing: if an Oracle column contains a NULL character (hex U+0000), DMS converts it to a space (U+0020) on a PostgreSQL target, because PostgreSQL cannot store it in a text column. Rare, but if you are migrating anything that stuffed binary into a VARCHAR, that is a real change to your data.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family four: the state DMS never moves</h2>



<p class="wp-block-paragraph">DMS moves rows. It does not move everything that makes a database work, and the gaps are not obvious until you point traffic at the new cluster.</p>



<h3 class="wp-block-heading">Sequences</h3>



<p class="wp-block-paragraph">Schema conversion creates the sequences. DMS copies the rows. Nothing advances the sequence counters to match the data that just arrived, so your target sequences sit at their starting value while the table already contains eight million rows.</p>



<p class="wp-block-paragraph">The first insert after cutover collides with an existing primary key. Then the second. Then every write for the next eight million attempts.</p>



<p class="wp-block-paragraph">Fix this after you stop replication and before you open traffic, not earlier, or you will do it twice.</p>



<pre class="wp-block-code"><code>-- Advance a sequence to match the data already loaded.
-- Run once per sequence, after CDC has stopped.
SELECT setval('app.invoices_id_seq',
              (SELECT max(id) FROM app.invoices));

-- Confirm before you open the doors.
SELECT last_value FROM app.invoices_id_seq;</code></pre>



<h3 class="wp-block-heading">Triggers on the target</h3>



<p class="wp-block-paragraph">If your converted schema has triggers and they are enabled during the load, they fire on every row DMS inserts. Audit tables get populated with a migration&#8217;s worth of fake activity. Derived columns get recalculated against half-loaded data. Anything that writes to another table doubles your write volume.</p>



<p class="wp-block-paragraph">Disable target triggers for the load, re-enable at cutover. There is a premigration assessment that checks for enabled triggers on target tables in task scope, which is a decent backstop against forgetting.</p>



<h3 class="wp-block-heading">Secondary indexes and foreign keys</h3>



<p class="wp-block-paragraph">Every secondary index present during full load is an index being maintained row by row while you are trying to move data as fast as possible. Foreign keys are worse, because DMS does not guarantee parent-before-child load order across tables.</p>



<p class="wp-block-paragraph">Standard sequence: drop or disable secondary indexes and foreign key constraints before full load, recreate them afterwards, then let CDC catch up. Keep primary keys, since CDC needs them. Script the recreation and check it into version control alongside everything else, because rebuilding forty indexes from memory at 2am is not a plan.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Validation you can actually trust</h2>



<p class="wp-block-paragraph">DMS data validation compares source and target row by row and reports mismatches. It supports Oracle and PostgreSQL-compatible endpoints in both directions, so an Oracle to Aurora PostgreSQL path is covered. Turn it on.</p>



<p class="wp-block-paragraph">Then understand what it will not tell you.</p>



<ul class="wp-block-list">
<li>It needs a primary key or unique index on both sides. Unkeyed tables are not validated, and those are exactly the tables most likely to have gone wrong.</li>



<li>Collation and sort order differ between Oracle and PostgreSQL. When they disagree, validation reports failures on records that are actually fine, and the noise trains people to ignore the report.</li>



<li>It does not run when the migration uses custom filtering, or when you are consolidating several source databases into one target.</li>



<li>It costs real resources. Validation issues its own queries against both databases, on top of the migration traffic. Budget for that on the source.</li>



<li>For LOB columns it compares checksums rather than values, using DBMS_CRYPTO on the Oracle side.</li>
</ul>



<p class="wp-block-paragraph">Because validation runs alongside migration, a task that stops takes validation with it. Validation-only tasks solve that: same endpoints, same table mappings, no data movement, running independently of the migration task. Use one when the validation load is hurting the source and you want to run it off-peak, or when you want validation to survive a migration task restart.</p>



<p class="wp-block-paragraph">None of that replaces your own reconciliation. Before cutover, run business-level checks on both sides: rows per table, sums of every monetary column, min and max of every date column, distinct counts on anything you join on. Those catch things row comparison does not, because they answer the question the business actually asks.</p>



<p class="wp-block-paragraph">And specifically for the LOB problem, compare maximum lengths, not just row counts.</p>



<pre class="wp-block-code"><code>-- On Aurora PostgreSQL, after the load.
-- Compare this against dbms_lob.getlength on the Oracle source.
SELECT count(*)                 AS row_count,
       max(length(description)) AS max_len,
       count(*) FILTER (WHERE description IS NULL) AS null_count
FROM   app.invoices;</code></pre>



<p class="wp-block-paragraph">If the target maximum is suspiciously round, exactly 16384 or exactly 32768, you have found your truncation.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting an Oracle to Aurora migration that stalls</h2>



<h3 class="wp-block-heading">Full load runs, then one table sits at zero</h3>



<p class="wp-block-paragraph">Almost always LOBs. Full LOB mode moves large objects one at a time with a source lookup per value, and on a table with a million large CLOBs that is a very long time with no visible progress. Check the table statistics for that table, then check whether it has LOB columns. Move it to its own task with limited or inline mode and an appropriate size.</p>



<h3 class="wp-block-heading">CDC latency climbs and never comes down</h3>



<p class="wp-block-paragraph">Look at which latency is climbing. Source latency means redo is arriving faster than DMS can read it, which is the LogMiner to Binary Reader conversation, or a redo volume problem on the source. Target latency means Aurora cannot apply fast enough, which points at indexes still present, triggers still enabled, or an undersized writer instance.</p>



<h3 class="wp-block-heading">Task fails on restart with missing archived logs</h3>



<p class="wp-block-paragraph">Retention. The logs DMS needed were removed before it came back. Increase archived log retention on the source to comfortably exceed your worst realistic outage, then restart from a fresh full load for the affected tables. There is no recovering the gap.</p>



<h3 class="wp-block-heading">Updates on the source do not appear on the target</h3>



<p class="wp-block-paragraph">Supplemental logging, nine times out of ten. Check whether the affected table has a primary key or unique index, and whether logging is configured at the right level for that answer. The premigration assessments cover both cases.</p>



<h3 class="wp-block-heading">Errors about a unique index on the target</h3>



<p class="wp-block-paragraph">DMS does not support replication to a table with a unique index built on a <code>coalesce</code> expression. If schema conversion produced one, or someone added it by hand to work around the empty-string problem, replication to that table will not work. Restructure the constraint.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Accepting the default limited LOB size without measuring the source. This is the truncation bug, and it has a specific, preventable cause.</li>



<li>Treating a green task and a passed validation as proof of correctness. Both are true when data has been truncated.</li>



<li>Skipping the premigration assessments because the wizard let you continue without them.</li>



<li>Forgetting to advance sequences, then discovering it under production write load.</li>



<li>Leaving triggers enabled on the target during full load and polluting audit tables.</li>



<li>Assuming schema conversion handled empty strings. It converted the syntax, not the semantics.</li>



<li>Setting archived log retention to the length of a normal outage rather than a bad one.</li>



<li>Doing the first end-to-end rehearsal on cutover night.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ol class="wp-block-list">
<li><strong>Run the assessment report first and size the manual work from it.</strong> The red items are your timeline, and they are better known in week one than week nine.</li>



<li><strong>Inventory LOB columns before configuring anything.</strong> Maximum size per column, in KB, written down.</li>



<li><strong>Separate the LOB-heavy tables into their own task.</strong> Different tables want different modes, and one global setting will always be wrong for someone.</li>



<li><strong>Enable supplemental logging deliberately, per table class.</strong> Key-level for keyed tables, all-columns for unkeyed ones.</li>



<li><strong>Drop secondary indexes and foreign keys for the load.</strong> Script the recreation, version it, test it.</li>



<li><strong>Rehearse the whole cutover at least twice</strong>, including the sequence advance and the trigger re-enable. A scratch environment for the dry run does not have to be expensive: a temporary Aurora cluster you tear down afterwards, or a cheap dedicated box from somewhere like InterServer or Hetzner running a trimmed source copy, both work.</li>



<li><strong>Run your own reconciliation alongside DMS validation.</strong> Counts, sums, min and max dates, maximum LOB lengths.</li>



<li><strong>Keep the Oracle source running and readable for a while after cutover.</strong> The bugs that surface at week three are the ones you can only diagnose by comparing against the original.</li>
</ol>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Should I migrate Oracle to Aurora PostgreSQL or Aurora MySQL?</h3>



<p class="wp-block-paragraph">Aurora PostgreSQL for anything with meaningful PL/SQL, because PL/pgSQL is close enough that automated conversion does most of the work, and the orafce extension covers a chunk of the rest. Aurora MySQL is defensible when the schema is plain tables, the logic lives in the application, and your team already operates MySQL. The deciding question is how many lines of procedural code you would have to rewrite by hand.</p>



<h3 class="wp-block-heading">Can AWS DMS migrate stored procedures and packages?</h3>



<p class="wp-block-paragraph">No. DMS moves data. Schema and code conversion is a separate job, handled by DMS Schema Conversion in the console or the AWS Schema Conversion Tool on the desktop. Both convert most objects automatically and produce an action-item list for the rest.</p>



<h3 class="wp-block-heading">Why is my DMS task green when data is missing?</h3>



<p class="wp-block-paragraph">Because truncation in limited LOB mode is expected behaviour, not an error. DMS writes a warning to the task log and carries on. Task status reflects whether the task is running, not whether the data is faithful. Compare maximum LOB lengths between source and target to catch it.</p>



<h3 class="wp-block-heading">Do I need downtime for an Oracle to Aurora migration?</h3>



<p class="wp-block-paragraph">Some, but it can be short. Full load plus CDC means the target stays current while you test, so the outage is limited to stopping writes, letting CDC drain, advancing sequences, re-enabling triggers, recreating indexes and repointing the application. Rehearse it and that usually fits a maintenance window rather than a weekend.</p>



<h3 class="wp-block-heading">Does DMS migrate sequences?</h3>



<p class="wp-block-paragraph">It copies row data, not sequence state. Schema conversion creates the sequence objects; you advance them yourself after stopping replication, using <code>setval</code> against the current maximum key value. Skipping this produces primary key collisions on the first write after cutover.</p>



<h3 class="wp-block-heading">LogMiner or Binary Reader for Oracle CDC?</h3>



<p class="wp-block-paragraph">LogMiner unless you have a reason. It is the default, needs fewer privileges, and handles ASM, encryption and compression without extra work. Move to Binary Reader when redo volume is high enough that LogMiner cannot keep up, when you are running several tasks against the same source, or when the CPU cost of parsing on the source database is unacceptable.</p>



<h3 class="wp-block-heading">Will DMS data validation catch every problem?</h3>



<p class="wp-block-paragraph">No. It needs a primary key or unique index on both sides, it can report false failures where Oracle and PostgreSQL collations disagree, and it does not run with custom filtering or multi-source consolidation. Treat it as one layer and add business-level reconciliation on top.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">An Oracle to Aurora migration does not usually announce its failures. The task goes green, validation passes, and the damage shows up weeks later as a truncated field, a duplicate key, a report whose totals moved slightly.</p>



<p class="wp-block-paragraph">So build the migration around proving correctness rather than proving completion. Measure your LOBs before you set a limit. Configure supplemental logging for the tables that have no key, not just the ones that do. Assume the semantic gaps between Oracle and PostgreSQL are still there after conversion, because they are. Advance the sequences. Then reconcile at the level the business cares about, not just row by row.</p>



<p class="wp-block-paragraph">Green means the pipe is open. It does not mean what came out the other end is what went in.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your Oracle to Aurora migration?</h2>



<p class="wp-block-paragraph">Most of the work on these projects is not running the wizard. It is finding the handful of things that will be wrong afterwards. That is the part I help with:</p>



<ul class="wp-block-list">
<li>Reviewing DMS task settings and table mappings for LOB truncation risk, per column, before you run a full load you cannot easily repeat.</li>



<li>Working through the schema conversion assessment report and turning the red items into an actual estimate rather than a wall of warnings.</li>



<li>Setting up supplemental logging and CDC correctly for tables with no primary key, including the LogMiner versus Binary Reader call for your redo volume.</li>



<li>Writing the cutover runbook: index recreation scripts, sequence advance, trigger re-enable, ordered, tested, and timed against a real rehearsal.</li>



<li>Building the reconciliation layer that sits on top of DMS validation, so you can sign off on the data with something better than a green tick.</li>



<li>Post-cutover triage when something on Aurora behaves differently from Oracle and nobody can work out which layer changed.</li>
</ul>



<p class="wp-block-paragraph">If you have a task settings JSON, an assessment report, or a validation failure log you cannot make sense of, send it over and I will tell you what I see in it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/oracle-to-aurora-migration/">Oracle to Aurora Migration: The Failures Nobody Catches Until Cutover</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Zendesk Data Integration with AWS Glue Zero-ETL: The Delete Gap That Skews Your Numbers</title>
		<link>https://john-nessime.com/blog/technical-guides/zendesk-aws-glue-zero-etl/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 16 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Change Data Capture]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Customer Support Analytics]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Zendesk]]></category>
		<category><![CDATA[Zero-ETL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=223</guid>

					<description><![CDATA[<p>AWS Glue zero-ETL replicates seven Zendesk entities, but only three of them ever remove a row. Here is how that gap quietly skews CSAT and knowledge base counts, plus the three IAM layers to wire, the two settings you cannot change after creation, and the CloudWatch metrics that make drift visible before someone spots it in a meeting.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/zendesk-aws-glue-zero-etl/">Zendesk Data Integration with AWS Glue Zero-ETL: The Delete Gap That Skews Your Numbers</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The support lead pings you on a Tuesday. The CSAT figure on the exec dashboard is higher than the one in Zendesk Explore, and somebody noticed during the meeting. Nothing failed. The Glue integration is green. CloudWatch shows ingestion succeeding on every interval. And yet the warehouse and the source disagree, and have quietly disagreed for weeks.</p>



<p class="wp-block-paragraph">That divergence is not a bug. It is documented behaviour, sitting in one table in the AWS docs that almost everyone scrolls past. This post covers what Zendesk data integration with AWS Glue zero-ETL actually replicates, the entities where deletes silently never propagate, the configuration choices you cannot undo after creation, how to wire the three IAM roles involved, and how to monitor the thing so drift shows up as an alarm instead of a meeting.</p>



<h2 class="wp-block-heading">What Zendesk AWS Glue zero-ETL actually gives you</h2>



<p class="wp-block-paragraph">Zero-ETL is AWS&#8217;s name for managed replication. You create a Glue connection to Zendesk, pick the entities you want, pick a target, and Glue handles the initial snapshot, the schema mapping, and ongoing change data capture. No Spark job. No bookmark logic. No pagination code hammering the Zendesk API and no retry handling for it.</p>



<p class="wp-block-paragraph">Supported targets are a general purpose Amazon S3 bucket through the lakehouse architecture of Amazon SageMaker, S3 Tables through the same lakehouse, Redshift Managed Storage, or an Amazon Redshift data warehouse directly. Data lands as Apache Iceberg, which is what makes row-level updates and deletes viable on object storage at all.</p>



<p class="wp-block-paragraph">What you give up is control. No transform step, no server-side filter, no column pruning at ingest. You get the entity as the connector sees it and you shape it downstream. For most support analytics that trade is fine. It stops being fine the moment your compliance team asks why deleted records are still queryable.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The delete gap: four of seven entities never remove a row</h2>



<p class="wp-block-paragraph">AWS Glue zero-ETL supports seven Zendesk entities. Only three of them replicate deletes.</p>



<ul class="wp-block-list">
<li><strong>tickets</strong> &#8211; create, update and delete all propagate</li>



<li><strong>users</strong> &#8211; create, update and delete all propagate</li>



<li><strong>organizations</strong> &#8211; create, update and delete all propagate</li>



<li><strong>satisfaction-rating</strong> &#8211; create and update only, <strong>no delete</strong></li>



<li><strong>articles</strong> &#8211; create and update only, <strong>no delete</strong></li>



<li><strong>calls</strong> &#8211; create and update only, <strong>no delete</strong></li>



<li><strong>legs</strong> (call legs) &#8211; create and update only, <strong>no delete</strong></li>
</ul>



<p class="wp-block-paragraph">When a satisfaction rating disappears in Zendesk, the row stays in your lakehouse. Permanently. Nothing errors, <code>IngestionSucceeded</code> fires as usual, and the row count in that table only ever goes up.</p>



<p class="wp-block-paragraph">Now think about what your CSAT query does. It averages a score column across a set of rows. If removed ratings never leave the target, your average drifts further from the source every cycle, and the skew is rarely random because retracted or disputed ratings are exactly the ones most likely to be removed. The same mechanism inflates knowledge base article counts, call volume, and any call leg analysis you build. It stays invisible until two dashboards get compared side by side.</p>



<h3 class="wp-block-heading">How to design around it</h3>



<p class="wp-block-paragraph">You cannot make the connector track deletes it does not track. So stop trying, and split the problem instead.</p>



<ol class="wp-block-list">
<li><strong>Run two integrations, not one.</strong> Put tickets, users and organizations in a continuously synced integration. Put the four append-only entities in a second integration you can tear down and rebuild on a cadence that matches how much drift you can tolerate.</li>



<li><strong>Treat the append-only tables as an event log, not as state.</strong> Query them through a view that reconciles against a periodic authoritative count, rather than trusting row presence to mean the record still exists.</li>



<li><strong>Prefer status fields over row existence.</strong> Where an entity exposes a field describing its state, filter on that field downstream. Check which fields your entities actually return before you build on this, because coverage varies by entity.</li>



<li><strong>Reconcile on a schedule.</strong> Pull counts from the Zendesk API for the four entities and compare against the target. Alert on divergence, not on an absolute number.</li>
</ol>



<p class="wp-block-paragraph">Splitting the integration costs you a second set of IAM wiring and a second thing to monitor. It buys you the ability to rebuild the drifting half without touching the half that is correct. That is a good trade.</p>



<h2 class="wp-block-heading">Entity coverage is narrower than the Zendesk API</h2>



<p class="wp-block-paragraph">Seven entities is not the Zendesk API. Ticket comments, ticket metrics, ticket audits, groups, views, macros and SLA policy definitions are not part of the zero-ETL entity set. If your reporting needs first response time, the audit trail behind a status change, or custom field definitions, zero-ETL alone will not get you there.</p>



<p class="wp-block-paragraph">The useful part is that the Glue connection you create is reusable. The same connection backs a normal Glue ETL job, so you can run zero-ETL for the bulk entities and a scheduled Spark job for the ones it does not cover:</p>



<pre class="wp-block-code"><code># Read a Zendesk entity through the same connection
# used by the zero-ETL integration. ENTITY_NAME takes
# the entity name, not the label, e.g. "tickets".
zendesk_read = glueContext.create_dynamic_frame.from_options(
    connection_type="Zendesk",
    connection_options={
        "connectionName": "my-zendesk-connection",
        "ENTITY_NAME": "tickets",
        "API_VERSION": "v2"
    }
)</code></pre>



<p class="wp-block-paragraph">One schema detail worth knowing before you write a single downstream query: the connector converts struct and list types to strings. Zendesk custom fields arrive as arrays of objects, so they land as serialized text, not as a nested column you can address with dot notation. Partitioning is also not supported on the Zendesk source. Plan for a parse step in your silver layer rather than discovering it in a broken dashboard.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Two decisions you cannot change after creation</h2>



<p class="wp-block-paragraph">Most Glue settings are editable. These two are not, and getting them wrong means deleting the integration and starting over.</p>



<h3 class="wp-block-heading">Continuous sync versus on-demand snapshot</h3>



<p class="wp-block-paragraph">By default the integration syncs continuously. Enable the on-demand snapshot setting and it does a single one-time replication with no ongoing CDC. That setting is locked once the integration exists. If you enabled it for a proof of concept and then decided you wanted CDC, you are recreating the integration and re-running the full load.</p>



<h3 class="wp-block-heading">Refresh interval, and why your target choice locks it</h3>



<p class="wp-block-paragraph">The refresh interval controls how often CDC pulls run. It accepts anything from 15 minutes up to 8640 minutes, which is six days. Here is the part that catches people:</p>



<ul class="wp-block-list">
<li>If the target is <strong>Amazon Redshift</strong>, the refresh interval cannot be modified after creation.</li>



<li>For <strong>other targets</strong>, including the SageMaker lakehouse on S3, you can change it later.</li>
</ul>



<p class="wp-block-paragraph">So the target you pick is not only an architecture decision, it is a flexibility decision. If you are not certain what freshness the business actually needs, land in the lakehouse first and query Redshift over it. You keep the ability to tune the interval once you have real usage data instead of a guess made in week one.</p>



<p class="wp-block-paragraph">On cost: AWS does not charge separately for the integration itself. You pay for everything around it, target storage, Redshift or Athena query compute, S3 requests, Data Catalog usage and CloudWatch logs. A 15 minute interval on seven entities produces far more small-file churn and log volume than a 6 hour one. Set the interval from the freshness a human will actually act on, not from the smallest number the field accepts.</p>



<h2 class="wp-block-heading">Wiring the three roles</h2>



<p class="wp-block-paragraph">There are three separate permission layers here and they fail in different ways, which is why &#8220;check IAM&#8221; is useless advice on its own.</p>



<ul class="wp-block-list">
<li><strong>The Zendesk OAuth credential.</strong> The connector uses the authorization code grant, so you get redirected to Zendesk to log in and approve. You can rely on the Glue-managed client application and supply only your instance URL, or register your own OAuth app in the Zendesk admin center and provide your own client ID and secret. The resulting access token does not expire, which is convenient and also means a revoked app in Zendesk is something you will only discover through failed ingestions.</li>



<li><strong>The source role.</strong> This is what lets the integration read through the connection. It is a prerequisite for SaaS sources and it is the step most people miss, because creating the connection successfully does not mean the integration can use it.</li>



<li><strong>The target role and catalog policy.</strong> This is what lets the integration write. A misconfigured catalog resource policy puts the integration into <code>NEEDS_ATTENTION</code>, not into a clear error at creation time.</li>
</ul>



<p class="wp-block-paragraph">The source role policy looks like this. Note <code>glue:RefreshOAuth2Tokens</code>, which is the one people leave out and then spend an afternoon debugging:</p>



<pre class="wp-block-code"><code>{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "GlueConnections",
            "Effect": "Allow",
            "Action": [
                "glue:GetConnections",
                "glue:GetConnection"
            ],
            "Resource": [
                "arn:aws:glue:*:111122223333:catalog",
                "arn:aws:glue:us-east-1:111122223333:connection/*"
            ]
        },
        {
            "Sid": "GlueActionBasedPermissions",
            "Effect": "Allow",
            "Action": [
                "glue:ListEntities",
                "glue:RefreshOAuth2Tokens"
            ],
            "Resource": ["*"]
        },
        {
            "Sid": "CloudWatchLogging",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": ["*"]
        }
    ]
}</code></pre>



<p class="wp-block-paragraph">The trust policy allows <code>glue.amazonaws.com</code> to assume it. Then you attach the role to the connection, which is a CLI-only step and does not appear in the console flow:</p>



<pre class="wp-block-code"><code>aws glue create-integration-resource-property 
  --resource-arn arn:aws:glue:us-east-1:123456789012:connection/my-zendesk-connection 
  --source-processing-properties "{"RoleArn" : "arn:aws:iam::123456789012:role/zendesk-zetl-source"}" 
  --region us-east-1</code></pre>



<p class="wp-block-paragraph">If you skip this, the connection tests fine and the integration still cannot read. That mismatch is the single most common reason a first attempt sits in <code>NEEDS_ATTENTION</code> with no obvious cause.</p>



<h2 class="wp-block-heading">Setting up the integration</h2>



<ol class="wp-block-list">
<li>If you are using your own OAuth app, register it in the Zendesk admin center under the API settings and note the client ID and secret. Store the secret in AWS Secrets Manager, one secret per Glue connection.</li>



<li>In Glue Studio, create a connection under Data Connections. Choose Zendesk as the connection type, supply your instance URL and environment, and complete the OAuth redirect.</li>



<li>Create the source role with the policy above and attach it to the connection using <code>create-integration-resource-property</code>.</li>



<li>Prepare the target: the Glue database or S3 Table bucket, the target role, the catalog resource policy, and any Lake Formation grants your account&#8217;s permission model requires.</li>



<li>Create the integration. Select entities, set the refresh interval, and decide continuous sync versus on-demand snapshot. Both of those are locked from here.</li>



<li>Watch the first full load complete before you point anything at the tables. Full load and CDC are separate load types in the metrics, and the numbers look very different.</li>
</ol>



<p class="wp-block-paragraph">Once it works, move it into code. Glue zero-ETL integrations are supported by CloudFormation and the AWS CDK, which is the difference between a thing one person built in the console and a thing your team can redeploy into another account.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Monitoring: the metrics that expose drift</h2>



<p class="wp-block-paragraph">Zero-ETL publishes to the <code>AWS/Glue/ZeroETL</code> CloudWatch namespace, with dimensions for <code>integrationArn</code>, <code>loadType</code> and <code>tableName</code>. The metrics are <code>InsertCount</code>, <code>UpdateCount</code>, <code>DeleteCount</code>, <code>IngestionSucceeded</code>, <code>IngestionFailed</code> and <code>LastSyncTimestamp</code>.</p>



<p class="wp-block-paragraph">The obvious alarm is on failure:</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-alarm 
  --alarm-name zendesk-zetl-ingestion-failed 
  --namespace AWS/Glue/ZeroETL 
  --metric-name IngestionFailed 
  --dimensions Name=integrationArn,Value=&lt;your-integration-arn&gt; 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 1 
  --threshold 1 
  --comparison-operator GreaterThanOrEqualToThreshold 
  --treat-missing-data notBreaching</code></pre>



<p class="wp-block-paragraph">That one catches loud failures. It does not catch the quiet one, which is an integration that stops running entirely. For that, alarm on <code>IngestionSucceeded</code> with <code>--treat-missing-data breaching</code> over a window longer than your refresh interval. Absence of success is the signal, not presence of failure.</p>



<p class="wp-block-paragraph">And here is the one specific to this post. Because <code>DeleteCount</code> carries a <code>tableName</code> dimension, you can see the delete gap directly in a graph. Plot <code>DeleteCount</code> per table. Tickets, users and organizations will show a nonzero line. Satisfaction ratings, articles, calls and legs will sit flat at zero forever. That flat line is not a broken metric, it is the behaviour, and having it on a dashboard is the cheapest way to keep the whole team aware of it.</p>



<p class="wp-block-paragraph">Glue also writes a system table into the target database recording the outcome of each full load and CDC run, with per-run record, insert and delete counts. Query that when CloudWatch is not granular enough. If you already run Grafana Cloud or Datadog, pulling the <code>AWS/Glue/ZeroETL</code> namespace in alongside your existing infrastructure dashboards puts pipeline health next to everything else on call already watches.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<p class="wp-block-paragraph">Integrations move through Creating, Active, Modifying, Syncing, Needs attention, Failed and Deleting. What each one means in practice:</p>



<ul class="wp-block-list">
<li><strong>Syncing</strong> means it hit a recoverable error and is re-seeding data. Not an emergency. Let it finish before you touch anything.</li>



<li><strong>Needs attention</strong> means you have to fix something, usually connection credentials, the source role, the target role, or the catalog resource policy. Once you fix it, there is no manual recovery action. Glue retries automatically on an exponential backoff schedule, so the gap between attempts grows over time. If you fixed a policy and nothing happened after two minutes, that is expected. Wait it out.</li>



<li><strong>Failed</strong> is terminal. Delete the integration and recreate it. There is no repair path.</li>
</ul>



<p class="wp-block-paragraph">For anything else, the CloudWatch logs emitted after each full load or CDC run carry the actual root cause. The console status is a summary, and a fairly lossy one. Read the logs first.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Building CSAT, article count or call volume metrics on raw row counts from the four entities that never delete.</li>



<li>Creating the connection and assuming the integration can use it. The source role attachment is a separate CLI step.</li>



<li>Choosing Redshift as the direct target during a proof of concept and locking a refresh interval you picked arbitrarily.</li>



<li>Setting a 15 minute interval because it was available, then being surprised by small-file churn and CloudWatch log volume.</li>



<li>Writing downstream queries that address Zendesk custom fields as nested columns. They arrive as strings.</li>



<li>Alarming only on <code>IngestionFailed</code>, so a stalled integration produces silence and silence looks like health.</li>



<li>Reaching for zero-ETL when the requirement is ticket comments or ticket metrics, which are not in the entity set at all.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Split delete-tracking and append-only entities into separate integrations so you can rebuild one without disturbing the other.</li>



<li>Land in the SageMaker lakehouse rather than straight into Redshift unless you are certain about freshness. It keeps the refresh interval editable.</li>



<li>Put <code>DeleteCount</code> per table on a dashboard. It makes an invisible behaviour visible to everyone, not just whoever read the docs.</li>



<li>Define the integration in CloudFormation or CDK once it works, so the next environment is a deploy rather than a repeat of the console clicking.</li>



<li>Keep a silver layer between the replicated tables and your BI tool. That is where you parse the stringified structs and apply the reconciliation logic, and it means schema surprises break one view instead of every dashboard.</li>



<li>Document which entities do not track deletes somewhere your analysts will actually read, next to the tables, not in a wiki page nobody opens.</li>
</ul>



<h3 class="wp-block-heading">When to use something else</h3>



<p class="wp-block-paragraph">Zero-ETL is a good fit when your target is already AWS, your entities are in the supported set, and you want AWS to own the pipeline. It is the wrong fit if you need entities outside that list, need transformation at ingest, or need the same Zendesk data landing somewhere that is not AWS. Managed connector platforms like Fivetran or Airbyte cover a wider slice of the Zendesk API and will replicate to non-AWS destinations, at the cost of another vendor relationship, another bill, and data leaving your account boundary on the way through. If the seven entities cover your reporting, zero-ETL is cheaper and simpler. If they do not, forcing it is worse than paying for a tool that fits.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Which Zendesk entities does AWS Glue zero-ETL support?</h3>



<p class="wp-block-paragraph">Tickets, users, organizations, satisfaction ratings, articles, calls and call legs. Tickets, users and organizations replicate creates, updates and deletes. The other four replicate creates and updates only.</p>



<h3 class="wp-block-heading">Can I get ticket comments or ticket metrics through zero-ETL?</h3>



<p class="wp-block-paragraph">No. They are not in the entity set. Use a Glue ETL job against the same Zendesk connection, or a connector platform with broader API coverage, and join the result to your replicated tables downstream.</p>



<h3 class="wp-block-heading">How fresh can the data be?</h3>



<p class="wp-block-paragraph">The refresh interval goes down to 15 minutes and up to six days. This is near real time replication, not streaming. If you need sub-minute latency on Zendesk events, zero-ETL is the wrong mechanism and you want webhooks into an event pipeline instead.</p>



<h3 class="wp-block-heading">Can I change the refresh interval later?</h3>



<p class="wp-block-paragraph">Only if the target is not Redshift. With Redshift as the target the interval is fixed at creation. With the SageMaker lakehouse and other targets you can modify it afterwards.</p>



<h3 class="wp-block-heading">Why is my integration stuck in NEEDS_ATTENTION after I fixed the permissions?</h3>



<p class="wp-block-paragraph">Because recovery is automatic but not immediate. Glue retries on exponential backoff, so if the integration has been unhealthy for a while the next retry may be some way off. There is no manual recovery command. Confirm the fix is correct, then wait.</p>



<h3 class="wp-block-heading">Does zero-ETL cost extra on top of Glue?</h3>



<p class="wp-block-paragraph">AWS does not bill a separate charge for the integration itself. You pay for the surrounding services: target storage, Redshift or Athena query compute, S3 requests, Data Catalog usage and CloudWatch logs. Refresh interval is the main lever, since it drives how many small commits and log entries you generate.</p>



<h3 class="wp-block-heading">Why do my Zendesk custom fields look like text?</h3>



<p class="wp-block-paragraph">The connector converts struct and list types to strings. Custom fields come through as serialized values rather than nested columns, so parse them in a downstream layer instead of querying them directly.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">Zendesk AWS Glue zero-ETL removes a genuinely tedious pipeline from your plate. The setup is fiddly in the IAM layer and then it mostly runs itself, which is exactly the point of a managed integration.</p>



<p class="wp-block-paragraph">The one thing worth carrying out of this post: four of the seven supported entities never replicate a delete. Satisfaction ratings, articles, calls and call legs only grow. Nothing warns you, every metric stays green, and your numbers drift a little further from Zendesk with every cycle. Design for that on day one, put <code>DeleteCount</code> per table on a dashboard, and you will not be the person explaining the discrepancy in a meeting six months from now.</p>



<h2 class="wp-block-heading">Need help with your Zendesk to AWS data pipeline?</h2>



<p class="wp-block-paragraph">I work with teams building and fixing replication pipelines on AWS. On this kind of setup, that usually means:</p>



<ul class="wp-block-list">
<li>Auditing an existing Zendesk zero-ETL integration and quantifying how far the target has drifted from the source</li>



<li>Designing the split between continuously synced and append-only entities, including the rebuild cadence and reconciliation queries</li>



<li>Untangling the source role, target role and catalog policy layers when an integration sits in <code>NEEDS_ATTENTION</code> with no clear cause</li>



<li>Building the silver layer that parses stringified Zendesk custom fields into something your BI tool can actually use</li>



<li>Setting up CloudWatch alarms and dashboards that catch a stalled integration, not just a failed one</li>



<li>Moving a console-built integration into CloudFormation or CDK so it can be deployed across accounts</li>
</ul>



<p class="wp-block-paragraph">Send me the integration status, a CloudWatch log excerpt, or the query that is giving you the wrong number, and I will tell you what I think is going on.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/zendesk-aws-glue-zero-etl/">Zendesk Data Integration with AWS Glue Zero-ETL: The Delete Gap That Skews Your Numbers</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>The Row Count That Only Goes Up: Building a Salesforce Data Lake on AWS</title>
		<link>https://john-nessime.com/blog/devops/salesforce-data-lake-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 02 Aug 2026 11:01:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AppFlow]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Change Data Capture]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Warehouse]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[SQL]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=83</guid>

					<description><![CDATA[<p>Your dashboard shows four percent more opportunities than Salesforce, and the extra rows were all deleted months ago. Incremental sync on SystemModstamp can never see a record that no longer exists. Here's how to build a Salesforce data lake on AWS that stays equal to the source.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-data-lake-aws/">The Row Count That Only Goes Up: Building a Salesforce Data Lake on AWS</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone in RevOps mentions that the pipeline dashboard shows about four percent more opportunities than Salesforce does. Not wildly wrong. Wrong enough that nobody trusts the number in a board meeting.</p>



<p class="wp-block-paragraph">You pull a sample of the extra rows and they all look fine. Real opportunity IDs, real amounts, real owners. You paste one into Salesforce and get nothing. The record was deleted eight months ago. So were the others.</p>



<p class="wp-block-paragraph">Your incremental extract has never deleted a row in its life. It asks Salesforce for everything where <code>SystemModstamp</code> is greater than the last run, and a record that no longer exists cannot come back in that result set. So the lake only ever grows, the gap widens a little every week, and nothing anywhere reports an error.</p>



<p class="wp-block-paragraph">That is the defining problem when you <strong>build a Salesforce data lake on AWS</strong>, and it is barely mentioned in the tutorials, which mostly stop once data lands in S3. Landing the data is the easy part. Keeping it equal to the source is the work.</p>



<p class="wp-block-paragraph">This covers the ingestion options and their honest trade-offs, how to structure the lake, how to handle deletes properly (harder than it sounds), the fields that go stale without telling you, staying inside your API budget, and the query layer on top.</p>



<h2 class="wp-block-heading">Why Salesforce is an awkward source</h2>



<p class="wp-block-paragraph">Four properties make it different from replicating a database, and every design decision below follows from them.</p>



<ul class="wp-block-list">
<li><strong>Deletes are soft, then permanent.</strong> A deleted record moves to the Recycle Bin, where it is visible for a limited window and then gone. Your ability to detect a deletion expires.</li>
<li><strong>Some fields are computed at read time.</strong> Formula fields are not stored. Salesforce works them out when you query, which means what you extracted is a snapshot, not a value that stays true.</li>
<li><strong>The API is a metered resource.</strong> Your org has a daily request allowance shared with every other integration. A careless extract can starve the tools sales actually uses.</li>
<li><strong>The schema changes without warning.</strong> Admins add fields as part of their normal job. Nobody tells the data team.</li>
</ul>



<h2 class="wp-block-heading">Choosing how to get the data out</h2>



<p class="wp-block-paragraph">Four realistic paths, roughly in order of how much you have to operate yourself.</p>



<p class="wp-block-paragraph"><strong>Amazon AppFlow</strong> is the native option: a managed connector that moves Salesforce objects into S3 on a schedule or on events, with no infrastructure to run. It handles both scheduled batch extracts and Salesforce Change Data Capture event flows, and it supports AWS PrivateLink to Salesforce, so the traffic does not have to cross the public internet. That last point matters more than it usually gets credit for.</p>



<p class="wp-block-paragraph">Where it gets thin: schema drift handling is basic, and mapping large numbers of objects through the console gets tedious fast. Define the flows in Terraform or CloudFormation from the start rather than clicking them, or you will end up with production configuration nobody can reproduce.</p>



<p class="wp-block-paragraph"><strong>A managed ELT vendor</strong> such as Fivetran or Airbyte handles schema drift, deletes and incremental logic for you, which is genuinely most of the hard work in this post. You pay per row or per connector, and on a large Salesforce org that number gets attention. Worth pricing against the engineering time before dismissing it.</p>



<p class="wp-block-paragraph"><strong>Rolling your own on the Bulk API</strong> gives you total control and makes you responsible for everything: pagination, retries, API budget, delete detection, schema evolution. I would only choose this when an existing tool genuinely cannot express what you need, and I would expect it to be a real service with monitoring rather than a Lambda someone wrote in an afternoon.</p>



<p class="wp-block-paragraph"><strong>Zero-copy sharing via Salesforce Data Cloud</strong> avoids replication entirely by querying Salesforce-held data from your warehouse. Attractive when compliance objects to copying CRM data, and it carries its own licensing and latency questions. Check what your Salesforce contract actually includes before designing around it.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Structure the lake so mistakes are cheap</h2>



<p class="wp-block-paragraph">Two layers, and the discipline to keep them separate.</p>



<p class="wp-block-paragraph"><strong>Raw</strong> is append-only and never edited. Every extract lands as a new partition, in Parquet, exactly as Salesforce returned it. You do not deduplicate here, you do not fix types here, and you certainly do not delete here. Raw is your ability to rebuild everything downstream when you discover a logic bug six months in, which you will.</p>



<pre class="wp-block-code"><code># Partition by extraction date, not by a business date. A bad run
# is then one partition to drop rather than a full-table repair.
s3://acme-lake/raw/salesforce/opportunity/extract_date=YYYY-MM-DD/</code></pre>



<p class="wp-block-paragraph"><strong>Curated</strong> is what people query: deduplicated to current state, typed properly, deletes applied. Build it as views over raw to begin with, and only materialise the ones that are genuinely too slow. Materialising early gives you a second copy of the truth to keep in sync, which is the problem you are already trying to solve.</p>



<pre class="wp-block-code"><code>-- Current state from an append-only raw layer: keep the most
-- recent version of each record and drop anything flagged deleted.
CREATE OR REPLACE VIEW curated.opportunity AS
SELECT *
FROM (
    SELECT o.*,
           ROW_NUMBER() OVER (
               PARTITION BY id ORDER BY systemmodstamp DESC
           ) AS rn
    FROM raw.opportunity o
)
WHERE rn = 1
  AND is_deleted = false;</code></pre>



<p class="wp-block-paragraph">Register both in the Glue Data Catalog so Athena and Redshift Spectrum see the same definitions. If you expect frequent updates and deletes rather than pure appends, an open table format like Apache Iceberg is worth the extra setup: row-level deletes, schema evolution and time travel are exactly the operations this workload needs, and they are painful to hand-roll on plain Parquet.</p>



<h2 class="wp-block-heading">Handling deletes properly</h2>



<p class="wp-block-paragraph">Here is the section that matters. There are three mechanisms and you need more than one, because each has a hole.</p>



<h3 class="wp-block-heading">1. Query the Recycle Bin</h3>



<p class="wp-block-paragraph">Standard SOQL hides deleted records. The REST API&#8217;s <code>queryAll</code> endpoint, the SOAP <code>queryAll()</code> call, or <code>ALL ROWS</code> in Apex all return them, and <code>IsDeleted</code> exists on virtually every object even though it does not appear in Setup.</p>



<pre class="wp-block-code"><code>-- Against /services/data/vXX.X/queryAll, not /query.
-- Substitute your own high-water mark for the placeholder.
SELECT Id, IsDeleted, SystemModstamp
FROM Opportunity
WHERE IsDeleted = true
  AND SystemModstamp &gt; {last_run_utc}</code></pre>



<p class="wp-block-paragraph">The hole: records sit in the Recycle Bin for fifteen days by default. Salesforce Classic offers an extended retention setting that pushes it to thirty. But the bin also has a capacity tied to your org&#8217;s storage allocation, and when it fills, the oldest records are purged early to make room. No alert, no warning. So your detection window is not fifteen days. It is fifteen days <em>or less</em>, and you find out which during a mass-delete week.</p>



<h3 class="wp-block-heading">2. Change Data Capture</h3>



<p class="wp-block-paragraph">Salesforce CDC publishes create, update, delete and undelete events as they happen, and AppFlow can consume them. This is the cleanest mechanism, because a delete arrives as an event rather than being inferred from an absence.</p>



<p class="wp-block-paragraph">The hole: it is a streaming subscription, so if your consumer is down long enough, you miss events. Event replay windows are finite. CDC is a good primary mechanism and a bad only mechanism.</p>



<h3 class="wp-block-heading">3. Periodic full ID reconciliation</h3>



<p class="wp-block-paragraph">This is the safety net, it catches everything the other two miss, and it is the one people skip because it feels crude. Pull nothing but the <code>Id</code> column for the whole object. One narrow field over the Bulk API is cheap even on millions of rows.</p>



<pre class="wp-block-code"><code>SELECT Id FROM Opportunity</code></pre>



<p class="wp-block-paragraph">Then diff it against the lake. Whatever is on your side and not on theirs no longer exists, whatever the reason:</p>



<pre class="wp-block-code"><code>-- Rows the lake still believes in and Salesforce does not.
SELECT l.id
FROM curated.opportunity l
LEFT JOIN staging.opportunity_ids s
       ON s.id = l.id
WHERE s.id IS NULL;</code></pre>



<p class="wp-block-paragraph">Run it weekly on your important objects. Soft-delete the differences in the curated layer rather than hard-deleting from raw, so you keep the audit trail.</p>



<p class="wp-block-paragraph">One more case this catches that nothing else does cleanly: <strong>merges</strong>. When an admin merges two Accounts, Contacts or Leads, the losing record is deleted and gets a <code>MasterRecordId</code> pointing at the survivor. If your lake keeps the loser, you are double-counting a customer, and the arithmetic is wrong in a way that looks like a data quality problem rather than a pipeline problem.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The other thing that goes stale silently</h2>



<p class="wp-block-paragraph">Formula fields are not stored anywhere. Salesforce computes them when you ask. So the value that landed in your lake is a snapshot from extraction time, and Salesforce will happily return something different tomorrow without the record&#8217;s <code>SystemModstamp</code> moving at all.</p>



<p class="wp-block-paragraph">Cross-object formulas are the worst version. A formula on Opportunity that references a field on its Account recalculates when the Account changes. The Opportunity did not change, so your incremental extract never picks it up. That column in your lake can be wrong indefinitely and nothing detects it.</p>



<p class="wp-block-paragraph">Three ways out, in order of preference:</p>



<ol class="wp-block-list">
<li><strong>Do not extract them.</strong> Pull the underlying fields and reimplement the calculation in your transformation layer, where it is version-controlled and testable. More work up front, correct forever after.</li>
<li><strong>Refresh them on a schedule.</strong> A narrow periodic extract of just the ID plus the formula columns, full-table. Cheap if the column list is short.</li>
<li><strong>Accept it and document it.</strong> Fine for a formula nobody reports on. Not fine for anything in a revenue calculation.</li>
</ol>



<p class="wp-block-paragraph">Keep a list of which columns in your lake are formula-derived. Six months from now, when a number disagrees with Salesforce, that list is the first thing you will want and the last thing anybody wrote down.</p>



<h2 class="wp-block-heading">Stay inside the API budget</h2>



<p class="wp-block-paragraph">Your Salesforce org has a daily API request allowance determined by edition and licence count, and it is shared. Exceed it and everything integrated with Salesforce stops working, not just your pipeline. That includes whatever marketing and support depend on, which is a conversation you would rather not have.</p>



<ul class="wp-block-list">
<li><strong>Use the Bulk API for extracts</strong>, not the REST query API. Bulk is designed for volume and consumes the allowance very differently.</li>
<li><strong>Incremental by default, full refresh by exception.</strong> A nightly full extract of a large object is the single most common way to blow the budget.</li>
<li><strong>Give the pipeline its own integration user.</strong> That way the API usage reports attribute consumption to it, and you can see your own footprint instead of arguing about it.</li>
<li><strong>Watch consumption as a metric.</strong> Salesforce exposes API usage; graph it. Finding out by outage is expensive.</li>
</ul>



<p class="wp-block-paragraph">One thing that surprises people: field-level security on that integration user determines which fields come back. A field can exist in Salesforce, be populated, and arrive in your lake as consistently null because the integration user&#8217;s profile cannot see it. Check the profile before debugging the pipeline.</p>



<h2 class="wp-block-heading">Query layer and access</h2>



<p class="wp-block-paragraph">Athena over the Glue catalog is the default answer and a good one: no cluster to run, pay per query, and it reads the curated views directly. Redshift Spectrum makes sense when you already run Redshift and want to join CRM data to warehouse tables.</p>



<p class="wp-block-paragraph">Cost control on Athena is mostly about scan volume, which means partitioning and columnar storage rather than query tuning. Parquet plus sensible partitions does more than any amount of SQL cleverness.</p>



<p class="wp-block-paragraph">On access: you have just replicated your customer database into object storage. Contacts and Leads are personal data, and Salesforce&#8217;s own field-level security does not follow it across. Encrypt the bucket, keep it private, use Lake Formation for column-level grants if different teams need different views, and decide the retention policy deliberately rather than by default. &#8220;We keep everything forever in S3 because it&#8217;s cheap&#8221; is a defensible engineering position and a poor compliance one.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Row counts drift upward over time</h3>



<p class="wp-block-paragraph">Deletes are not being applied. Run the full ID reconciliation and see how many rows come back. If it is a lot, also check for merged records via <code>MasterRecordId</code> before assuming the extract is at fault.</p>



<h3 class="wp-block-heading">A column is always null in the lake, populated in Salesforce</h3>



<p class="wp-block-paragraph">Field-level security on the integration user&#8217;s profile. Log in as that user, or check the profile&#8217;s field permissions, before touching the pipeline.</p>



<h3 class="wp-block-heading">A value disagrees with Salesforce but the record looks current</h3>



<p class="wp-block-paragraph">Almost certainly a formula field, especially a cross-object one. Check whether the column is formula-derived; if it is, the record&#8217;s modstamp never moved and your incremental extract had no reason to refetch it.</p>



<h3 class="wp-block-heading">Extract fails partway on a large object</h3>



<p class="wp-block-paragraph">Query timeouts or governor limits. Chunk by date range or by ID range rather than pulling the object in one go, and make each chunk independently retryable so a failure costs you one slice rather than the whole run.</p>



<h3 class="wp-block-heading">New Salesforce fields never appear</h3>



<p class="wp-block-paragraph">Most connectors map fields explicitly at configuration time and do not add new ones on their own. Poll the object&#8217;s describe metadata on a schedule and alert when the field list changes, so schema drift is a notification rather than a discovery.</p>



<h3 class="wp-block-heading">Athena queries are slow or expensive</h3>



<p class="wp-block-paragraph">You are scanning too much. Check partitioning first, then whether the files are Parquet rather than JSON or CSV, then whether many tiny files are being read per query. Small-file proliferation is the usual culprit on frequently-run extracts, and compaction fixes it.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Building incremental sync on <code>SystemModstamp</code> alone and never handling deletes.</li>
<li>Assuming the Recycle Bin gives you a guaranteed fifteen-day detection window.</li>
<li>Treating CDC as sufficient on its own, with no reconciliation behind it.</li>
<li>Ignoring merges, so a merged customer is counted twice forever.</li>
<li>Extracting formula fields and treating them as durable values.</li>
<li>Nightly full extracts of large objects, and the API outage that follows.</li>
<li>Sharing an integration user with other tools, so nobody can attribute API consumption.</li>
<li>Mutating the raw layer, which destroys your ability to rebuild.</li>
<li>Clicking AppFlow flows together in the console with no infrastructure as code.</li>
<li>Landing JSON or CSV instead of Parquet, then paying for it on every Athena query.</li>
<li>No alerting on schema drift, so new fields are found by a user asking.</li>
<li>Replicating Contacts and Leads into S3 with no retention policy or access controls.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Append-only raw layer, curated views on top, and never edit raw.</li>
<li>Use at least two delete mechanisms, one of which is periodic full ID reconciliation.</li>
<li>Soft-delete in curated rather than hard-deleting, so the audit trail survives.</li>
<li>Reimplement formula logic downstream instead of trusting extracted formula values.</li>
<li>Bulk API for extracts, incremental by default.</li>
<li>A dedicated integration user with a documented, deliberately scoped profile.</li>
<li>Partition by extraction date and store Parquet, or use Iceberg if you need row-level updates.</li>
<li>Define flows, catalogs and permissions as code.</li>
<li>Monitor API consumption and row-count delta against Salesforce as first-class metrics.</li>
<li>Alert on schema changes rather than discovering them.</li>
<li>Encrypt, restrict and set a retention policy on personal data the day you land it, not later.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">How do I handle deleted Salesforce records in a data lake?</h3>



<p class="wp-block-paragraph">Combine mechanisms. Query the Recycle Bin with <code>queryAll</code> and <code>IsDeleted</code> for recent deletions, consume Change Data Capture events for real-time coverage, and run a periodic full ID reconciliation as the backstop. No single one of the three is complete on its own.</p>



<h3 class="wp-block-heading">Should I use AppFlow or a third-party ELT tool?</h3>



<p class="wp-block-paragraph">AppFlow if you want to stay inside AWS, value PrivateLink connectivity, and are willing to handle schema drift and deletes yourself. A managed vendor if you would rather buy those solved and can live with per-row pricing. Price both against the engineering time honestly; the build-it-yourself option is usually costed at zero and is not.</p>



<h3 class="wp-block-heading">Parquet or Iceberg?</h3>



<p class="wp-block-paragraph">Parquet with date partitions is fine for append-only raw. Iceberg earns its complexity in the curated layer, where you want row-level updates and deletes, schema evolution and time travel. Salesforce data changes constantly, so that need is real rather than theoretical.</p>



<h3 class="wp-block-heading">How often should I sync?</h3>



<p class="wp-block-paragraph">Match the decisions people actually make with the data. Hourly incrementals suit most reporting; sub-minute freshness usually means CDC and a real streaming consumer, which is a much larger commitment. Schedule the ID reconciliation separately and less often, weekly is normally enough.</p>



<h3 class="wp-block-heading">Will this use up my Salesforce API limit?</h3>



<p class="wp-block-paragraph">It can, and the failure is shared with every other integration on the org. Use the Bulk API, stay incremental, give the pipeline its own user so consumption is attributable, and graph usage. Do not find out from an outage.</p>



<h3 class="wp-block-heading">Do I need Salesforce Data Cloud?</h3>



<p class="wp-block-paragraph">Not to build a lake on AWS. It becomes interesting when compliance objects to copying CRM data at all, since zero-copy sharing avoids replication. Check what your existing Salesforce agreement covers before designing around it, because the licensing is a real factor.</p>



<h3 class="wp-block-heading">Why do my numbers differ from Salesforce reports?</h3>



<p class="wp-block-paragraph">Check in this order: missing deletes, merged records counted twice, stale formula fields, fields the integration user cannot see, and only then your own transformation logic. The first two account for most of it, and the last is where people look first.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">Getting Salesforce data into S3 is a configuration exercise you can finish in an afternoon. Keeping that data equal to Salesforce is an ongoing engineering problem, and the parts that break do not raise errors. They produce numbers that are slightly wrong, in a consistent direction, for months.</p>



<p class="wp-block-paragraph">So build the reconciliation before you build the dashboards. A weekly job that pulls every ID and diffs it against the lake is unglamorous, cheap, and the only thing that will tell you the truth when someone asks why the two systems disagree.</p>



<h2 class="wp-block-heading">Building or fixing one of these?</h2>



<p class="wp-block-paragraph">Most of the Salesforce lakes I get asked to look at work correctly for the first quarter and then quietly diverge. Work I take on:</p>



<ul class="wp-block-list">
<li>Building a Salesforce to S3 pipeline end to end: AppFlow or Bulk API ingestion, Glue catalog, Athena or Redshift Spectrum query layer.</li>
<li>Auditing an existing lake for drift, and reporting exactly how many rows disagree with Salesforce and why.</li>
<li>Implementing proper delete handling: CDC consumption, Recycle Bin queries and scheduled ID reconciliation.</li>
<li>Reworking formula-derived columns into version-controlled transformations that stay correct.</li>
<li>API budget work: moving extracts to Bulk, splitting integration users, and monitoring consumption before it causes an outage.</li>
<li>Access and retention design for CRM data in S3, including encryption, Lake Formation grants and deletion policies.</li>
</ul>



<p class="wp-block-paragraph">Tell me which objects you replicate and how you currently detect deletes, and I will tell you where the drift is coming from.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-data-lake-aws/">The Row Count That Only Goes Up: Building a Salesforce Data Lake on AWS</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
