<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>SQL | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/sql/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/sql/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Thu, 29 Oct 2026 06:58:01 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>SQL | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/sql/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>AWS Schema Conversion Tool for Oracle: What Converts and What Quietly Doesn&#8217;t</title>
		<link>https://john-nessime.com/blog/technical-guides/aws-schema-conversion-tool-oracle/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 03 Oct 2026 06:43:14 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Databases]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Aurora]]></category>
		<category><![CDATA[Amazon RDS]]></category>
		<category><![CDATA[AWS DMS]]></category>
		<category><![CDATA[AWS SCT]]></category>
		<category><![CDATA[Cutover Planning]]></category>
		<category><![CDATA[Data Type Mapping]]></category>
		<category><![CDATA[Database Migration]]></category>
		<category><![CDATA[Migration Assessment]]></category>
		<category><![CDATA[Oracle]]></category>
		<category><![CDATA[PL/SQL]]></category>
		<category><![CDATA[PostgreSQL]]></category>
		<category><![CDATA[Schema Conversion]]></category>
		<category><![CDATA[SQL]]></category>
		<category><![CDATA[Vendor Lock-In]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=388</guid>

					<description><![CDATA[<p>The assessment report grades syntax, not behavior. A four-tier breakdown of what AWS SCT converts cleanly, what it converts while silently changing meaning, what it only handles through the extension pack, what it refuses outright, and the whole category it never looks at.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/aws-schema-conversion-tool-oracle/">AWS Schema Conversion Tool for Oracle: What Converts and What Quietly Doesn&#8217;t</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Open the assessment report on a mid-sized Oracle schema and the top-line number looks fantastic. Ninety-something percent of objects convert automatically. The red items are counted, categorized, and given effort estimates. It reads like a project plan you could take to a steering committee.</p>



<p class="wp-block-paragraph">The part that hurts you later is not on that page.</p>



<p class="wp-block-paragraph">The AWS Schema Conversion Tool for Oracle grades <em>syntax</em>. It tells you whether it could emit valid PostgreSQL for an object. It does not tell you whether that object will behave the same way at runtime. A trigger can convert with a green tick and fire in a different order. A column can convert with a green tick and silently drop the time component off every date in the table. A procedure can convert with a green tick and run twenty times slower because every string function call now goes through a wrapper.</p>



<p class="wp-block-paragraph">This post sorts the tool&#8217;s output into four buckets: what converts cleanly, what converts while quietly changing meaning, what converts only through the extension pack, and what it refuses outright. Then the category nobody budgets for, which is everything it never looks at.</p>



<h2 class="wp-block-heading">What the AWS Schema Conversion Tool for Oracle actually does</h2>



<p class="wp-block-paragraph">It is a Java desktop application. It connects over JDBC to source and target, reads the Oracle data dictionary, parses the DDL and PL/SQL it finds, and generates equivalent PostgreSQL objects and PL/pgSQL. The converted code stays inside the project until you explicitly apply it to the target or save it out as SQL files.</p>



<p class="wp-block-paragraph">Three things follow from that description, and they matter more than any feature list.</p>



<ul class="wp-block-list">
<li>It only sees what is stored in the database. Code that lives in your application, your job scheduler, or a reporting tool is invisible to it unless you point the separate application SQL conversion feature at those files yourself.</li>

<li>It only sees what the connecting user can see. Connect with a thin account and half the schema will not appear in the report at all.</li>

<li>It moves structure and code. It does not move rows. That is AWS DMS, and it is a separate exercise with its own failure modes.</li>
</ul>



<p class="wp-block-paragraph">There is also a command line interface, which is the part most teams discover too late. It uses scenario files rather than flag-style arguments, and it needs Amazon Corretto 11 installed alongside the tool.</p>



<pre class="wp-block-code"><code># Interactive mode, run from the AWS SCT app folder.
# End each command with a / on its own line.
java -jar AWSSchemaConversionToolBatch.jar -type interactive

# Script mode, running a saved .scts scenario end to end.
java -jar /opt/aws-schema-conversion-tool/lib/app/AWSSchemaConversionToolBatch.jar 
  -type scts -script /home/sct/project/convert.scts</code></pre>



<p class="wp-block-paragraph">Script mode turns a one-off conversion into something you can rerun, which matters because on a real migration you will convert the schema many times as you clean up the source. Re-clicking through the GUI each time is how mistakes creep in.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Tier one: converts cleanly, and you can trust it</h2>



<p class="wp-block-paragraph">Credit where it is due. For the bulk of an ordinary OLTP schema the tool is good, and hand-writing this would be miserable.</p>



<ul class="wp-block-list">
<li>Table definitions: names, columns, ordinary data types, NOT NULL, defaults, primary keys, unique constraints, foreign keys.</li>

<li>Standard b-tree indexes, including most single-column functional indexes.</li>

<li>Ordinary views. The basic CREATE VIEW syntax is close to identical between the two engines.</li>

<li>Sequences and identity columns, despite the syntax looking nothing alike.</li>

<li>ANSI-standard SQL, joins, subqueries, and almost all window and analytic functions.</li>

<li>PL/SQL flow control: loops, blocks, conditionals, exception structure.</li>

<li>The four standard transaction isolation levels, plus BEGIN, COMMIT, and ROLLBACK.</li>
</ul>



<p class="wp-block-paragraph">If your schema is mostly tables, views, and reporting SQL with modest procedural code, the automation figure in the report is roughly honest and you should believe it.</p>



<h2 class="wp-block-heading">Tier two: converts green, behaves differently</h2>



<p class="wp-block-paragraph">This is the tier that costs money, because none of it shows up as a red action item. The object converted. The DDL applied. The application connected. Something is wrong three weeks later and nobody thinks to look back at the schema conversion.</p>



<h3 class="wp-block-heading">Oracle DATE is not PostgreSQL date</h3>



<p class="wp-block-paragraph">Oracle&#8217;s DATE type carries a time component down to the second. PostgreSQL&#8217;s <code>date</code> is a calendar day and nothing more. Map an Oracle DATE to a PostgreSQL <code>date</code> and every timestamp in that column is truncated, permanently, without an error. The correct target is <code>timestamp</code>. Check the conversion settings and check the generated DDL before you apply, because this one is not recoverable after the data lands.</p>



<h3 class="wp-block-heading">NUMBER without precision becomes numeric, and numeric is slow</h3>



<p class="wp-block-paragraph">An unqualified Oracle NUMBER is an extremely flexible container, so the default mapping to PostgreSQL <code>numeric</code> is technically correct. It is also arbitrary-precision arithmetic implemented in software. Put that on a surrogate key that participates in every join and index in your application and you have bought yourself a performance problem that no amount of index tuning will fix.</p>



<p class="wp-block-paragraph">Find the candidates before conversion, not after:</p>



<pre class="wp-block-code"><code>-- Oracle: unqualified NUMBER columns, the ones that default to numeric.
SELECT table_name, column_name, data_precision, data_scale
FROM   user_tab_columns
WHERE  data_type = 'NUMBER'
AND    data_precision IS NULL
ORDER  BY table_name, column_name;</code></pre>



<p class="wp-block-paragraph">For each one, look at the actual minimum, maximum, and scale of the data, then decide whether it is really an <code>integer</code>, a <code>bigint</code>, or a genuine <code>numeric(p,s)</code>. Surrogate keys and foreign keys are almost always integers pretending to be NUMBER. Money columns usually are not.</p>



<h3 class="wp-block-heading">Empty string is NULL in Oracle and is not in PostgreSQL</h3>



<p class="wp-block-paragraph">This is the single most reliable source of post-cutover bugs, and it is not a conversion defect. Oracle treats an empty string as NULL. PostgreSQL treats it as a zero-length string, which is a real value. Every <code>IS NULL</code> check in your procedural code and your application was written under Oracle&#8217;s rule.</p>



<p class="wp-block-paragraph">Concatenation flips too. In Oracle, concatenating a NULL is a no-op. In PostgreSQL, concatenating a NULL with the <code>||</code> operator yields NULL for the whole expression. A report that used to print a name with a missing middle initial now prints nothing at all.</p>



<p class="wp-block-paragraph">After the data lands, count the divergence before you let users near it:</p>



<pre class="wp-block-code"><code>-- PostgreSQL: how many rows are empty string rather than NULL?
SELECT count(*) FILTER (WHERE middle_name = '')   AS empty_string,
       count(*) FILTER (WHERE middle_name IS NULL) AS real_null
FROM   app.customer;</code></pre>



<p class="wp-block-paragraph">If <code>empty_string</code> is non-zero anywhere, you have a decision to make: normalize the data on load, or audit every NULL check that touches those columns. Normalizing is usually cheaper and is far easier to verify.</p>



<h3 class="wp-block-heading">The warnings the tool does raise, that people scroll past</h3>



<p class="wp-block-paragraph">Some of these semantic differences do appear in the action list, just not in red. They are worth reading individually rather than in aggregate.</p>



<ul class="wp-block-list">
<li>GREATEST and LEAST are flagged as potentially producing different results from the source, because of how each engine handles NULL arguments and type coercion.</li>

<li>Timestamp handling inside converted partitioned tables is flagged as potentially divergent.</li>

<li>Converted code is flagged as possibly incorrect where bind variable names, user-defined functions, or dynamic SQL are involved. That is the tool telling you it guessed.</li>

<li>Trigger firing order is not carried over. Oracle&#8217;s FOLLOWS and PRECEDES clauses have no equivalent, and PostgreSQL fires triggers on the same event in alphabetical order by trigger name. If your Oracle triggers depended on sequencing, you now depend on naming.</li>

<li>An empty exception block in the converted output is called out explicitly. Oracle code that swallowed errors will now swallow them differently.</li>
</ul>



<p class="wp-block-paragraph">One more that is not an action code at all: Oracle PL/SQL routinely issues COMMIT in the middle of a procedure. A PostgreSQL function cannot do that. Either the logic becomes a procedure invoked with CALL, or the transaction boundary moves out into the calling application. That is a design decision, not a find-and-replace.</p>



<h2 class="wp-block-heading">Tier three: converts, but only through the extension pack</h2>



<p class="wp-block-paragraph">When there is no native PostgreSQL equivalent for something Oracle provides, the tool installs an extension pack: an additional schema on the target, named <code>aws_oracle_ext</code> for Oracle sources, containing wrapper functions, tables, and supporting objects that emulate the source behavior. For some capabilities like job scheduling and sending mail, the pack also deploys AWS Lambda functions.</p>



<p class="wp-block-paragraph">So a call to <code>INSTR</code> in your PL/SQL comes out the other side as <code>aws_oracle_ext.INSTR</code>. Package variables, which have no PostgreSQL equivalent because PL/pgSQL has no packages, get routed through getter and setter functions in the pack.</p>



<p class="wp-block-paragraph">The case for accepting this is real. If the goal is off Oracle licensing by a fixed date, the pack converts code that would otherwise need a developer per object. That is the difference between a migration that ships and one that stalls in refactoring, and nobody gets a bonus for elegant PL/pgSQL on a project that missed its window.</p>



<p class="wp-block-paragraph">The case against is equally real:</p>



<ul class="wp-block-list">
<li>Wrapper functions sit between your code and the engine. In hot loops, that overhead is measurable, and the package-variable emulation in particular is not free.</li>

<li>You have swapped one dependency for another. The pack is AWS-specific code living in your database, which is an awkward thing to explain if the stated goal was ending vendor lock-in.</li>

<li>Every time the pack is applied, its components are overwritten. Anything you hand-tuned inside that schema is gone.</li>

<li>Upgrades and target-version changes now have an extra moving part to validate.</li>
</ul>



<p class="wp-block-paragraph">There is a middle path. The tool can be configured to use the open-source <code>orafce</code> extension, which implements a subset of Oracle functions and packages natively in PostgreSQL, in place of some pack wrappers. My rule of thumb: use the pack to get the migration over the line, then treat the size of the <code>aws_oracle_ext</code> call graph as a debt register you burn down deliberately, starting with whatever appears in your slowest queries. Rewriting collections as native arrays or JSON, and package variables as parameters, buys back most of the performance.</p>



<h2 class="wp-block-heading">Tier four: it stops and hands you the work</h2>



<p class="wp-block-paragraph">These are the red items: the honest part of the report, and paradoxically the least dangerous, because you can see them and plan around them. Not exhaustive, but this is what turns up on real Oracle schemas.</p>



<ul class="wp-block-list">
<li><strong>Database links.</strong> No automation at all. The mechanism has to be rebuilt, usually on <code>postgres_fdw</code>, and foreign data wrappers do not support calling user-defined functions on the remote side. If your architecture leans on links, this is a workstream, not a task.</li>

<li><strong>Autonomous transactions.</strong> PostgreSQL has no direct equivalent. Logging patterns that relied on committing independently of the parent transaction need redesign, often with <code>dblink</code> back to the same instance or an out-of-band queue.</li>

<li><strong>ROWID and UROWID.</strong> No equivalent, and any code that reads ROWID from a table is flagged separately. Row-identity tricks in PL/SQL are a rewrite.</li>

<li><strong>Dynamic SQL and DBMS_SQL.</strong> Anything assembled at runtime the parser cannot resolve statically comes back untouched or with a health warning.</li>

<li><strong>Synonyms.</strong> Not supported, and neither is their usage, so every reference through a synonym has to be resolved by hand or mimicked with views and search paths.</li>

<li><strong>Global temporary tables, index-organized tables, external tables, clustered tables, object tables, nested tables, virtual columns, BFILE.</strong> Storage and type features with no target equivalent.</li>

<li><strong>Bitmap indexes, domain indexes, multi-column functional indexes.</strong> The first two have no counterpart; the third is a documented conversion gap.</li>

<li><strong>Associative arrays, VARRAY of VARRAY, multi-dimensional arrays, object type methods, member functions in user-defined types.</strong> Collection-heavy PL/SQL is where the automation percentage collapses.</li>

<li><strong>Compound triggers, system triggers, REFERENCING clauses, conditional predicates.</strong> The trigger model differs enough that anything beyond a plain row trigger needs attention.</li>

<li><strong>Java stored routines and table functions.</strong> Not converted.</li>

<li><strong>Query hints.</strong> Partially converted, and you should delete them all first anyway. Hints tuned for Oracle&#8217;s optimizer are noise to a different planner. Add them back only after schema, index, and query work have failed.</li>

<li><strong>Materialized view incremental refresh and DML on materialized views.</strong> The view itself may convert; the refresh strategy will not.</li>
</ul>



<h3 class="wp-block-heading">Some red items are more conservative than your actual target</h3>



<p class="wp-block-paragraph">Here is a detail that saves real budget. The action code index and the migration playbook were written against a baseline target, and PostgreSQL has moved since. MERGE is listed as unsupported, but PostgreSQL has had a native MERGE statement since version 15. FOR UPDATE SKIP LOCKED is listed as unsupported, and it has been in PostgreSQL for a long time. GROUPING SETS, CUBE, and ROLLUP are listed as unsupported and are also long-standing PostgreSQL features.</p>



<p class="wp-block-paragraph">What this means practically: an action code tells you the converter will not emit the construct for you. It is not a statement about your target engine. Before you scope a rewrite for anything on that list, check the feature against the PostgreSQL version you are actually deploying. Some of those &#8220;rewrites&#8221; are a copy-paste with a syntax tweak.</p>



<h2 class="wp-block-heading">What it never looks at</h2>



<p class="wp-block-paragraph">This is the category that wrecks timelines, because it is not in the report at all. Not green, not red. Out of scope, therefore invisible.</p>



<ul class="wp-block-list">
<li><strong>Application code.</strong> There is a separate application SQL conversion feature that scans source files for embedded SQL, but it is opt-in, file-based, and you have to aim it. Nothing happens automatically.</li>

<li><strong>Scheduled work.</strong> DBMS_SCHEDULER jobs, cron entries calling SQL*Plus, shell wrappers, ETL controllers. The schema converts; the thing that runs it every night does not.</li>

<li><strong>Grants, roles, profiles, and proxy authentication.</strong> Your security model is a separate design exercise on the target.</li>

<li><strong>Oracle features with no schema footprint.</strong> Data Guard, RAC, ASM, RMAN, Oracle Text indexes, fine-grained access control policies. Each is its own replacement decision.</li>

<li><strong>Optimizer statistics and execution plans.</strong> Nothing about performance carries over. You are starting from zero on the target, and that is the right expectation to set with stakeholders early.</li>

<li><strong>Everything downstream.</strong> BI dashboards, reporting layers, and integration jobs that embed Oracle-flavored SQL will break on the day you cut over, and none of them appear in the assessment.</li>
</ul>



<p class="wp-block-paragraph">Inventory these before conversion, not after. A one-page list of every consumer of the schema is worth more than a week of tuning.</p>



<h2 class="wp-block-heading">Desktop tool or DMS Schema Conversion?</h2>



<p class="wp-block-paragraph">DMS Schema Conversion is the managed, browser-based version built on the same engine. Instead of a laptop install you create an instance profile, register source and target data providers with credentials in Secrets Manager, and run the project from the console, with metadata staged in S3.</p>



<p class="wp-block-paragraph">AWS now points standard OLTP conversions at the managed option, and generative AI assisted conversion is available there on supported paths. The desktop tool remains available and still covers ground the managed feature does not.</p>



<ul class="wp-block-list">
<li><strong>Pick the desktop tool</strong> when you need scripting through the CLI, want to edit the SQL for source and target inside the project, need migration rules and tree filters, are converting a data warehouse target, or want the application SQL and ETL conversion features.</li>

<li><strong>Pick DMS Schema Conversion</strong> when your source is reachable from a VPC, you would rather not manage a workstation with database credentials on it, your conversion path and region are supported, and the generative AI assist is useful to you.</li>
</ul>



<p class="wp-block-paragraph">Check supported paths, versions, and regions before committing either way; they differ between the two. On the desktop side, note the supported operating systems are Windows, Fedora, and Ubuntu, all 64-bit. If your analysts are on Macs, plan for a Linux jump box. A small VPS from a provider like Contabo or InterServer, or an EC2 instance inside the VPC, is the usual answer, and it has the side benefit of keeping the JDBC connection close to the source rather than dragging a dictionary read across a home broadband link.</p>



<p class="wp-block-paragraph">If the Oracle source is on-premises and the conversion box is not, you need a network path that a DBA will sign off on. That is normally a site-to-site VPN or Direct Connect. For an individual consultant working against a client&#8217;s staging environment, a commercial VPN such as NordVPN or Surfshark is sometimes what is on offer, but treat it as a stopgap: it gets you connectivity, not an audit trail.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">The CLI jar refuses to open</h3>



<p class="wp-block-paragraph">A common failure is an unexpected error opening <code>AWSSchemaConversionToolBatch.jar</code>. Recent OpenJDK builds cap the size of jar signature files and the SCT jar exceeds the default. Raising that limit on the java invocation is the widely reported workaround:</p>



<pre class="wp-block-code"><code>java -Djdk.jar.maxSignatureFileSize=20000000 
  -jar AWSSchemaConversionToolBatch.jar -type interactive</code></pre>



<p class="wp-block-paragraph">Confirm you are on the Corretto 11 runtime the tool expects before chasing anything more exotic.</p>



<h3 class="wp-block-heading">It hangs on a large schema, or will not connect at all</h3>



<p class="wp-block-paragraph">Thousands of objects in one schema puts the JVM under memory pressure; Global settings exposes JVM options, so raise the heap and retry on a machine with real memory. Separately, JDBC driver paths live in Global settings under Drivers, per engine. A driver sitting on disk but not registered there fails in a way that looks like a network problem and is not.</p>



<h3 class="wp-block-heading">&#8220;The package body doesn&#8217;t include source code&#8221;</h3>



<p class="wp-block-paragraph">The source is wrapped. Obfuscated PL/SQL cannot be parsed, so it cannot be converted. Either find the original in version control or accept a from-scratch rewrite. Finding this on a critical package late in the project is a bad day, so check for wrapped objects on day one.</p>



<h3 class="wp-block-heading">Objects will not apply, or the report is missing things</h3>



<p class="wp-block-paragraph">Failures on apply are usually dependency order: an object references a parent that was not converted or sits outside the schema mapping, and the action list says so explicitly. Convert the whole dependency closure, not just the objects you care about. A short report is usually privileges, since the tool reads through the connecting account&#8217;s visibility. Compare its object counts against a direct dictionary count before trusting any percentage in it.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating the automation percentage as a completion estimate. It measures objects, not effort, and one gnarly package can outweigh four hundred clean tables.</li>

<li>Applying to the target before reviewing the generated DDL. Save to SQL, read it, diff it on the next run.</li>

<li>Accepting default type mappings wholesale. DATE and unqualified NUMBER are the two that reliably hurt.</li>

<li>Carrying Oracle hints across. Start with none and earn each one back.</li>

<li>Scoping rewrites straight off the action codes without checking your target PostgreSQL version first.</li>

<li>Leaving project files and scenario scripts containing database credentials sitting on a laptop after the engagement. They are plain files; when the work is done, wipe them properly with a tool like O&amp;O SafeErase rather than dragging them to the trash.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ol class="wp-block-list">
<li>Connect with an account that can see the whole schema, then verify object counts against the dictionary before reading any percentage.</li>

<li>Profile your NUMBER and DATE columns first and set the type mappings deliberately. Do this before the first conversion, not after the first load.</li>

<li>Script the conversion with the CLI so every run is identical and reviewable.</li>

<li>Sort red action items by object criticality, not by count. Twenty flags on a nightly batch job matter more than two hundred on a deprecated reporting view.</li>

<li>Write the semantic test suite before cutover: empty string versus NULL, date boundaries, numeric rounding, trigger ordering, transaction behavior in converted routines.</li>

<li>Instrument the target from day one. Query-level metrics in Amazon CloudWatch, or a Grafana stack against pg_stat_statements, will show you which extension pack wrappers are actually costing you before users complain.</li>
</ol>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is the AWS Schema Conversion Tool free?</h3>



<p class="wp-block-paragraph">The tool itself is a free download from AWS. What you pay for is everything around it: the compute you run it on, the target database, DMS replication instances for the data move, and the engineering time on the objects it hands back to you. That last item is the whole budget on most migrations.</p>



<h3 class="wp-block-heading">Does it migrate data as well as schema?</h3>



<p class="wp-block-paragraph">No. It converts structure and code. Moving rows is AWS DMS, which handles both a full load and ongoing change capture. They are designed to be used together, and each has its own set of behaviors worth understanding separately.</p>



<h3 class="wp-block-heading">Can I skip the extension pack entirely?</h3>



<p class="wp-block-paragraph">You can, but you are then hand-writing PostgreSQL equivalents for every Oracle built-in the pack would have covered. Configuring the tool to prefer <code>orafce</code> where possible is the usual compromise: fewer AWS-specific wrappers, open-source implementation, still automated. Full independence is achievable and is a refactoring project, not a setting.</p>



<h3 class="wp-block-heading">Why did my report say 95% automatic and the project still take months?</h3>



<p class="wp-block-paragraph">Because the percentage counts objects, ignores runtime semantics entirely, and excludes application code, scheduled jobs, security, and downstream consumers. Tier two and the out-of-scope list in this post are where the months go.</p>



<h3 class="wp-block-heading">Should I clean up the Oracle schema before or after conversion?</h3>



<p class="wp-block-paragraph">Before, every time. Dropping dead objects, unwrapping code you own, and removing synonyms on the source reduces the surface the tool has to reason about and shrinks the red list. It is also the only cleanup you can do while you still have a working system to test against.</p>



<h3 class="wp-block-heading">How do I verify the conversion actually worked?</h3>



<p class="wp-block-paragraph">Row counts and DDL comparison are the floor, not the ceiling. Run the same business queries against both engines and compare results, with deliberate coverage of NULL handling, date boundaries, sorting and collation, and numeric rounding. Anything that only shows up under concurrency needs a load test, not a smoke test.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The AWS Schema Conversion Tool for Oracle is a good parser and an honest one about the things it cannot parse. The red list is trustworthy. The trouble is that a green tick means &#8220;I emitted valid PostgreSQL,&#8221; not &#8220;this behaves the way it did.&#8221; Dates lose their time, unqualified NUMBER lands as slow arithmetic, empty strings stop being NULL, triggers reorder themselves alphabetically, and none of that appears as a problem in the report.</p>



<p class="wp-block-paragraph">Use it for what it is good at, which is the tedium you should never do by hand. Then spend your review time on the objects that passed, because that is where the expensive surprises live. Treat the assessment report as an inventory, never as a schedule.</p>



<h2 class="wp-block-heading">Need a second pair of eyes on an Oracle to PostgreSQL conversion?</h2>



<p class="wp-block-paragraph">Most of the value in this work is in the review, not the click. Things I help with:</p>



<ul class="wp-block-list">
<li>Reading an assessment report properly: separating object counts from real effort, and re-scoping red items against the PostgreSQL version you are actually deploying.</li>

<li>Auditing type mappings before you apply, with a column-level profile of NUMBER and DATE usage and a concrete mapping decision for each.</li>

<li>Building the semantic test suite that catches empty-string and NULL divergence, trigger ordering, and rounding differences before users find them.</li>

<li>Scripting the conversion through the SCT CLI so it is repeatable, reviewable, and lives in version control alongside the generated DDL.</li>

<li>Planning the extension pack burn-down: which <code>aws_oracle_ext</code> calls are costing you, and what the native PostgreSQL replacement looks like.</li>

<li>Mapping the out-of-scope surface, meaning scheduled jobs, grants, downstream reports and integrations, so cutover day has no surprises in it.</li>
</ul>



<p class="wp-block-paragraph">Send me an assessment report, a converted DDL file, or the action item list, and I will tell you what I would actually worry about in it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>

<p>The post <a href="https://john-nessime.com/blog/technical-guides/aws-schema-conversion-tool-oracle/">AWS Schema Conversion Tool for Oracle: What Converts and What Quietly Doesn&#8217;t</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Customer Sentiment Analysis From CRM and Support Data: Building a Score You Can Actually Trust</title>
		<link>https://john-nessime.com/blog/technical-guides/customer-sentiment-analysis-crm-support-data/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 22 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Comprehend]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[CRM Analytics]]></category>
		<category><![CDATA[Customer Support Analytics]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[EU AI Act]]></category>
		<category><![CDATA[NLP]]></category>
		<category><![CDATA[PII Redaction]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[Sentiment Analysis]]></category>
		<category><![CDATA[SQL]]></category>
		<category><![CDATA[Zendesk]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=250</guid>

					<description><![CDATA[<p>Most customer sentiment analysis pipelines run perfectly and still produce a number nobody should act on. Here are the failure families that cause it, from labels stamped on the wrong message to averages taken over a scale that was never numeric, plus a pipeline shape and a legal boundary worth knowing before you ship.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/customer-sentiment-analysis-crm-support-data/">Customer Sentiment Analysis From CRM and Support Data: Building a Score You Can Actually Trust</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The CX lead pings you on a Thursday afternoon: &#8220;Sentiment has been sitting at 3.2 for four months. Is the pipeline broken?&#8221;</p>



<p class="wp-block-paragraph">You check. Every extraction job ran. Every ticket got a label. No rows dropped, no schema drift, no dead letters. Nothing is broken. The number is just useless, and it has been useless since the day it shipped.</p>



<p class="wp-block-paragraph">That is the normal outcome for customer sentiment analysis built on CRM and support data. The connectors work, the model returns labels, the dashboard renders on schedule, and the output still cannot tell anyone whether customers are getting happier or angrier. These are not pipeline failures. They are measurement failures, which is exactly why your monitoring never catches them.</p>



<p class="wp-block-paragraph">This post walks through the failure families I check first when a sentiment programme is producing numbers nobody trusts: where the label gets stamped, what happens when you average an ordinal scale, who is missing from the data entirely, why a general-purpose model misreads support English, where the CRM join breaks, and one regulatory boundary that catches more teams than it should. Then a pipeline shape that holds up, and the checks worth running before anyone puts this on a slide.</p>



<h2 class="wp-block-heading">What the score is actually measuring</h2>



<p class="wp-block-paragraph">Before anything else, get specific about the unit of analysis. &#8220;Customer sentiment&#8221; is a phrase people use for at least four different measurements:</p>



<ul class="wp-block-list">
<li><strong>Message sentiment</strong> — the tone of one comment in one thread.</li>

<li><strong>Ticket sentiment</strong> — usually the first inbound message, sometimes a rollup.</li>

<li><strong>Contact sentiment</strong> — one human&#8217;s tone across all their tickets.</li>

<li><strong>Account sentiment</strong> — the thing the business actually cares about, and the one nobody computes correctly.</li>
</ul>



<p class="wp-block-paragraph">Most stacks compute the second and label it the fourth. That single substitution causes more bad decisions than any model choice you will make downstream. A model with mediocre accuracy applied at the right unit beats an excellent model applied at the wrong one.</p>



<h2 class="wp-block-heading">Failure one: the label is stamped at the wrong moment</h2>



<p class="wp-block-paragraph">This is the expensive one, and it is invisible from the pipeline side because nothing errors.</p>



<p class="wp-block-paragraph">Zendesk&#8217;s intelligent triage is the clearest example because the documentation is honest about it. Zendesk states that most intelligent triage classifications are based on a ticket&#8217;s first message only, with an admin-configurable option to update topic and sentiment from the latest end-user message. If nobody turned that option on, your &#8220;customer sentiment&#8221; field is a permanent record of how annoyed someone was in their opening paragraph.</p>



<p class="wp-block-paragraph">Think about what that does to a trend line. The opening message of a support ticket is negative almost by definition, because people do not open tickets to say things are going well. What you want to know is whether the interaction <em>ended</em> well. A ticket that opens furious and closes with a thank-you is a support win, and a first-message-only score records it as a loss forever.</p>



<p class="wp-block-paragraph">Two consequences follow that people miss:</p>



<ul class="wp-block-list">
<li>Your sentiment metric is structurally incapable of moving in response to anything the support team does. Hire better agents, cut response time in half, rewrite your macros: the number will not budge, because it was set before any of that happened.</li>

<li>It <em>will</em> move in response to marketing, onboarding and product changes upstream, because those change who opens tickets and how angry they are on arrival. So the metric drifts for reasons the team watching it cannot influence.</li>
</ul>



<p class="wp-block-paragraph">Zendesk also documents that when an agent corrects the sentiment field on a ticket, the correction does not retrain the model. Worth knowing before you build a human-review workflow on the assumption that it feeds back into anything. It gives you a cleaner column, not a better classifier.</p>



<p class="wp-block-paragraph">The fix is to decide your stamping policy explicitly and store it. Score the first inbound message, score the last inbound message, and store both, along with the delta. The delta is the interesting column. Sentiment recovery within a thread is a far better signal of support quality than absolute sentiment ever was.</p>



<h2 class="wp-block-heading">Failure two: averaging a scale that was never numeric</h2>



<p class="wp-block-paragraph">Zendesk exposes sentiment as five ordered values from Very negative to Very positive, and Zendesk Explore ships a prebuilt report for average customer sentiment on a one-to-five scale. Amazon Comprehend returns one of <code>POSITIVE</code>, <code>NEGATIVE</code>, <code>NEUTRAL</code> or <code>MIXED</code> plus per-class confidence scores. Google Cloud Natural Language returns a polarity score and a magnitude. Azure AI Language returns document- and sentence-level labels with confidences.</p>



<p class="wp-block-paragraph">Only some of those are numbers, and none of the label sets are interval scales. The gap between Very negative and Negative is not the same size as the gap between Neutral and Positive, and there is no defensible reason to assume it is. So when you compute <code>AVG(sentiment)</code>, you are doing arithmetic on rank labels and getting a value that has no units.</p>



<p class="wp-block-paragraph">That would be a pedantic complaint if it were only a purity issue. It is not, because support sentiment distributions are usually bimodal. People write in angry or they write in grateful. Very few tickets are genuinely neutral. Take the mean of a two-humped distribution and you land in the valley between the humps, in a bucket where almost nobody actually is. Worse, the mean is stable there: the humps can grow, shrink and swap size while the average barely moves. That is your flat 3.2.</p>



<p class="wp-block-paragraph">Use share metrics instead. They are ordinal-safe, they move, and they are directly actionable.</p>



<pre class="wp-block-code"><code>-- Share of negative tickets, weighted by account value.
-- Ordinal-safe: no arithmetic on the label itself, only counting.
SELECT
    date_trunc('week', t.created_at)                      AS week,
    COUNT(*)                                              AS tickets,
    COUNT(*) FILTER (
        WHERE t.sentiment IN ('very_negative', 'negative')
    )                                                     AS negative_tickets,
    ROUND(
        100.0 * COUNT(*) FILTER (
            WHERE t.sentiment IN ('very_negative', 'negative')
        ) / NULLIF(COUNT(*), 0)
    , 1)                                                  AS pct_negative,
    SUM(a.arr) FILTER (
        WHERE t.sentiment IN ('very_negative', 'negative')
    )                                                     AS arr_behind_negative
FROM   support_tickets  t
JOIN   crm_contacts     c ON c.id = t.contact_id
JOIN   crm_accounts     a ON a.id = c.account_id
WHERE  t.created_at &gt;= date_trunc('week', CURRENT_DATE) - INTERVAL '26 weeks'
GROUP  BY 1
ORDER  BY 1;</code></pre>



<p class="wp-block-paragraph">Three things are happening there worth calling out. <code>FILTER</code> counts rows in a category without touching the label as a number. <code>NULLIF</code> stops a zero-ticket week from throwing a division error and silently killing the job. And the last column is the one executives will actually read, because a percentage of tickets is a support metric while revenue sitting behind negative sentiment is a business metric.</p>



<h2 class="wp-block-heading">Failure three: the people who left never showed up in the data</h2>



<p class="wp-block-paragraph">Support data only contains customers who chose to contact you. That is a selection effect, not a sample.</p>



<p class="wp-block-paragraph">The dangerous version is this: negative sentiment falls, everyone celebrates, and the actual cause is that the angriest cohort stopped writing in because they had already decided to leave. Silence reads as improvement. In a quarter where churn is rising, a falling negative-sentiment share is a warning sign, not a win.</p>



<p class="wp-block-paragraph">You cannot fix this inside the support dataset, which is precisely why the CRM join matters. Two guards worth building:</p>



<ul class="wp-block-list">
<li><strong>Contact rate as a companion metric.</strong> Tickets per active account per month, tracked alongside sentiment. Sentiment improving while contact rate collapses is a disengagement pattern, not a satisfaction one.</li>

<li><strong>Went-quiet detection.</strong> Accounts that used to file tickets regularly and have filed none in a defined window. In most books of business this list is a better churn predictor than the sentiment score itself, and it costs one window function to produce.</li>
</ul>



<h2 class="wp-block-heading">Failure four: the model learned someone else&#8217;s language</h2>



<p class="wp-block-paragraph">General-purpose sentiment APIs are trained largely on review-style and social text. Support English is a different dialect, and the mismatches are systematic rather than random.</p>



<ul class="wp-block-list">
<li><strong>Problem statements are not complaints.</strong> &#8220;The export fails with a 500&#8221; is a factual bug report. Review-trained models frequently score it negative because failure vocabulary dominates.</li>

<li><strong>Politeness masks severity.</strong> A calmly worded message from an enterprise admin saying the production sync has been down since Tuesday is your worst ticket of the week and will score close to neutral.</li>

<li><strong>Domain vocabulary inverts.</strong> &#8220;Killed the process&#8221;, &#8220;the job died&#8221;, &#8220;aborted the run&#8221; are neutral technical descriptions carrying heavy negative weight in general corpora.</li>

<li><strong>Your own text pollutes the corpus.</strong> Auto-acknowledgements, signatures, quoted reply chains and CSAT survey footers all get scored if you feed the raw comment body. Quoted history is the big one: score a five-reply thread naively and you have scored message one five times.</li>

<li><strong>Non-native English reads flat.</strong> Simpler sentence structure and less emotional vocabulary push scores toward neutral for a whole segment of your customer base, which quietly biases any regional comparison.</li>
</ul>



<p class="wp-block-paragraph">Independent evaluations of the major cloud sentiment APIs on general text tend to land well below the accuracy figures in vendor marketing, and the providers frequently disagree with each other on the same input. Treat published benchmark numbers as irrelevant to your decision. The only number that matters is accuracy on a few hundred of your own tickets, hand-labelled by someone who understands your product.</p>



<h3 class="wp-block-heading">Building the evaluation set</h3>



<ol class="wp-block-list">
<li>Pull a stratified random sample of tickets: across channels, across regions, across account tiers. Do not sample only recent tickets, or you bake in whatever was happening that month.</li>

<li>Have two people label independently, using a written rubric with worked examples of each class.</li>

<li>Measure agreement between the humans first. If your two support leads disagree on a third of the tickets, the task is underspecified and no model will fix that. Tighten the rubric and relabel.</li>

<li>Only then score the candidate models against the human labels, and look at the confusion matrix rather than accuracy. Where a model confuses classes tells you whether it is usable.</li>

<li>Freeze that set. It becomes your regression test when a vendor silently updates a model version.</li>
</ol>



<p class="wp-block-paragraph">Step three is the one teams skip, and it is the one that saves the project. Human agreement is the ceiling on model performance. There is no point chasing 90% against labels your own experts only agree on 70% of the time.</p>



<h2 class="wp-block-heading">Failure five: the CRM join is where the value and the breakage both live</h2>



<p class="wp-block-paragraph">A sentiment label with no account attached is trivia. Two hundred irritated free-tier users and one irritated account paying for a third of your renewals are not the same event, and the raw ticket table cannot tell them apart.</p>



<p class="wp-block-paragraph">The join from helpdesk to CRM is almost always on email, and email is a terrible key. Expect all of this:</p>



<ul class="wp-block-list">
<li>Shared inboxes such as ops@ or billing@, mapping to no single contact or to several.</li>

<li>Personal addresses used by people who work at a customer, invisible to any domain-matching logic.</li>

<li>Contacts who left and were deleted in Salesforce, HubSpot or Zoho CRM, orphaning every historical ticket they filed.</li>

<li>Account hierarchies, where the subsidiary files the ticket and the parent holds the contract.</li>

<li>Mergers and renames, which quietly rewrite history if your warehouse dimension is not versioned.</li>
</ul>



<p class="wp-block-paragraph">The engineering answer is boring and effective: treat unmatched tickets as a first-class, monitored quantity rather than something the join silently discards. Track match rate as a data quality metric with an alert threshold. When it drops, something changed in the CRM, and you want to know that before the number on the dashboard shifts for a reason nobody can explain.</p>



<p class="wp-block-paragraph">Also version the account dimension. If you overwrite the tier or the ARR in place, then every historical aggregate silently rewrites itself the next time someone upgrades, and last quarter&#8217;s report stops reproducing.</p>



<h2 class="wp-block-heading">The legal boundary that catches sentiment projects</h2>



<p class="wp-block-paragraph">This one is worth reading carefully if you have EU customers or EU staff, because the line does not fall where most people assume, and a lot of contact-centre software sells both sides of it in one bundle.</p>



<p class="wp-block-paragraph">Article 5(1)(f) of the EU AI Act prohibits AI systems that infer emotions of a person in the workplace or in education settings. The prohibitions in Article 5 have applied since 2 February 2025, and breaches sit in the Act&#8217;s highest penalty band. What matters for a sentiment project is the scope, which is narrow on two axes at once:</p>



<ul class="wp-block-list">
<li>It applies to emotion inference <strong>from biometric data</strong> — faces, voiceprints, physiological signals. The European Commission&#8217;s guidance takes the position that inferring emotion from written text is not based on biometric data and therefore falls outside the prohibition.</li>

<li>It applies to <strong>workplace and education contexts</strong>. Your customers writing into a helpdesk are neither.</li>
</ul>



<p class="wp-block-paragraph">So scoring the text of customer tickets and CRM notes generally sits outside the ban. Running voice-based emotion or stress analysis on your own support agents&#8217; calls, for coaching or quality scoring, is the case the prohibition was written for. Plenty of platforms ship customer-facing text sentiment and agent-facing voice sentiment as adjacent features in the same product, and the pitch does not distinguish them. If you are the deployer, the compliance obligation is yours, and a vendor&#8217;s assurance is not a defence.</p>



<p class="wp-block-paragraph">Separately, Article 50 carries transparency obligations for emotion recognition systems where they remain lawful. And none of this displaces GDPR: ticket bodies are full of personal data, and shipping them to a third-party API is a processing decision that needs a basis, a processor agreement and a residency answer.</p>



<p class="wp-block-paragraph">I am an engineer, not a lawyer. Treat the above as the shape of the question to take to counsel, not as advice.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">A pipeline shape that survives contact</h2>



<p class="wp-block-paragraph">Nothing exotic. The discipline is in what you store, not what you call.</p>



<ol class="wp-block-list">
<li><strong>Extract incrementally</strong> from the helpdesk and CRM on an updated-at cursor, not a full reload. Land raw payloads immutably before you transform anything, so a bad model run is re-runnable without re-hitting the source API.</li>

<li><strong>Clean the text.</strong> Strip quoted reply chains, signatures, automated acknowledgements and survey footers. Keep the cleaned text as its own column so a scoring bug is debuggable later.</li>

<li><strong>Redact personal data</strong> before it leaves your boundary. Regex handles the structured patterns: emails, phone numbers, card and account numbers, order IDs. Named-entity detection handles names and addresses. Amazon Comprehend exposes <code>DetectPiiEntities</code> and <code>ContainsPiiEntities</code> for this if you are already in AWS.</li>

<li><strong>Score in batches</strong>, with the model identifier and version written into every row.</li>

<li><strong>Aggregate at account level</strong> with share metrics and value weighting, against a versioned account dimension.</li>

<li><strong>Alert on movement, not on level.</strong> Nobody acts on &#8220;sentiment is 3.2&#8221;. People act on &#8220;this named account&#8217;s negative share tripled in two weeks&#8221;.</li>
</ol>



<p class="wp-block-paragraph">Step four&#8217;s version column is the one people leave out and regret. Managed sentiment endpoints get retrained. When labels shift on unchanged text, the version column is the difference between a five-minute explanation and a week of forensics.</p>



<p class="wp-block-paragraph">Here is what a batch call looks like against Amazon Comprehend, which is a reasonable default if your warehouse already lives in AWS:</p>



<pre class="wp-block-code"><code>import boto3

comprehend = boto3.client("comprehend", region_name="eu-west-1")

# BatchDetectSentiment accepts a maximum of 25 documents per call,
# and DetectSentiment caps each document at 5 KB of UTF-8 text.
# Chunk to both limits or the call fails for the whole batch.
BATCH_SIZE = 25
MAX_BYTES  = 5000

def score_batch(texts):
    payload = [t.encode("utf-8")[:MAX_BYTES].decode("utf-8", "ignore")
               for t in texts[:BATCH_SIZE]]
    resp = comprehend.batch_detect_sentiment(
        TextList=payload,
        LanguageCode="en",
    )
    # ResultList is index-aligned to the input; ErrorList holds
    # per-document failures. Read both or you lose rows silently.
    return resp["ResultList"], resp["ErrorList"]</code></pre>



<p class="wp-block-paragraph">The truncation on the encoded bytes rather than the character count is deliberate: the limit is measured in bytes, and a ticket full of accented characters or CJK text hits it much earlier than the character count suggests. The <code>ErrorList</code> read matters for the same reason a dead letter queue matters. A partial batch success that you never inspect is data loss that looks like a clean run.</p>



<p class="wp-block-paragraph">For a quick sanity check on a single string before wiring any of this up, the CLI is faster than writing code:</p>



<pre class="wp-block-code"><code>aws comprehend detect-sentiment 
    --region eu-west-1 
    --language-code "en" 
    --text "Still waiting on the refund you promised last week."

{
    "SentimentScore": {
        "Mixed": 0.0033542951568961143,
        "Positive": 0.0086313202530145600,
        "Neutral": 0.1495875907897949,
        "Negative": 0.8384268 
    },
    "Sentiment": "NEGATIVE",
    "LanguageCode": "en"
}</code></pre>



<p class="wp-block-paragraph">Note that the response carries per-class confidences, not just the winning label. Store the whole <code>SentimentScore</code> object. A ticket scored NEGATIVE at 0.51 and one scored NEGATIVE at 0.98 are very different tickets, and the label alone throws that away. Confidence is also what lets you route only the ambiguous cases to a more expensive model or a human, instead of paying for everything.</p>



<h3 class="wp-block-heading">Choosing where the model runs</h3>



<p class="wp-block-paragraph">Three broad options, and the trade-offs are real in both directions.</p>



<ul class="wp-block-list">
<li><strong>Native helpdesk features.</strong> Zendesk intelligent triage, Salesforce Einstein, HubSpot and Zoho CRM all ship some form of built-in sentiment. Zero integration work, labels land on the ticket where agents can see them, and routing rules can consume them directly. You get the vendor&#8217;s definition of sentiment, no control over stamping policy beyond the toggles they expose, and data that is awkward to move.</li>

<li><strong>Managed NLP APIs.</strong> Amazon Comprehend, Azure AI Language and Google Cloud Natural Language. Per-character billing, no infrastructure, useful extras such as aspect-level or targeted sentiment. Comprehend&#8217;s targeted sentiment, which attaches sentiment to specific entities in the text, is English-only, and that constraint bites hard if your support volume is multilingual. Bill by volume of text, so a batch backfill of five years of tickets is a genuinely different cost event from steady-state scoring.</li>

<li><strong>Self-hosted transformers.</strong> A fine-tuned encoder model from the Hugging Face ecosystem, running on your own box. Best accuracy on your domain once tuned, no per-call cost, no text leaving your perimeter, which resolves most residency arguments before they start. In exchange you own model serving, evaluation and drift. A mid-tier dedicated server from a provider like Contabo or Hetzner runs CPU inference for batch workloads perfectly well, and batch is what this workload is. You do not need a GPU to score yesterday&#8217;s tickets overnight.</li>
</ul>



<p class="wp-block-paragraph">If you have no strong residency constraint and a small volume, start with a managed API. The self-hosted path is worth it when you have enough labelled data to fine-tune, or a legal reason the text cannot leave. Not before.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Sentiment is flat regardless of what the team does</h3>



<p class="wp-block-paragraph">Two likely causes. Either you are averaging an ordinal scale over a bimodal distribution, or you are scoring first messages only. Check the distribution shape first, since it takes one query. If it has two humps, switch to share metrics before you touch anything else.</p>



<h3 class="wp-block-heading">Everything scores neutral</h3>



<p class="wp-block-paragraph">Usually a text-cleaning problem. If quoted reply chains, signatures and boilerplate survive into the scored text, the actual message is diluted by noise and the model regresses to neutral. Print the exact string you are sending for twenty tickets and read them. This bug is invisible in aggregate and obvious in the raw payload.</p>



<h3 class="wp-block-heading">The trend shifted overnight with no matching business event</h3>



<p class="wp-block-paragraph">Check the model version column and the CRM match rate on the same day. A managed endpoint retrain and a CRM field change both produce exactly this signature. If you did not store the version, this is the moment you start.</p>



<h3 class="wp-block-heading">One region always looks happier</h3>



<p class="wp-block-paragraph">Before concluding anything about that region, check the language distribution. Non-native English and machine-translated tickets both drift toward neutral, which shows up as &#8220;less negative&#8221; in a share metric. Compare like with like by language, or score in the original language where the provider supports it.</p>



<h3 class="wp-block-heading">Batch scoring jobs fail intermittently</h3>



<p class="wp-block-paragraph">Almost always the document size limit, hit by one oversized ticket with a pasted log file or stack trace in the body. Truncate on encoded byte length before the call and check the per-document error list rather than assuming a successful HTTP response means every document scored.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Reporting an average of an ordinal label set and calling it a score.</li>

<li>Scoring the first message and describing the result as customer satisfaction.</li>

<li>Treating unmatched tickets as an inner-join casualty instead of a monitored metric.</li>

<li>Sending raw ticket bodies to a third-party API with no redaction step.</li>

<li>Discarding confidence scores and keeping only the winning label.</li>

<li>Trusting a vendor accuracy figure over a few hundred of your own hand-labelled tickets.</li>

<li>Overwriting the account dimension in place, so historical reports quietly stop reproducing.</li>

<li>Assuming a helpdesk vendor&#8217;s compliance posture transfers to you as the deployer.</li>
</ul>



<h2 class="wp-block-heading">Best practices for customer sentiment analysis</h2>



<ul class="wp-block-list">
<li>Write down the unit of analysis before writing any code, and make sure the dashboard label matches it.</li>

<li>Store first-message sentiment, last-message sentiment and the delta. Report the delta.</li>

<li>Use share-of-negative and value-weighted share, never a mean over labels.</li>

<li>Keep a frozen, hand-labelled evaluation set and re-run it on a schedule as a regression test.</li>

<li>Persist the model identifier, model version and full confidence distribution on every scored row.</li>

<li>Redact personal data before the text crosses a network boundary, and log what was redacted, not the values.</li>

<li>Monitor CRM match rate, contact rate and language mix alongside the sentiment number itself.</li>

<li>Alert on named-account movement, not on the global level.</li>

<li>Know which side of the biometric line each feature you enable sits on, and put the answer in writing.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is customer sentiment analysis accurate enough to act on?</h3>



<p class="wp-block-paragraph">For prioritisation and trend detection, generally yes. For individual decisions about a specific customer, no. The reliable pattern is to use it as a filter that surfaces conversations for a human to read, rather than as a verdict. Any single ticket&#8217;s label should be treated as a hint, not a fact.</p>



<h3 class="wp-block-heading">Should I use my helpdesk&#8217;s built-in sentiment or build my own pipeline?</h3>



<p class="wp-block-paragraph">Start with the built-in feature. It is free or cheap, it lands where agents work, and it tells you within a month whether anyone actually uses sentiment for anything. Build your own when you need a different stamping policy, cross-system aggregation, account-value weighting, or control over where the text is processed. Those are the four reasons that justify the effort.</p>



<h3 class="wp-block-heading">Can I use an LLM instead of a dedicated sentiment API?</h3>



<p class="wp-block-paragraph">You can, and the quality on nuance, sarcasm and domain vocabulary is usually better because you can describe your product and your customers in the prompt. The costs are latency, price per token at backfill scale, and output stability. Pin the model version, constrain the output to a fixed label set, and set temperature to zero, or you will get labels that drift between runs on identical text. A pragmatic middle ground is a cheap classifier by default with an LLM as a fallback on low-confidence cases.</p>



<h3 class="wp-block-heading">How much historical data do I need before the trend means anything?</h3>



<p class="wp-block-paragraph">Enough to cover your seasonality, which for most businesses means at least a full year. Support volume and tone move with billing cycles, product releases, holidays and renewal periods. A quarter of data will show you a trend that is really just the calendar.</p>



<h3 class="wp-block-heading">Does sentiment analysis on support tickets fall under the EU AI Act ban?</h3>



<p class="wp-block-paragraph">The Article 5(1)(f) prohibition targets emotion inference from biometric data in workplace and education contexts, and Commission guidance treats text-based sentiment as outside that scope. Scoring customer-written text is generally not caught. Voice or facial emotion analysis applied to your own employees is squarely the thing it prohibits. Transparency duties and GDPR obligations still apply regardless, and this is a question for your counsel rather than your engineer.</p>



<h3 class="wp-block-heading">What is a good target for share of negative tickets?</h3>



<p class="wp-block-paragraph">There is no cross-industry benchmark worth quoting, because the number depends entirely on your product category, your support channels and how the model was calibrated. Your own baseline is the only meaningful comparison. Measure for a full cycle, then set targets against that.</p>



<h3 class="wp-block-heading">How do I handle multilingual support volume?</h3>



<p class="wp-block-paragraph">Detect the language, store it as a column, and segment every report by it. Do not translate to English and then score, since translation flattens emotional register and adds a second error source. Check which of your provider&#8217;s features are English-only before designing around them; some advanced modes, including Comprehend&#8217;s targeted sentiment, are narrower than the base sentiment endpoint.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A customer sentiment analysis pipeline that runs cleanly every night and produces a number nobody can act on is not a partially working system. It is a broken one, and the breakage is upstream of anything your monitoring can see.</p>



<p class="wp-block-paragraph">So before optimising a model or shopping for a platform, answer three questions in writing: which message gets scored, what unit the number describes, and which accounts it is weighted by. Get those right with a mediocre classifier and you have something useful. Get them wrong with the best model on the market and you have a very accurate description of how annoyed people were in their opening sentence.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help making this work on your data?</h2>



<p class="wp-block-paragraph">I build and fix the data plumbing behind customer analytics. On this particular problem, that usually means:</p>



<ul class="wp-block-list">
<li>Auditing an existing sentiment metric to find out why it never moves, and rebuilding it on share and delta metrics that do.</li>

<li>Building incremental extraction from Zendesk, Salesforce, HubSpot or Zoho CRM into a warehouse, with cursor state, retries and dead-letter handling that survives an API outage.</li>

<li>Fixing the helpdesk-to-CRM identity join, including shared inboxes, account hierarchies and a versioned account dimension so historical reports keep reproducing.</li>

<li>Standing up a redaction layer so ticket text is scrubbed of personal data before it reaches any third-party model endpoint.</li>

<li>Running a proper evaluation: stratified sample, dual human labelling, inter-rater agreement, confusion matrices across candidate models, and a frozen regression set.</li>

<li>Setting up account-level alerting in Grafana or your BI tool that fires on movement in named accounts rather than on a global average.</li>
</ul>



<p class="wp-block-paragraph">If you have a dashboard that has been flat for months, send me the query behind it and a screenshot of the distribution. That is usually enough to tell you what is wrong before we talk about scope.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/customer-sentiment-analysis-crm-support-data/">Customer Sentiment Analysis From CRM and Support Data: Building a Score You Can Actually Trust</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Oracle to Aurora Migration: The Failures Nobody Catches Until Cutover</title>
		<link>https://john-nessime.com/blog/devops/oracle-to-aurora-migration/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 17 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Aurora]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS DMS]]></category>
		<category><![CDATA[Change Data Capture]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Cutover Planning]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Database Migration]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Oracle]]></category>
		<category><![CDATA[PostgreSQL]]></category>
		<category><![CDATA[Schema Conversion]]></category>
		<category><![CDATA[SQL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=218</guid>

					<description><![CDATA[<p>Most Oracle to Aurora migrations don't fail loudly. They finish green, pass validation, and quietly hand you truncated LOBs, sequences stuck at 1, and empty strings that used to be NULL. Here are the four failure families to check before you cut over, with the SQL and task settings that catch them.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/oracle-to-aurora-migration/">Oracle to Aurora Migration: The Failures Nobody Catches Until Cutover</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Three weeks after cutover, someone in finance sends you a spreadsheet. Forty-one invoice records have a description field that stops mid-sentence. Not blank, not garbled. Just shorter than it used to be.</p>



<p class="wp-block-paragraph">So you open the AWS DMS console. The task is green. Full load completed, change data capture running, zero errors. Data validation passed on every table in scope.</p>



<p class="wp-block-paragraph">Both of those things are true at the same time, and that is the part that catches people out. An Oracle to Aurora migration almost never fails with a stack trace and a rollback. It fails with a green dashboard and a handful of rows that are subtly, permanently wrong.</p>



<p class="wp-block-paragraph">This post is not a walkthrough of the console wizard. AWS documents that well enough. It covers the four families of failure I would go looking for before signing off on an Oracle to Aurora migration: large objects that arrive truncated, change capture that records less than you assume, the semantic gaps between Oracle and PostgreSQL that compile cleanly and behave differently, and the database state that DMS simply does not move for you. Then how to validate properly, and what to do when a task stalls.</p>



<h2 class="wp-block-heading">Pick the Aurora engine before you pick the migration tool</h2>



<p class="wp-block-paragraph">Amazon Aurora comes in a PostgreSQL-compatible edition and a MySQL-compatible edition, and the choice shapes everything downstream. Do not let it get decided by whoever on the team happens to like MySQL.</p>



<p class="wp-block-paragraph">Aurora PostgreSQL is the realistic default coming from Oracle, and the reason is procedural code. PL/SQL and PL/pgSQL are close enough that automated conversion gets you a long way: packages, cursors, exceptions, custom types, and analytic functions all have recognisable counterparts. PostgreSQL also has the <code>orafce</code> extension, which reimplements a chunk of Oracle&#8217;s built-in functions and packages inside a compatibility schema, so a lot of converted code can keep calling things it already calls.</p>



<p class="wp-block-paragraph">The honest case for Aurora MySQL: if the Oracle database is essentially a table store with the business logic living in the application tier, and your team already runs MySQL in production, you will move faster on the engine you can operate at 3am. Migration difficulty is real, but so is the cost of running a database nobody on the rota understands. If your schema has no packages, no triggers worth keeping and no PL/SQL, that argument holds.</p>



<p class="wp-block-paragraph">Where it falls apart is the moment you find twelve thousand lines of PL/SQL that someone will have to rewrite by hand. At that point the engine choice has stopped being an operational preference and become a rewrite project.</p>



<h3 class="wp-block-heading">Schema conversion: SCT or the DMS console</h3>



<p class="wp-block-paragraph">There are two ways to convert the schema. The AWS Schema Conversion Tool is a desktop Java application that connects to both databases, reads the source objects, and emits target DDL plus an assessment report of everything it could not convert. DMS Schema Conversion is the same idea run from the DMS console using instance profiles and data providers, with no local install and no JDBC drivers to manage.</p>



<p class="wp-block-paragraph">Start with the console version. It is less setup and the assessment report is the thing you actually want early: a list of objects with action items, so you can size the manual work before committing to a date. Fall back to the desktop tool when you need something the console flow does not cover, such as converting application SQL files or SQL*Plus scripts.</p>



<p class="wp-block-paragraph">Either way, treat the assessment report as a project plan, not a progress bar. The red items are the schedule.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: LOB columns that arrive shorter than they left</h2>



<p class="wp-block-paragraph">This is the invoice description problem, and it is the single most common way an Oracle to Aurora migration loses data without telling anyone loudly.</p>



<p class="wp-block-paragraph">DMS gives you three ways to handle CLOB, NCLOB and BLOB columns:</p>



<ul class="wp-block-list">
<li><strong>Full LOB mode</strong> moves every LOB regardless of size. DMS has no idea how big they are, so it moves them one at a time, piece by piece. Correct, and slow.</li>



<li><strong>Limited LOB mode</strong> asks you for a maximum size up front. DMS pre-allocates memory and loads LOBs in bulk, which is dramatically faster. Anything larger than the limit is <em>truncated to the limit</em>, and a warning goes into the task log.</li>



<li><strong>Inline LOB mode</strong> sets a threshold. LOBs under it go inline with the row; larger ones fall back to the full LOB path with a lookup against the source. Good when most of your LOBs are small and a few are not.</li>
</ul>



<p class="wp-block-paragraph">Read the middle one again. Limited LOB mode is the default answer to &#8220;the full load is too slow&#8221;, it is the mode most tutorials use, and it silently clips your data. The warning lands in the task log, mixed in with everything else, and nobody is reading task logs line by line on a table with four million rows.</p>



<p class="wp-block-paragraph">Worse, the task still reports success. Truncation is not an error condition in DMS. It is expected behaviour for the mode you asked for.</p>



<h3 class="wp-block-heading">Measure before you set the limit</h3>



<p class="wp-block-paragraph">The fix is boring: go and find out how big your LOBs actually are, per column, on the source. Run this against Oracle for every LOB column in scope.</p>



<pre class="wp-block-code"><code>-- Largest LOB values in a single column, biggest first.
-- dbms_lob.getlength returns length in characters for CLOB/NCLOB,
-- bytes for BLOB. Check the top rows, not the average.
SELECT dbms_lob.getlength(description) AS lob_length
FROM   app.invoices
ORDER  BY dbms_lob.getlength(description) DESC
FETCH  FIRST 10 ROWS ONLY;

-- Same thing expressed in KB, which is the unit the DMS
-- "Max LOB size (K)" setting expects.
SELECT MAX(dbms_lob.getlength(description)) / 1024 AS max_kb
FROM   app.invoices;</code></pre>



<p class="wp-block-paragraph">Two things to know once you have the numbers.</p>



<p class="wp-block-paragraph">First, with Oracle as a source, DMS treats LOBs as VARCHAR data wherever it can, because bulk-fetching them is much faster than the LOB API. Oracle&#8217;s VARCHAR ceiling is 32K, so a limited LOB size under 32K is where the mode performs best. If your real maximum is comfortably under that, limited LOB mode is genuinely the right call.</p>



<p class="wp-block-paragraph">Second, memory. During full load, DMS pre-allocates roughly the max LOB size multiplied by the commit rate multiplied by the number of LOB columns. Push the limit up and that product grows fast. When the replication instance cannot allocate it, it starts swapping, and your &#8220;faster&#8221; mode becomes slower than full LOB mode. If limited LOB mode is dragging, drop the commit rate before you drop the size limit.</p>



<h3 class="wp-block-heading">Split the LOB tables into their own task</h3>



<p class="wp-block-paragraph">The pattern I reach for first: one task with limited LOB mode for the bulk of the schema, a second task for the handful of tables with genuinely large LOBs, using full or inline mode. You can also override task-level LOB settings per table inside the table mapping rules, which keeps it to a single task if you prefer.</p>



<pre class="wp-block-code"><code>{
  "TargetMetadata": {
    "SupportLobs": true,
    "FullLobMode": false,
    "LimitedSizeLobMode": true,
    "LobMaxSize": 16,
    "InlineLobMaxSize": 0,
    "LobChunkSize": 64,
    "BatchApplyEnabled": false
  }
}</code></pre>



<p class="wp-block-paragraph">That is limited mode with a 16 KB ceiling. To use inline mode instead, set <code>FullLobMode</code> to true and give <code>InlineLobMaxSize</code> a non-zero value, which is the threshold below which LOBs travel inline.</p>



<p class="wp-block-paragraph">One requirement that bites during ongoing replication: for CDC, tables with LOB columns need a primary key, because DMS looks the LOB value up in the source rather than reading it out of the redo stream. Full load does not care. CDC does. A LOB table with no primary key will load fine and then fail to replicate changes.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family two: CDC that captures less than you think</h2>



<p class="wp-block-paragraph">Ongoing replication is what buys you a short cutover window. It is also where the assumptions hide.</p>



<p class="wp-block-paragraph">DMS reads Oracle&#8217;s redo logs. For that to work at all, the source has to be in ARCHIVELOG mode with enough retention that DMS can still find the logs it needs after a pause, a network blip or an overnight stall. Retention is the one people get wrong: a task that stops for six hours against a two-hour retention window does not resume, it fails, and you restart from a full load.</p>



<h3 class="wp-block-heading">Supplemental logging is not optional</h3>



<p class="wp-block-paragraph">Oracle&#8217;s redo logs, by default, contain enough information for Oracle to recover the database. That is not the same as enough information to reconstruct a row-level change for a different database. Supplemental logging is what closes that gap.</p>



<pre class="wp-block-code"><code>-- Minimal supplemental logging at the database level.
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;

-- Identification key logging: writes every column of a row's
-- primary key into redo on UPDATE, even when the key didn't change.
-- Without this, DMS cannot reliably match the row on the target.
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS;

-- Per-table alternative, if you only need a subset of tables
-- and want to keep the source overhead down.
ALTER TABLE app.invoices ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS;</code></pre>



<p class="wp-block-paragraph">The case that actually bites is a table with no primary key and no unique index. For those, key-level logging gives DMS nothing to work with, and it needs supplemental logging on <em>all</em> columns so the before-and-after image of the row lands in redo. Miss that, and DMS cannot build a safe UPDATE or DELETE statement for the target. You will not notice during full load. You will notice when an update on the source produces no change on the target, or changes the wrong row.</p>



<p class="wp-block-paragraph">DMS ships premigration assessments that check exactly this. Run them. There are individual checks for supplemental logging on keyed tables, supplemental logging on all columns for unkeyed tables, ARCHIVELOG configuration, the CDC method in use, target permissions, and triggers enabled on the target. It takes minutes and it replaces a category of guesswork.</p>



<h3 class="wp-block-heading">LogMiner or Binary Reader</h3>



<p class="wp-block-paragraph">DMS can read redo two ways. Oracle LogMiner is an Oracle-side API; DMS calls it and Oracle does the parsing. Binary Reader is DMS&#8217;s own parser reading the redo files directly.</p>



<p class="wp-block-paragraph">If you specify nothing, you get LogMiner. That is usually correct. It needs fewer privileges, it handles ASM without extra plumbing, and it supports Oracle features like encryption and compression that the direct parser does not cover. The grants are small:</p>



<pre class="wp-block-code"><code>GRANT EXECUTE ON DBMS_LOGMNR TO dms_user;
GRANT SELECT ON V_$LOGMNR_LOGS TO dms_user;
GRANT SELECT ON V_$LOGMNR_CONTENTS TO dms_user;
GRANT LOGMINING TO dms_user;</code></pre>



<p class="wp-block-paragraph">Binary Reader earns its keep in one situation: a source generating redo faster than LogMiner can chew through it, or several concurrent migration tasks against the same database. LogMiner runs inside the source instance, so its CPU cost lands on the database you are trying not to disturb. Binary Reader shifts that work to the replication instance.</p>



<p class="wp-block-paragraph">The trade-off is setup. Binary Reader needs additional privileges and file-level access to the redo logs, which means either a directory the replication instance can reach or a copy step. If your source is a busy production Oracle instance and CDC latency is climbing steadily rather than spiking, that is the signal to switch. Otherwise stay on LogMiner and spend the complexity budget elsewhere.</p>



<p class="wp-block-paragraph">Whichever you pick, put CDC latency on a dashboard for the whole replication window, not just cutover day. The DMS task metrics publish to CloudWatch, and pulling them into whatever you already run for alerting, whether that is Grafana Cloud, Datadog or New Relic, is worth the twenty minutes. Source latency climbing while target latency stays flat means you are not reading redo fast enough. Both climbing together usually means the target cannot apply fast enough.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family three: the semantics Oracle and PostgreSQL do not share</h2>



<p class="wp-block-paragraph">Schema conversion produces code that compiles. Compiling is not the same as behaving identically, and this is where a migration goes wrong in ways no tool flags.</p>



<h3 class="wp-block-heading">Empty string is not NULL</h3>



<p class="wp-block-paragraph">Oracle treats the empty string and NULL as the same thing. PostgreSQL does not. This one difference propagates into comparisons, concatenation, unique constraints, and any procedural code that checks whether a value is &#8220;empty&#8221;.</p>



<pre class="wp-block-code"><code>-- PostgreSQL
SELECT '' IS NULL;        -- false
SELECT NULL || 'abc';     -- NULL
SELECT '' || 'abc';       -- abc

-- Oracle
SELECT CASE WHEN '' IS NULL THEN 'yes' ELSE 'no' END FROM dual;  -- yes
SELECT NULL || 'abc' FROM dual;                                  -- abc
SELECT '' || 'abc' FROM dual;                                    -- abc</code></pre>



<p class="wp-block-paragraph">Every one of those lines is a place a report can quietly change its numbers. Unique constraints are worse. Oracle stores an empty string as NULL, and NULLs are not compared for uniqueness, so it will happily accept many rows with an empty value in a uniquely constrained column. PostgreSQL sees zero-length strings, which are equal to each other, so the second one is rejected. Same schema, same data, different outcome, and the failure surfaces as an application error nobody can reproduce locally.</p>



<p class="wp-block-paragraph">There is no switch for this. Either the application changes, or you wrap the affected functions, or you normalise on write with a trigger or a check constraint. Decide which, per column, before cutover. The orafce extension helps with function behaviour but it is not a global compatibility mode, and enabling the parts that change NULL semantics has its own consequences.</p>



<h3 class="wp-block-heading">The rest of the list</h3>



<ul class="wp-block-list">
<li><strong>ROWID</strong> has no PostgreSQL equivalent. DMS Schema Conversion can emulate it with a bigint or a character varying column, but if application code depends on ROWID ordering or reuse semantics, emulation will not save you.</li>



<li><strong>SYSDATE and time zones.</strong> Conversion settings let you choose whether to emulate Oracle&#8217;s time zone handling or use native PostgreSQL behaviour. Native is faster. It is only safe if the database and the application genuinely run in the same time zone.</li>



<li><strong>TO_CHAR, TO_DATE, TO_NUMBER format masks.</strong> Oracle accepts parameters PostgreSQL does not. The converter emulates them by default, which is correct and slower. Turning emulation off is a per-codebase decision, not a global one.</li>



<li><strong>DUAL</strong> does not exist. PostgreSQL does not need a FROM clause, so most references just get dropped, but any dynamic SQL that builds a query string around DUAL needs finding.</li>



<li><strong>Synonyms</strong> have no direct equivalent. Search path adjustments and views cover most cases.</li>



<li><strong>Materialized views</strong> can be converted to real materialized views or to plain tables. The refresh semantics differ, so check what the reporting layer expects.</li>



<li><strong>NUMBER without a scale.</strong> An Oracle NUMBER column with no scale specified converts to something with a default, and if the source holds values wider than that default you get rounding or overflow. There is a premigration assessment specifically for this.</li>
</ul>



<p class="wp-block-paragraph">One data-level quirk worth knowing: if an Oracle column contains a NULL character (hex U+0000), DMS converts it to a space (U+0020) on a PostgreSQL target, because PostgreSQL cannot store it in a text column. Rare, but if you are migrating anything that stuffed binary into a VARCHAR, that is a real change to your data.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family four: the state DMS never moves</h2>



<p class="wp-block-paragraph">DMS moves rows. It does not move everything that makes a database work, and the gaps are not obvious until you point traffic at the new cluster.</p>



<h3 class="wp-block-heading">Sequences</h3>



<p class="wp-block-paragraph">Schema conversion creates the sequences. DMS copies the rows. Nothing advances the sequence counters to match the data that just arrived, so your target sequences sit at their starting value while the table already contains eight million rows.</p>



<p class="wp-block-paragraph">The first insert after cutover collides with an existing primary key. Then the second. Then every write for the next eight million attempts.</p>



<p class="wp-block-paragraph">Fix this after you stop replication and before you open traffic, not earlier, or you will do it twice.</p>



<pre class="wp-block-code"><code>-- Advance a sequence to match the data already loaded.
-- Run once per sequence, after CDC has stopped.
SELECT setval('app.invoices_id_seq',
              (SELECT max(id) FROM app.invoices));

-- Confirm before you open the doors.
SELECT last_value FROM app.invoices_id_seq;</code></pre>



<h3 class="wp-block-heading">Triggers on the target</h3>



<p class="wp-block-paragraph">If your converted schema has triggers and they are enabled during the load, they fire on every row DMS inserts. Audit tables get populated with a migration&#8217;s worth of fake activity. Derived columns get recalculated against half-loaded data. Anything that writes to another table doubles your write volume.</p>



<p class="wp-block-paragraph">Disable target triggers for the load, re-enable at cutover. There is a premigration assessment that checks for enabled triggers on target tables in task scope, which is a decent backstop against forgetting.</p>



<h3 class="wp-block-heading">Secondary indexes and foreign keys</h3>



<p class="wp-block-paragraph">Every secondary index present during full load is an index being maintained row by row while you are trying to move data as fast as possible. Foreign keys are worse, because DMS does not guarantee parent-before-child load order across tables.</p>



<p class="wp-block-paragraph">Standard sequence: drop or disable secondary indexes and foreign key constraints before full load, recreate them afterwards, then let CDC catch up. Keep primary keys, since CDC needs them. Script the recreation and check it into version control alongside everything else, because rebuilding forty indexes from memory at 2am is not a plan.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Validation you can actually trust</h2>



<p class="wp-block-paragraph">DMS data validation compares source and target row by row and reports mismatches. It supports Oracle and PostgreSQL-compatible endpoints in both directions, so an Oracle to Aurora PostgreSQL path is covered. Turn it on.</p>



<p class="wp-block-paragraph">Then understand what it will not tell you.</p>



<ul class="wp-block-list">
<li>It needs a primary key or unique index on both sides. Unkeyed tables are not validated, and those are exactly the tables most likely to have gone wrong.</li>



<li>Collation and sort order differ between Oracle and PostgreSQL. When they disagree, validation reports failures on records that are actually fine, and the noise trains people to ignore the report.</li>



<li>It does not run when the migration uses custom filtering, or when you are consolidating several source databases into one target.</li>



<li>It costs real resources. Validation issues its own queries against both databases, on top of the migration traffic. Budget for that on the source.</li>



<li>For LOB columns it compares checksums rather than values, using DBMS_CRYPTO on the Oracle side.</li>
</ul>



<p class="wp-block-paragraph">Because validation runs alongside migration, a task that stops takes validation with it. Validation-only tasks solve that: same endpoints, same table mappings, no data movement, running independently of the migration task. Use one when the validation load is hurting the source and you want to run it off-peak, or when you want validation to survive a migration task restart.</p>



<p class="wp-block-paragraph">None of that replaces your own reconciliation. Before cutover, run business-level checks on both sides: rows per table, sums of every monetary column, min and max of every date column, distinct counts on anything you join on. Those catch things row comparison does not, because they answer the question the business actually asks.</p>



<p class="wp-block-paragraph">And specifically for the LOB problem, compare maximum lengths, not just row counts.</p>



<pre class="wp-block-code"><code>-- On Aurora PostgreSQL, after the load.
-- Compare this against dbms_lob.getlength on the Oracle source.
SELECT count(*)                 AS row_count,
       max(length(description)) AS max_len,
       count(*) FILTER (WHERE description IS NULL) AS null_count
FROM   app.invoices;</code></pre>



<p class="wp-block-paragraph">If the target maximum is suspiciously round, exactly 16384 or exactly 32768, you have found your truncation.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting an Oracle to Aurora migration that stalls</h2>



<h3 class="wp-block-heading">Full load runs, then one table sits at zero</h3>



<p class="wp-block-paragraph">Almost always LOBs. Full LOB mode moves large objects one at a time with a source lookup per value, and on a table with a million large CLOBs that is a very long time with no visible progress. Check the table statistics for that table, then check whether it has LOB columns. Move it to its own task with limited or inline mode and an appropriate size.</p>



<h3 class="wp-block-heading">CDC latency climbs and never comes down</h3>



<p class="wp-block-paragraph">Look at which latency is climbing. Source latency means redo is arriving faster than DMS can read it, which is the LogMiner to Binary Reader conversation, or a redo volume problem on the source. Target latency means Aurora cannot apply fast enough, which points at indexes still present, triggers still enabled, or an undersized writer instance.</p>



<h3 class="wp-block-heading">Task fails on restart with missing archived logs</h3>



<p class="wp-block-paragraph">Retention. The logs DMS needed were removed before it came back. Increase archived log retention on the source to comfortably exceed your worst realistic outage, then restart from a fresh full load for the affected tables. There is no recovering the gap.</p>



<h3 class="wp-block-heading">Updates on the source do not appear on the target</h3>



<p class="wp-block-paragraph">Supplemental logging, nine times out of ten. Check whether the affected table has a primary key or unique index, and whether logging is configured at the right level for that answer. The premigration assessments cover both cases.</p>



<h3 class="wp-block-heading">Errors about a unique index on the target</h3>



<p class="wp-block-paragraph">DMS does not support replication to a table with a unique index built on a <code>coalesce</code> expression. If schema conversion produced one, or someone added it by hand to work around the empty-string problem, replication to that table will not work. Restructure the constraint.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Accepting the default limited LOB size without measuring the source. This is the truncation bug, and it has a specific, preventable cause.</li>



<li>Treating a green task and a passed validation as proof of correctness. Both are true when data has been truncated.</li>



<li>Skipping the premigration assessments because the wizard let you continue without them.</li>



<li>Forgetting to advance sequences, then discovering it under production write load.</li>



<li>Leaving triggers enabled on the target during full load and polluting audit tables.</li>



<li>Assuming schema conversion handled empty strings. It converted the syntax, not the semantics.</li>



<li>Setting archived log retention to the length of a normal outage rather than a bad one.</li>



<li>Doing the first end-to-end rehearsal on cutover night.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ol class="wp-block-list">
<li><strong>Run the assessment report first and size the manual work from it.</strong> The red items are your timeline, and they are better known in week one than week nine.</li>



<li><strong>Inventory LOB columns before configuring anything.</strong> Maximum size per column, in KB, written down.</li>



<li><strong>Separate the LOB-heavy tables into their own task.</strong> Different tables want different modes, and one global setting will always be wrong for someone.</li>



<li><strong>Enable supplemental logging deliberately, per table class.</strong> Key-level for keyed tables, all-columns for unkeyed ones.</li>



<li><strong>Drop secondary indexes and foreign keys for the load.</strong> Script the recreation, version it, test it.</li>



<li><strong>Rehearse the whole cutover at least twice</strong>, including the sequence advance and the trigger re-enable. A scratch environment for the dry run does not have to be expensive: a temporary Aurora cluster you tear down afterwards, or a cheap dedicated box from somewhere like InterServer or Hetzner running a trimmed source copy, both work.</li>



<li><strong>Run your own reconciliation alongside DMS validation.</strong> Counts, sums, min and max dates, maximum LOB lengths.</li>



<li><strong>Keep the Oracle source running and readable for a while after cutover.</strong> The bugs that surface at week three are the ones you can only diagnose by comparing against the original.</li>
</ol>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Should I migrate Oracle to Aurora PostgreSQL or Aurora MySQL?</h3>



<p class="wp-block-paragraph">Aurora PostgreSQL for anything with meaningful PL/SQL, because PL/pgSQL is close enough that automated conversion does most of the work, and the orafce extension covers a chunk of the rest. Aurora MySQL is defensible when the schema is plain tables, the logic lives in the application, and your team already operates MySQL. The deciding question is how many lines of procedural code you would have to rewrite by hand.</p>



<h3 class="wp-block-heading">Can AWS DMS migrate stored procedures and packages?</h3>



<p class="wp-block-paragraph">No. DMS moves data. Schema and code conversion is a separate job, handled by DMS Schema Conversion in the console or the AWS Schema Conversion Tool on the desktop. Both convert most objects automatically and produce an action-item list for the rest.</p>



<h3 class="wp-block-heading">Why is my DMS task green when data is missing?</h3>



<p class="wp-block-paragraph">Because truncation in limited LOB mode is expected behaviour, not an error. DMS writes a warning to the task log and carries on. Task status reflects whether the task is running, not whether the data is faithful. Compare maximum LOB lengths between source and target to catch it.</p>



<h3 class="wp-block-heading">Do I need downtime for an Oracle to Aurora migration?</h3>



<p class="wp-block-paragraph">Some, but it can be short. Full load plus CDC means the target stays current while you test, so the outage is limited to stopping writes, letting CDC drain, advancing sequences, re-enabling triggers, recreating indexes and repointing the application. Rehearse it and that usually fits a maintenance window rather than a weekend.</p>



<h3 class="wp-block-heading">Does DMS migrate sequences?</h3>



<p class="wp-block-paragraph">It copies row data, not sequence state. Schema conversion creates the sequence objects; you advance them yourself after stopping replication, using <code>setval</code> against the current maximum key value. Skipping this produces primary key collisions on the first write after cutover.</p>



<h3 class="wp-block-heading">LogMiner or Binary Reader for Oracle CDC?</h3>



<p class="wp-block-paragraph">LogMiner unless you have a reason. It is the default, needs fewer privileges, and handles ASM, encryption and compression without extra work. Move to Binary Reader when redo volume is high enough that LogMiner cannot keep up, when you are running several tasks against the same source, or when the CPU cost of parsing on the source database is unacceptable.</p>



<h3 class="wp-block-heading">Will DMS data validation catch every problem?</h3>



<p class="wp-block-paragraph">No. It needs a primary key or unique index on both sides, it can report false failures where Oracle and PostgreSQL collations disagree, and it does not run with custom filtering or multi-source consolidation. Treat it as one layer and add business-level reconciliation on top.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">An Oracle to Aurora migration does not usually announce its failures. The task goes green, validation passes, and the damage shows up weeks later as a truncated field, a duplicate key, a report whose totals moved slightly.</p>



<p class="wp-block-paragraph">So build the migration around proving correctness rather than proving completion. Measure your LOBs before you set a limit. Configure supplemental logging for the tables that have no key, not just the ones that do. Assume the semantic gaps between Oracle and PostgreSQL are still there after conversion, because they are. Advance the sequences. Then reconcile at the level the business cares about, not just row by row.</p>



<p class="wp-block-paragraph">Green means the pipe is open. It does not mean what came out the other end is what went in.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your Oracle to Aurora migration?</h2>



<p class="wp-block-paragraph">Most of the work on these projects is not running the wizard. It is finding the handful of things that will be wrong afterwards. That is the part I help with:</p>



<ul class="wp-block-list">
<li>Reviewing DMS task settings and table mappings for LOB truncation risk, per column, before you run a full load you cannot easily repeat.</li>



<li>Working through the schema conversion assessment report and turning the red items into an actual estimate rather than a wall of warnings.</li>



<li>Setting up supplemental logging and CDC correctly for tables with no primary key, including the LogMiner versus Binary Reader call for your redo volume.</li>



<li>Writing the cutover runbook: index recreation scripts, sequence advance, trigger re-enable, ordered, tested, and timed against a real rehearsal.</li>



<li>Building the reconciliation layer that sits on top of DMS validation, so you can sign off on the data with something better than a green tick.</li>



<li>Post-cutover triage when something on Aurora behaves differently from Oracle and nobody can work out which layer changed.</li>
</ul>



<p class="wp-block-paragraph">If you have a task settings JSON, an assessment report, or a validation failure log you cannot make sense of, send it over and I will tell you what I see in it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/oracle-to-aurora-migration/">Oracle to Aurora Migration: The Failures Nobody Catches Until Cutover</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Data Lake vs Data Warehouse for CRM Analytics: Volume Is the Wrong Question</title>
		<link>https://john-nessime.com/blog/devops/data-lake-vs-data-warehouse-crm-analytics/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 06 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Business Intelligence]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[CRM Analytics]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Warehouse]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[SQL]]></category>
		<category><![CDATA[Zero-ETL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=153</guid>

					<description><![CDATA[<p>Everyone argues this one on data volume, and volume is the argument that matters least: CRM data is small enough that both architectures handle it comfortably. What actually decides data lake vs data warehouse for CRM analytics is how much point-in-time history you need, how fast the schema churns, what shape your queries are, and who is going to maintain the thing. Includes a decision procedure you can run in an afternoon.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/data-lake-vs-data-warehouse-crm-analytics/">Data Lake vs Data Warehouse for CRM Analytics: Volume Is the Wrong Question</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question never arrives as an architecture question. It arrives as someone in RevOps asking whether they can get pipeline coverage by rep, by week, for the last two years, including the deals that were deleted along the way.</p>



<p class="wp-block-paragraph">You look at the CRM reporting tab. It cannot do that. So now you are picking a platform, and within about ten minutes someone will say &#8220;we should build a data lake,&#8221; and someone else will say &#8220;just put it in the warehouse,&#8221; and the deciding argument will be about how much data there is.</p>



<p class="wp-block-paragraph">That argument is almost always the wrong one. This post covers data lake vs data warehouse specifically for CRM analytics: why volume rarely decides it, what actually does, a decision procedure you can run in an afternoon, and where zero-ETL has changed the maths recently enough that older advice is stale.</p>



<h2 class="wp-block-heading">Start with the number, then stop using it</h2>



<p class="wp-block-paragraph">Go and count your rows. Not estimate. Count.</p>



<p class="wp-block-paragraph">A mid-sized company&#8217;s entire Salesforce org is usually a few million rows across the objects anyone actually reports on: accounts, contacts, opportunities, activities, cases. Add HubSpot, Zendesk and a billing system and you are still typically in the low tens of gigabytes once it is written as compressed Parquet. Activity and event history is the one object that can genuinely run large, and it is worth checking separately.</p>



<p class="wp-block-paragraph">Numbers at that scale do not stress either architecture. A data warehouse will not break a sweat. A lake will not break a sweat. The lake-versus-warehouse debate as it is usually framed, the one about petabytes and elasticity and separating storage from compute, is a debate about a scale problem you do not have.</p>



<p class="wp-block-paragraph">So count the rows, confirm you are small, and then stop using volume as an argument. Everything below is about what decides it instead.</p>



<h2 class="wp-block-heading">Data lake vs data warehouse: the three options, honestly</h2>



<h3 class="wp-block-heading">Warehouse only</h3>



<p class="wp-block-paragraph">Load CRM objects straight into Redshift, Snowflake, BigQuery or Postgres. Model with SQL. Point the BI tool at it.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> it is the fastest path to an analyst being productive, and that is not a small thing. Query performance on joins is better than a lake gives you without tuning. Access control is mature and row-level. Every BI tool connects natively. The people who will maintain this already know SQL and probably do not know Spark.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> semi-structured data is awkward, and CRM payloads are full of it. Custom field bags, JSON blobs, email bodies, call transcripts. You can store JSON in a modern warehouse but you will not enjoy querying it at any depth. Storing many years of daily snapshots gets expensive on warehouse storage in a way it does not on object storage. And you are more locked in: moving between warehouses is a migration project, not a repoint.</p>



<h3 class="wp-block-heading">Lake only</h3>



<p class="wp-block-paragraph">Land raw API responses to S3, catalogue them, query with Athena or Spark. Iceberg or Delta as the table format if you want anything resembling a database.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> raw retention is nearly free, so keeping every version of every record for years is a storage line item rather than a decision. Schema evolution is genuinely easier: when a SaaS admin adds a field, the lake absorbs it. Unstructured content lives alongside structured tables, which matters more now that people want to run models over call notes and support tickets. And an open table format means the data is not hostage to one query engine.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> you have taken on work a warehouse does for you. Small-file compaction, partitioning, table maintenance, catalogue hygiene. Query latency on interactive dashboards is worse unless you invest in layout. Governance is a build, not a setting. And the per-query cost model punishes exactly the workload CRM analytics generates: lots of small, repeated, unpredictable queries from a BI tool. Athena bills on data scanned with a per-query minimum, which means a dashboard refreshing thirty tiles is thirty billable scans whether or not anyone is looking.</p>



<h3 class="wp-block-heading">Lakehouse</h3>



<p class="wp-block-paragraph">Raw and historical data in object storage as Iceberg tables; curated marts in the warehouse or queried in place by a warehouse engine. This is where most serious setups end up, and it is what AWS, Databricks and Snowflake have all converged on.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> you get cheap deep history and expensive fast queries in the right places. Raw payloads stay replayable when a vendor changes a field type. Analysts still get warehouse-shaped tables.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> it is two systems. Two cost models, two sets of permissions, two places a table can be defined and drift. For a five-person company with one CRM, this is architecture as procrastination. The complexity is real and it should be earning something.</p>



<h2 class="wp-block-heading">What actually decides it</h2>



<p class="wp-block-paragraph">Four levers, in the order they usually matter for CRM data.</p>



<h3 class="wp-block-heading">1. How much history you need, and whether deletes matter</h3>



<p class="wp-block-paragraph">This is the big one and it is the one people skip.</p>



<p class="wp-block-paragraph">CRM systems mutate in place. An opportunity&#8217;s stage, amount and close date are overwritten as the deal moves. If you only ever sync current state, you can report on the pipeline as it is today and you cannot report on what it looked like last March. Every question about pipeline movement, forecast accuracy, stage velocity or &#8220;what changed since the QBR&#8221; needs point-in-time history, and the CRM will not give it to you retroactively. Deleted records are worse: they are simply gone.</p>



<p class="wp-block-paragraph">If you need history, you need somewhere cheap to keep daily snapshots or a change feed. That pushes toward object storage, or toward a warehouse with a genuinely cheap storage tier. If a rolling twelve months of current-state data covers every question anyone asks, the warehouse handles it alone.</p>



<h3 class="wp-block-heading">2. How fast the schema churns</h3>



<p class="wp-block-paragraph">Count the custom fields added to your main CRM object in the last six months. If it is two, schema churn is not your problem. If it is forty, and half of them are from a business unit you do not talk to, a rigid warehouse schema becomes a standing maintenance tax and the lake&#8217;s tolerance for drift starts paying for itself.</p>



<h3 class="wp-block-heading">3. The shape of the queries</h3>



<p class="wp-block-paragraph">Scheduled dashboards hitting the same tables repeatedly favour a warehouse, because per-second compute amortises across many queries and results are cached. Redshift Serverless, for example, bills in processing-unit-hours per second with a short minimum and shuts down when idle, which suits bursty BI traffic.</p>



<p class="wp-block-paragraph">Exploratory one-off analysis across wide, rarely-touched history favours a lake, because you only pay when someone asks. Most CRM analytics is overwhelmingly the first kind. That fact alone settles a lot of these arguments.</p>



<h3 class="wp-block-heading">4. Who is going to maintain it</h3>



<p class="wp-block-paragraph">The least technical criterion and often the most decisive. If the answer is &#8220;a RevOps analyst who is very good at SQL and has no platform team behind them,&#8221; build the warehouse. A lake handed to someone who cannot maintain it becomes an unqueryable pile of Parquet in about eight months. That is not a hypothetical failure mode; it is the normal one.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">A decision procedure you can run this afternoon</h2>



<ol class="wp-block-list">
<li><strong>Count rows per object</strong> for everything anyone reports on. Note which objects are over about ten million rows. Usually it is activities and nothing else.</li>

<li><strong>Write down the ten questions</strong> the business actually asks. Not the ones you imagine. Mark each one as needing current state or point-in-time history.</li>

<li><strong>Count schema changes</strong> on your main objects over the last six months.</li>

<li><strong>Estimate query volume:</strong> number of dashboards, tiles per dashboard, refresh frequency, plus rough ad-hoc queries per week. This is the number that drives cost, not storage.</li>

<li><strong>Model both costs against those numbers.</strong> For the lake: bytes scanned per query times queries per month. For the warehouse: compute-hours per month plus storage. Do it on a napkin. At CRM scale the answer is usually obvious and usually favours the warehouse.</li>

<li><strong>Name the maintainer.</strong> If you cannot name a person, pick the option that needs less of them.</li>
</ol>



<p class="wp-block-paragraph">Read together, most CRM-only shops land on: warehouse for the curated layer, cheap object storage for raw history, and no ambition beyond that until something forces it. That is a lakehouse, but the small honest kind, and it is a very different commitment from standing up a full lake platform.</p>



<h2 class="wp-block-heading">Where zero-ETL changed the maths</h2>



<p class="wp-block-paragraph">A lot of writing on this topic predates a genuine shift, so it is worth stating plainly.</p>



<p class="wp-block-paragraph">Managed zero-ETL integrations now replicate CRM and support platforms into AWS targets without you building a pipeline. AWS Glue supports SaaS sources including Salesforce, SAP, ServiceNow, Zendesk and Zoho CRM, landing into Amazon Redshift, the SageMaker lakehouse, or S3 Tables with Iceberg underneath. Salesforce has pushed the other direction with zero-copy federation, letting Data Cloud read Iceberg tables in Databricks, Snowflake, BigQuery or Redshift in place, and letting those platforms read Data Cloud objects back.</p>



<p class="wp-block-paragraph">Two consequences for this decision:</p>



<ul class="wp-block-list">
<li><strong>Ingestion is no longer the differentiator.</strong> &#8220;The lake is easier to load into&#8221; was a real argument when you were hand-rolling connectors. It is much weaker when the same managed integration targets either destination.</li>

<li><strong>Iceberg makes the choice less permanent.</strong> If your raw layer is Iceberg tables in object storage, adding a warehouse engine later is a configuration change rather than a migration. That is a genuine reason to prefer open table formats even if you never build a &#8220;lake&#8221; as such.</li>
</ul>



<p class="wp-block-paragraph">Two honest caveats. Zero-ETL integrations cover a curated list of sources and regions, so check yours before designing around them. And a managed replica is still current-state replication plus change capture, not a modelled history; you still have to decide how snapshots and slowly changing dimensions get built.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<ul class="wp-block-list">
<li><strong>&#8220;We need a lake because we&#8217;ll do AI later.&#8221;</strong> Reasonable in spirit, wrong in sequence. Build the thing that answers this quarter&#8217;s questions, keep raw payloads in object storage so you have not destroyed your options, and revisit. Raw retention is the cheap insurance; a lake platform is not.</li>

<li><strong>&#8220;Storage is cheaper in a lake.&#8221;</strong> True and almost always irrelevant. At CRM volumes, storage is a rounding error next to compute and, more to the point, next to the salary of whoever maintains the thing.</li>

<li><strong>&#8220;The warehouse is a single point of vendor lock-in.&#8221;</strong> Fair, and it is why open table formats matter. But if you replicate raw data to object storage, the lock-in is on your transformation logic, not your data, and that is portable if you keep it in SQL and version control.</li>

<li><strong>&#8220;We&#8217;ll just query the CRM API directly and skip both.&#8221;</strong> Works until a dashboard fans out into rate limits and the whole thing becomes a data pipeline you did not design. If more than one person is asking questions, you need a copy.</li>

<li><strong>&#8220;Let&#8217;s do both properly from day one.&#8221;</strong> This is the expensive mistake. Two systems, two cost models, and tables that quietly disagree, in service of a workload that fits comfortably in one.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is a data lake overkill for one CRM?</h3>



<p class="wp-block-paragraph">For a single CRM with current-state reporting needs, usually yes. The point at which it stops being overkill is when you need years of point-in-time history, when unstructured content like call transcripts becomes part of the analysis, or when you are integrating enough systems that a common raw layer is cheaper than a dozen warehouse loaders.</p>



<h3 class="wp-block-heading">How do I get historical CRM snapshots if I only have current state?</h3>



<p class="wp-block-paragraph">You mostly cannot get it retroactively, which is why this decision is urgent rather than academic. Start capturing now: either daily full snapshots to partitioned object storage, or a change feed if the platform offers one, then build slowly changing dimension tables from that. Field history tracking inside the CRM covers a limited number of fields and is not a substitute.</p>



<h3 class="wp-block-heading">Does the lakehouse make this comparison obsolete?</h3>



<p class="wp-block-paragraph">It blurs the storage layer, not the operational reality. You still choose a query engine, a cost model, a governance model and a maintainer. The useful question has shifted from &#8220;lake or warehouse&#8221; to &#8220;how much of this do I want to operate myself,&#8221; which is a better question anyway.</p>



<h3 class="wp-block-heading">Where does reverse ETL fit?</h3>



<p class="wp-block-paragraph">If you plan to push computed values back into the CRM, such as a lead score or account health field, that pushes toward the warehouse. Reverse ETL tools connect to warehouses far more readily than to query engines over object storage, and the write path wants a stable, modelled table rather than a raw one.</p>



<h3 class="wp-block-heading">Can I start with one and move later?</h3>



<p class="wp-block-paragraph">Moving from lake to warehouse is straightforward: you are loading modelled tables into a new engine. Moving from warehouse to lake is harder, because the history only exists inside the warehouse and the transformations are written in its dialect. The asymmetric fix is to land raw data in object storage regardless of which you query from. That single habit preserves the option cheaply.</p>



<h3 class="wp-block-heading">What about just using the CRM&#8217;s own analytics product?</h3>



<p class="wp-block-paragraph">Worth pricing seriously before building anything. It is genuinely the right answer when all your questions are about one system, and it removes an entire pipeline from your life. It stops being the right answer the moment a question spans CRM plus billing plus product usage, because you will be paying to ingest the other systems into a platform priced for CRM data.</p>



<h2 class="wp-block-heading">The short version</h2>



<p class="wp-block-paragraph">In the data lake vs data warehouse question for CRM analytics, volume is the argument everyone reaches for and it is the one that matters least. Your CRM data is small. Both options handle it.</p>



<p class="wp-block-paragraph">What decides it is how much history you need, how fast the schema moves, what shape the queries are, and who is going to keep it running. Answer those four and the architecture falls out on its own. And whatever you pick, land the raw payloads in object storage from day one, because that is the cheap decision that keeps every other decision reversible.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second opinion on your CRM analytics stack?</h2>



<p class="wp-block-paragraph">I help teams make this call without over-building, and unpick it when it has already been over-built. Things I can help with:</p>



<ul class="wp-block-list">
<li>Running the sizing and cost model above against your real query volume, not a vendor calculator.</li>

<li>Designing point-in-time history capture for CRM objects, including deletes and slowly changing dimensions.</li>

<li>Setting up zero-ETL or connector-based ingestion from Salesforce, HubSpot, Zendesk or Zoho into Redshift, Snowflake or an Iceberg lake.</li>

<li>Migrating an unmaintainable pile of Parquet into something an analyst can actually query.</li>

<li>Cutting query costs on an existing setup: partitioning, table layout, dashboard refresh patterns.</li>

<li>Building the curated CRM models that BI tools sit on, in SQL and in version control.</li>
</ul>



<p class="wp-block-paragraph">If you are mid-argument about this, send me your row counts and the ten questions the business wants answered. That is usually enough to settle it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/data-lake-vs-data-warehouse-crm-analytics/">Data Lake vs Data Warehouse for CRM Analytics: Volume Is the Wrong Question</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>AWS Glue Data Quality for SaaS Data: Catching the Breakage Nobody Deployed</title>
		<link>https://john-nessime.com/blog/devops/aws-glue-data-quality-saas-data/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 05 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Anomaly Detection]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Observability]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[DQDL]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[SQL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=148</guid>

					<description><![CDATA[<p>A SaaS admin changes a field and your pipeline stays green while the numbers drift. A practical guide to AWS Glue Data Quality for SaaS sources: where to run the checks, why nested payloads need flattening before DQDL can see them, which rule catches which failure, and the dynamic rules that pass silently because they have no history yet.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-glue-data-quality-saas-data/">AWS Glue Data Quality for SaaS Data: Catching the Breakage Nobody Deployed</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Nobody deployed anything. That is the part that makes it hard to explain.</p>



<p class="wp-block-paragraph">A Salesforce admin renames a custom field on Tuesday afternoon. Your connector keeps running, your Glue job keeps succeeding, the crawler happily adds a new column and quietly stops populating the old one. Three weeks later someone asks why the pipeline report has a cliff in it, and you spend an afternoon reconstructing a change nobody in engineering made and nobody logged.</p>



<p class="wp-block-paragraph">This post is about using AWS Glue Data Quality to catch that class of breakage. It is aimed at people pulling from SaaS sources into S3 or a lakehouse: Salesforce, HubSpot, Zendesk, Stripe, Shopify, Zoho, an internal partner API. I will cover where to run the checks, the flattening constraint that will bite you before anything else does, which DQDL rule actually catches which SaaS failure, when dynamic rules and anomaly detection quietly do nothing, and what should happen when a rule fails.</p>



<h2 class="wp-block-heading">Why SaaS data breaks in ways your own database doesn&#8217;t</h2>



<p class="wp-block-paragraph">When you own the source database, the schema is under change control: someone writes a migration, it gets reviewed, and if it breaks a consumer there is a commit to point at. A SaaS source has none of that. Three consequences:</p>



<ul class="wp-block-list">
<li><strong>The schema is edited by people who are not engineers.</strong> A sales ops admin adding a picklist value is a production schema change with no review, no notice and no version number.</li>

<li><strong>The payload is nested.</strong> SaaS APIs return structs and arrays: line items, related objects, custom field bags. That shape matters more than you would expect, for reasons in the next section but one.</li>

<li><strong>Absence is ambiguous.</strong> A field missing from a payload might mean &#8220;not set,&#8221; &#8220;not permitted for this user,&#8221; or &#8220;we removed it.&#8221; The connector usually cannot tell you which, and neither can your table.</li>
</ul>



<p class="wp-block-paragraph">Glue Data Quality is a rules engine built on Deequ that evaluates declarative checks written in DQDL, the Data Quality Definition Language, against a dataset. It is well suited to the first and third problems. The second one needs work before it is useful at all.</p>



<h2 class="wp-block-heading">Decide where the checks run before you write a rule</h2>



<p class="wp-block-paragraph">There are two places to evaluate a ruleset, and they answer different questions.</p>



<h3 class="wp-block-heading">Inside the Glue ETL job</h3>



<p class="wp-block-paragraph">The <code>EvaluateDataQuality</code> transform runs mid-pipeline against a DynamicFrame. It answers &#8220;is this batch safe to load?&#8221; and it can act on the answer: stop the job, or split failing rows into a quarantine prefix. This is also the only placement where row-level results exist, which matters because &#8220;which 400 rows are bad&#8221; is a far more useful answer than &#8220;the rule failed.&#8221;</p>



<h3 class="wp-block-heading">Against a Data Catalog table</h3>



<p class="wp-block-paragraph">You can attach a ruleset to a catalog table and run it on a schedule, independent of any job. This answers &#8220;is the table currently trustworthy?&#8221; and surfaces a quality score analysts can see before they query. The trade-off is that it is after the fact: by the time a catalog evaluation fails, the bad batch is already queryable. Analyzers are also not supported for catalog evaluations, so some dynamic-rule tooling behaves differently there.</p>



<p class="wp-block-paragraph">In practice I run both. The in-job ruleset is small and strict, and exists to stop garbage landing. The catalog ruleset is broader and looser, and exists so a table&#8217;s state is visible to people who will never open Glue Studio.</p>



<h2 class="wp-block-heading">The flattening problem that comes first</h2>



<p class="wp-block-paragraph">Here is the constraint that catches most people on SaaS data, stated plainly in the AWS documentation and easy to skim past: <strong>DQDL cannot evaluate nested or list-type columns.</strong> No structs, no arrays.</p>



<p class="wp-block-paragraph">Almost every interesting SaaS payload is nested. A Salesforce opportunity carries a related account struct, a Stripe invoice carries an array of line items, a Zendesk ticket carries a custom fields array of key-value pairs. Point a ruleset at the raw landed data and the fields you most want to check are the ones the engine cannot see.</p>



<p class="wp-block-paragraph">So you flatten first, with Glue&#8217;s relationalize or an explicit projection, then evaluate. That works, but it changes what your rules mean:</p>



<ul class="wp-block-list">
<li><strong>Your rules now describe the projection, not the payload.</strong> If your flattening step drops a field, no rule will notice, because from DQDL&#8217;s point of view that field was never there.</li>

<li><strong>Exploding an array changes the grain.</strong> One invoice with five line items becomes five rows. Every <code>RowCount</code> rule you write against that table is now counting line items, and a customer who starts itemising differently will move your row count without any data being wrong.</li>

<li><strong>Custom-field bags need pivoting, not flattening.</strong> A key-value array flattened naively gives you rows, not columns, and you cannot write a <code>ColumnValues</code> rule against a value whose meaning depends on a sibling key column. Pivot the ones you care about into real columns and check those.</li>
</ul>



<p class="wp-block-paragraph">The practical rule: put your quality checks immediately after the flattening step, and treat the flattening step itself as code that needs its own test. Glue Data Quality will not cover it for you.</p>



<h2 class="wp-block-heading">Matching SaaS breakage to the rule that catches it</h2>



<p class="wp-block-paragraph">DQDL has a long rule list and most guides just enumerate it. More useful is the other direction: what actually goes wrong with SaaS sources, and what catches each one.</p>



<h3 class="wp-block-heading">A field is renamed, removed, or added</h3>



<p class="wp-block-paragraph">This is the Tuesday-afternoon scenario. Two rules, doing two different jobs:</p>



<pre class="wp-block-code"><code>Rules = [
    ColumnExists "annual_revenue__c",
    ColumnCount = 47
]</code></pre>



<p class="wp-block-paragraph"><code>ColumnExists</code> protects the specific fields your downstream models depend on. <code>ColumnCount</code> protects against everything else: a field silently added, or one you forgot you needed. The first is precise and the second is a tripwire, and you want both because a rename shows up as one column gone and one column arrived, which leaves the count unchanged.</p>



<p class="wp-block-paragraph"><code>SchemaMatch</code> compares against a reference dataset&#8217;s schema, which is stricter and useful if you maintain a golden reference table. Bear in mind a hard column count on a SaaS object fails every time an admin adds a field, so decide whether that blocks or just warns.</p>



<h3 class="wp-block-heading">A field still exists but stopped being populated</h3>



<p class="wp-block-paragraph">Permissions change, a workflow gets disabled, an integration user loses field-level access. The column is still in the schema and is now all nulls.</p>



<pre class="wp-block-code"><code>Rules = [
    IsComplete "opportunity_id",
    Completeness "lead_source" &gt; 0.85,
    Completeness "email" &gt; 0.60 where "record_type = 'Contact'"
]</code></pre>



<p class="wp-block-paragraph"><code>IsComplete</code> is the absolute version: no nulls at all, right for keys. <code>Completeness</code> takes a threshold, which is what you want for optional business fields where a hard rule would fail constantly. The <code>where</code> clause is the one people miss: SaaS objects are usually polymorphic, and a completeness threshold that makes sense for contacts is nonsense across the whole table. Scope the rule to the record type.</p>



<h3 class="wp-block-heading">The sync ran short</h3>



<p class="wp-block-paragraph">A paginated extract stops early, a rate limit gets absorbed by retries, an incremental cursor drifts. The job succeeds and delivers less data than it should have. A static <code>RowCount &gt; 1000</code> stops catching this the moment you cross a thousand rows. Dynamic rules compare against your own history instead:</p>



<pre class="wp-block-code"><code>Rules = [
    RowCount &gt; min(last(3)),
    Sum "amount" &gt; avg(last(5)) * 0.7,
    DetectAnomalies "RowCount"
]

Analyzers = [
    RowCount,
    DistinctValuesCount "owner_id"
]</code></pre>



<p class="wp-block-paragraph"><code>last(k)</code> pulls the last k recorded values of that metric for the same dataset; with k greater than one you need an aggregation like <code>avg</code>, <code>min</code> or <code>max</code> to reduce them to one number. <code>DetectAnomalies</code> goes further, using forecasting over stored history to flag deviations with no threshold from you at all. That is the right tool for anything seasonal: a B2B pipeline that dips every weekend trips a naive trailing-average rule every Monday, while anomaly detection learns the shape.</p>



<p class="wp-block-paragraph">Analyzers deserve their own mention: they gather a metric without asserting anything, which builds the history dynamic rules and anomaly detection need. <code>DistinctValuesCount</code> on an owner or account column is a strong early signal for a partial sync, because row count can look fine while the number of distinct accounts represented quietly halves.</p>



<h3 class="wp-block-heading">Records arrive twice, or the key stops being a key</h3>



<p class="wp-block-paragraph">Replays, overlapping incremental windows and connector retries all produce duplicates, and SaaS &#8220;IDs&#8221; are less unique than vendor docs imply once you have merged sandboxes or multiple accounts.</p>



<pre class="wp-block-code"><code>Rules = [
    IsPrimaryKey "id",
    Uniqueness "external_reference" &gt; 0.99
]</code></pre>



<p class="wp-block-paragraph"><code>IsPrimaryKey</code> asserts unique and non-null together. <code>Uniqueness</code> with a threshold is the softer version for fields that are mostly-but-not-perfectly unique, which describes a lot of SaaS reference fields honestly.</p>



<h3 class="wp-block-heading">Objects stop agreeing with each other</h3>



<p class="wp-block-paragraph">Multi-object extracts rot quietly. Opportunities reference accounts that were never synced; invoice totals stop matching line item sums because one object&#8217;s extract failed and the other&#8217;s didn&#8217;t.</p>



<p class="wp-block-paragraph"><code>ReferentialIntegrity</code> checks that values in one dataset&#8217;s column exist in another&#8217;s. <code>RowCountMatch</code> and <code>AggregateMatch</code> compare counts and summary metrics across two datasets. These catch a broken join before an analyst does, and they need the multi-frame form of the transform with the second dataset supplied as an additional source.</p>



<h3 class="wp-block-heading">The sync stopped and nobody noticed</h3>



<p class="wp-block-paragraph">The quietest failure of all: nothing is wrong with the data, there just isn&#8217;t any new data. <code>DataFreshness</code> checks a timestamp column against a window. It is deliberately simple and its comparison granularity is limited, so for anything more nuanced the custom SQL rule is the escape hatch, running a query against the dataset and evaluating the result against an expression. DQDL is case sensitive and that rule type&#8217;s exact capitalisation has appeared more than one way across the docs and the visual builder, so copy it from the current DQDL reference rather than from memory.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Static, dynamic and ML: when each one lies to you</h2>



<p class="wp-block-paragraph">All three check styles have a mode where they pass while the data is wrong. Knowing which is which is most of the skill.</p>



<ul class="wp-block-list">
<li><strong>Static rules lie by going stale.</strong> A threshold that was correct when you wrote it stops being correct as the business grows, and a rule that can no longer fail is worse than no rule because it looks like coverage.</li>

<li><strong>Dynamic rules lie when they have no history.</strong> This one is genuinely nasty. <code>last(k)</code> reads previously recorded values for that metric, and if there aren&#8217;t any, it falls back to a default of zero. Add <code>RowCount &gt; avg(last(3))</code> to an existing job today and it will pass trivially for the first few runs, because almost any row count beats zero. It looks like a working check from the moment you deploy it. It isn&#8217;t one until it has accumulated runs.</li>

<li><strong>Anomaly detection lies while it is warming up, and after you retrain it badly.</strong> It needs a run of history before its forecasts mean anything, and if you leave a period of genuinely bad data in the training window, it learns that the bad shape is normal. You can exclude specific statistics from the calculation in the Data Quality tab, and you should, after any incident.</li>
</ul>



<p class="wp-block-paragraph">So when you add a dynamic or ML-based rule, write down the date and treat it as unarmed for the first couple of weeks. Keep a crude static rule alongside it in the meantime.</p>



<h2 class="wp-block-heading">What should happen when a rule fails</h2>



<p class="wp-block-paragraph">The transform returns a collection. The two members you care about are the row-level outcomes, which give you the original data with an evaluation result column appended per row, and the rule outcomes, which give you one row per rule with its verdict.</p>



<pre class="wp-block-code"><code>from awsgluedq.transforms import EvaluateDataQuality

dq_results = EvaluateDataQuality().process_rows(
    frame=flattened_opportunities,
    ruleset=ruleset,
    publishing_options={
        "dataQualityEvaluationContext": "salesforce_opportunity",
        "enableDataQualityCloudWatchMetrics": "true",
        "enableDataQualityResultsPublishing": "true",
    },
)

row_level = SelectFromCollection.apply(
    dfc=dq_results, key="rowLevelOutcomes"
)</code></pre>



<p class="wp-block-paragraph"><code>dataQualityEvaluationContext</code> is the namespace your CloudWatch metrics land under. Name it after the source object rather than the job, because you will eventually want to alert per object.</p>



<p class="wp-block-paragraph">From there, three response patterns:</p>



<ol class="wp-block-list">
<li><strong>Fail fast.</strong> Stop the job on failure so nothing lands. Right for structural rules: if the primary key isn&#8217;t unique, downstream is going to be wrong no matter what you do next.</li>

<li><strong>Quarantine.</strong> Filter the row-level outcomes on the evaluation result column, write passing rows to the target and failing rows to a separate prefix. Right for value-level rules where 2% bad records shouldn&#8217;t block 98% good ones. The important discipline is that somebody has to actually look at the quarantine prefix; an unmonitored quarantine is a delete with extra steps.</li>

<li><strong>Warn and continue.</strong> Let everything land, publish the metrics, alert on the score. Right for new rules you don&#8217;t trust yet, and for the catalog-side ruleset.</li>
</ol>



<p class="wp-block-paragraph">Whichever you pick, get the failure out of Glue and into wherever your team actually looks. Results and anomaly predictions can be written back to catalog tables so they are queryable with plain SQL, and CloudWatch metrics plus EventBridge events give you the hook for paging or pausing a downstream pipeline. A Grafana or Datadog dashboard fed from those metrics beats a score in a console tab nobody opens.</p>



<h2 class="wp-block-heading">Troubleshooting AWS Glue Data Quality on SaaS pipelines</h2>



<h3 class="wp-block-heading">&#8220;Input data does not include column X&#8221;</h3>



<p class="wp-block-paragraph">A rule references a column the engine cannot find. On SaaS data that means one of three things: the field disappeared from the source, your flattening step renamed it, or the Data Catalog is out of sync with the underlying table. Check the catalog schema against the landed files before touching the ruleset; rerunning the crawler resolves a surprising share of these.</p>



<h3 class="wp-block-heading">A rule type isn&#8217;t recognised</h3>



<p class="wp-block-paragraph">DQDL is case sensitive and the rule set has grown over releases, so a rule copied from an older blog post can parse fine in your head and not at all in the engine. Check the rule name against the current DQDL rule type reference, and check that the feature is available on your Glue version. Anomaly detection in particular has a version floor.</p>



<h3 class="wp-block-heading">A dynamic rule never fails</h3>



<p class="wp-block-paragraph">Almost always the empty-history problem described above. Look at the statistics for that metric in the Data Quality tab. If there are fewer recorded points than your <code>last(k)</code> asks for, the rule is not doing what you think. It will start working on its own; the mistake is trusting it in the meantime.</p>



<h3 class="wp-block-heading">Evaluation is slow or the job runs out of memory</h3>



<p class="wp-block-paragraph">Each rule triggers analyzer computation over the dataset, and a large ruleset on a wide flattened SaaS table adds up fast. The transform exposes caching and observations-scope options to tune this. Before reaching for a bigger worker type, cut the ruleset down: most contain a dozen recommended rules nobody reads the results of.</p>



<h3 class="wp-block-heading">A &#8220;column reference is ambiguous&#8221; error</h3>



<p class="wp-block-paragraph">Classic flattening artefact. Two nested paths collapsed to the same leaf name, so <code>datechecked</code> now matches two things. Fix it in the projection with explicit aliases rather than trying to disambiguate inside DQDL.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Accepting the recommended ruleset wholesale. Recommendations are generated from a sample of current data, so they encode today&#8217;s accidents as tomorrow&#8217;s requirements.</li>

<li>Writing rules against nested data and assuming they cover the whole payload.</li>

<li>Deploying a dynamic rule and treating it as armed on day one.</li>

<li>Applying whole-table completeness thresholds to a polymorphic object instead of scoping with a <code>where</code> clause.</li>

<li>Quarantining failed rows to a prefix nobody monitors.</li>

<li>Writing <code>RowCount</code> rules against an exploded array without noticing the grain changed.</li>

<li>Failing the job on every rule, including cosmetic ones, until the team starts rerunning with checks disabled.</li>

<li>Keeping the ruleset in the console instead of in version control next to the job.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Split rules into blocking and reporting tiers, and be honest about which is which. A rule that always fails and never stops anything trains people to ignore the tab.</li>

<li>Pair <code>ColumnExists</code> with <code>ColumnCount</code>. One protects the fields you named, the other catches the drift you did not anticipate.</li>

<li>Add analyzers early, even for metrics you have no rule for yet. History has to accumulate before dynamic rules and anomaly detection are worth anything, and you cannot backfill it.</li>

<li>Keep DQDL in Git alongside the job definition and deploy it with Terraform or CloudFormation. Rulesets are code and should move through the same review as the pipeline they guard.</li>

<li>Namespace the evaluation context per source object so CloudWatch metrics and alerts can be routed per vendor.</li>

<li>Exclude incident periods from the anomaly detection training window once the incident is resolved.</li>

<li>Surface results where stakeholders already look. Writing outcomes back to catalog tables makes them queryable from Athena, which beats a score in a console tab.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How much does AWS Glue Data Quality cost?</h3>



<p class="wp-block-paragraph">It bills on Glue DPU-hours like any other Glue workload, so cost is a function of evaluation time and DPUs used. The dimension that surprises people is anomaly detection, which consumes additional capacity per statistic analysed, so enabling it broadly across many columns and tables is where the bill moves. Model it against the current Glue pricing page; rates change and vary by region.</p>



<h3 class="wp-block-heading">Can it check data before it lands in S3?</h3>



<p class="wp-block-paragraph">Yes, if the check runs inside the Glue job. The transform evaluates a DynamicFrame in memory, so you can validate after extraction and before the write. That is the placement that lets you stop a bad load rather than report on one.</p>



<h3 class="wp-block-heading">Does it work on nested JSON from a SaaS API?</h3>



<p class="wp-block-paragraph">Not directly. DQDL does not evaluate nested or list-type columns, so you have to flatten the structure first and run the ruleset against the flattened frame. Plan the flattening deliberately, because your rules only ever see what the projection exposes.</p>



<h3 class="wp-block-heading">How does it compare to Great Expectations, Soda or Monte Carlo?</h3>



<p class="wp-block-paragraph">Glue Data Quality&#8217;s real advantage is that it is already inside the runtime and the catalog: no extra infrastructure, no separate agent, one IAM story. The disadvantages are real too. DQDL is less expressive than Python-based assertions, the nested-data limitation is a hard wall, and it only covers what Glue can read. Great Expectations gives you more expressive checks at the cost of running it yourself; Monte Carlo and similar platforms add lineage and cross-warehouse coverage Glue does not attempt. Glue-centric stack, start here. Several engines, a standalone tool fits better.</p>



<h3 class="wp-block-heading">Should I use the recommended rules?</h3>



<p class="wp-block-paragraph">As a starting draft, yes. As a ruleset, no. Recommendations are inferred from a sample of the data as it is today, which means any existing quality problem gets encoded as the expected baseline. Read every generated rule and delete the ones you would not have written yourself.</p>



<h3 class="wp-block-heading">Can I reconcile row counts against the SaaS source itself?</h3>



<p class="wp-block-paragraph">Not directly; it only sees datasets it can read. The usual pattern is to have the extraction step record the count the API reported, land that alongside the data as a control table, then use <code>RowCountMatch</code> or <code>AggregateMatch</code> to compare the two. That turns &#8220;did we get everything&#8221; into a rule the engine can evaluate.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">AWS Glue Data Quality is not really a data quality product. It is a way of writing down what you believe about a dataset so a machine can check it every run, and its value is entirely determined by whether those beliefs are written precisely enough to be wrong.</p>



<p class="wp-block-paragraph">For SaaS sources that means two habits. Flatten deliberately and check immediately after, because your rules only see the projection. And treat every new dynamic or ML-based rule as unarmed until it has history, because a rule that cannot fail yet looks exactly like a rule that is passing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with data quality on your Glue pipelines?</h2>



<p class="wp-block-paragraph">I work with teams whose SaaS pipelines run green while the numbers drift. Things I can help with:</p>



<ul class="wp-block-list">
<li>Writing a DQDL ruleset for a specific SaaS object that catches schema drift instead of just restating the current schema.</li>

<li>Designing the flattening and pivoting step for nested payloads and custom-field bags so the checks actually cover what matters.</li>

<li>Setting up quarantine and fail-fast paths in Glue ETL, with the routing and alerting that makes a quarantine prefix something people act on.</li>

<li>Cross-object reconciliation between SaaS extracts and control counts recorded at extraction time.</li>

<li>Tuning anomaly detection and dynamic rules, including cleaning incident periods out of the training window.</li>

<li>Getting data quality results out of the console and into Grafana, CloudWatch alarms or an Athena-queryable table.</li>
</ul>



<p class="wp-block-paragraph">If you have a ruleset that keeps passing while the data drifts, send me the DQDL and a schema dump. That is usually enough to see where the gap is.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-glue-data-quality-saas-data/">AWS Glue Data Quality for SaaS Data: Catching the Breakage Nobody Deployed</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Shopify Sales Dashboard with AWS: Build One That Actually Reconciles</title>
		<link>https://john-nessime.com/blog/technical-guides/shopify-sales-dashboard-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 04 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Business Intelligence]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Ecommerce Analytics]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[Partition Projection]]></category>
		<category><![CDATA[QuickSight]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[SQL]]></category>
		<category><![CDATA[Webhooks]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=142</guid>

					<description><![CDATA[<p>Most Shopify dashboards built on AWS work perfectly for about three weeks, then quietly drift away from the numbers in the Shopify admin. Here is why that happens, and how to design the ingestion, storage and query layers so your totals still reconcile six months in.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/shopify-sales-dashboard-aws/">Shopify Sales Dashboard with AWS: Build One That Actually Reconciles</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone in the finance channel posts two screenshots side by side. On the left, the dashboard you built. On the right, the Shopify admin. The totals don&#8217;t match, and they&#8217;re not off by a rounding error either. They&#8217;re off by enough that nobody wants to use your dashboard for anything that matters.</p>



<p class="wp-block-paragraph">The frustrating part is that the pipeline is fine. Webhooks are arriving, Lambda is running clean, S3 has the files, Athena returns rows. Nothing is broken in the way monitoring understands &#8220;broken&#8221;. The pipeline is faithfully recording something that is no longer true.</p>



<p class="wp-block-paragraph">This post is about building a Shopify sales dashboard with AWS that survives that conversation. Not the wiring, which is well documented and mostly straightforward, but the design decisions that determine whether your numbers still hold up six months in. I&#8217;ll cover the three ingestion paths and when each one is the right call, why append-only pipelines drift, how to lay out S3 and Athena so recomputation is cheap, and what to do when the totals are already wrong.</p>



<h2 class="wp-block-heading">Why a Shopify sales dashboard with AWS drifts from the admin</h2>



<p class="wp-block-paragraph">Here&#8217;s the thing that catches almost everyone: <strong>a Shopify order is not an event, it&#8217;s a mutable record.</strong></p>



<p class="wp-block-paragraph">An event pipeline assumes facts are immutable once written. A payment happened. A shipment left. You append it, you never touch it again, and the sum of the log is the truth. That model is why streaming architectures are so clean, and it&#8217;s exactly wrong for order data.</p>



<p class="wp-block-paragraph">An order created on Monday can be edited on Tuesday, partially refunded on Friday, and fully refunded three weeks later. Every one of those changes belongs, financially, to Monday. If your pipeline appends the <code>orders/create</code> payload and never revisits it, Monday&#8217;s revenue is frozen at the moment of checkout and it will only ever be too high.</p>



<p class="wp-block-paragraph">This is the invisible failure. Nothing alerts. No queue backs up. Your dashboard is confidently wrong, and the gap widens roughly in proportion to your return rate. A store with a two percent return rate takes a long time to notice. A fashion store running thirty percent returns notices in about a month, usually via an angry accountant.</p>



<h3 class="wp-block-heading">The four adjustments that move historical numbers</h3>



<ul class="wp-block-list">
<li><strong>Refunds.</strong> Full or partial. A refund carries its own <code>created_at</code>, which is when the money moved back. The order it belongs to has a different, earlier date. You need both, and which one you attribute to depends on whether finance wants cash-basis or order-basis reporting. Ask before you build.</li>

<li><strong>Order edits.</strong> A merchant adds a line item or adjusts a quantity after the fact. The original payload is now stale. Shopify exposes both the original and the current totals precisely because of this.</li>

<li><strong>Cancellations.</strong> A cancelled order keeps existing in the API. If you filter only on payment status you will happily keep counting it.</li>

<li><strong>Test and draft orders.</strong> Test orders carry a flag marking them as such. Nobody remembers to filter these until a QA run during a quiet week produces a suspicious spike.</li>
</ul>



<p class="wp-block-paragraph">The design consequence is simple to state and annoying to implement: <strong>your pipeline must be able to recompute any past day.</strong> Every storage and partitioning decision below follows from that one requirement.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Getting data out of Shopify: three paths, three trade-offs</h2>



<p class="wp-block-paragraph">Before anything else: new Shopify apps are built on the GraphQL Admin API. The REST Admin API has been designated a legacy API and new public apps must use GraphQL. If you&#8217;re starting fresh, start there. If you inherited a REST integration, it probably still runs, but you&#8217;re on borrowed time and you should plan the migration rather than discover the deadline.</p>



<h3 class="wp-block-heading">Path 1: EventBridge partner event source</h3>



<p class="wp-block-paragraph">Shopify can deliver webhooks straight into an Amazon EventBridge partner event bus in your account. No public endpoint, no API Gateway, no HMAC verification code, because verification only applies to HTTPS deliveries. Shopify&#8217;s own docs confirm EventBridge and Pub/Sub deliveries skip it.</p>



<p class="wp-block-paragraph">You create the source in the Shopify app configuration using your AWS account ID, region and a source name, then associate it with an event bus in the EventBridge console and write rules to route it. The address you register with Shopify is the <em>partner event source</em> ARN, not the event bus ARN. That distinction accounts for a large share of the &#8220;I set it up and nothing arrives&#8221; threads on the Shopify forums.</p>



<p class="wp-block-paragraph">A rule matching everything from the Shopify partner source looks like this. Start broad, then narrow once you&#8217;ve seen the real shape of an event:</p>



<pre class="wp-block-code"><code>{
  "source": [ { "prefix": "aws.partner/shopify.com" } ]
}</code></pre>



<p class="wp-block-paragraph">Send that to an SQS queue with a dead-letter queue attached rather than straight to Lambda. Buffering gives you a replay buffer when a downstream deploy goes wrong, and the DLQ means a bad payload parks itself instead of poisoning the whole rule. This is the path I reach for first for anything already on AWS.</p>



<h3 class="wp-block-heading">Path 2: HTTPS webhooks into API Gateway and Lambda</h3>



<p class="wp-block-paragraph">The conventional route, and the right one if you need webhook delivery outside AWS too, or you want the payloads to pass through something you fully control. The cost is that you now own an internet-facing endpoint and the HMAC verification on it.</p>



<p class="wp-block-paragraph">Verify against the <strong>raw request body</strong>, before any JSON parsing. Re-serialising the payload changes byte-for-byte content and the signature will never match. Use a constant-time comparison so the check doesn&#8217;t leak timing information:</p>



<pre class="wp-block-code"><code>import base64, hashlib, hmac

def verify(raw_body: bytes, header_hmac: str, secret: str) -&gt; bool:
    digest = hmac.new(secret.encode(), raw_body, hashlib.sha256).digest()
    computed = base64.b64encode(digest).decode()
    return hmac.compare_digest(computed, header_hmac)</code></pre>



<p class="wp-block-paragraph">Shopify sends the signature in the <code>X-Shopify-Hmac-SHA256</code> header, base64-encoded, computed with your app&#8217;s client secret over the raw body. Store that secret in Secrets Manager or as an SSM SecureString parameter, not in a Lambda environment variable.</p>



<p class="wp-block-paragraph">Acknowledge fast. Shopify&#8217;s timeout is short and it retries with backoff over a finite window, so a handler that does real work inline will generate a wall of duplicate deliveries during a flash sale, exactly when you can least afford it. Return 2xx immediately, do the work asynchronously.</p>



<h3 class="wp-block-heading">Path 3: scheduled GraphQL bulk pull</h3>



<p class="wp-block-paragraph">Webhooks give you low latency. They do not give you completeness. Anything that fails past its retry window is gone, and Shopify will eventually remove a subscription that keeps failing. That&#8217;s a silent data loss mode with no local symptom at all.</p>



<p class="wp-block-paragraph">So run a scheduled reconciliation pull alongside the stream. Shopify&#8217;s GraphQL bulk operations are built for this: you submit a query, it runs asynchronously, and you fetch a JSONL result file when it finishes. That&#8217;s the right tool for backfills and nightly catch-up, rather than paginating thousands of pages against a points-based rate limiter and getting throttled halfway through.</p>



<p class="wp-block-paragraph">A nightly job that re-pulls the last seven to fourteen days and overwrites those partitions costs almost nothing and quietly fixes every category of drift described above. If you build one thing from this post, build that.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Decide what &#8220;revenue&#8221; means before you write a line of SQL</h2>



<p class="wp-block-paragraph">This is where most reconciliation arguments actually live, and it isn&#8217;t an engineering problem at all until you&#8217;ve had the conversation.</p>



<p class="wp-block-paragraph">Shopify&#8217;s own sales reporting builds total sales from gross sales, minus discounts, minus returns, plus taxes and shipping. Gift card sales sit outside that in a separate finance report. If your dashboard sums order totals and calls it revenue, you have built a different metric with the same name, and it will disagree with the admin forever no matter how good your pipeline is.</p>



<p class="wp-block-paragraph">Write the definition down. Put it in the dashboard as a tooltip. When someone challenges a number, you want the argument to be about the definition, not about whether your infrastructure works.</p>



<h3 class="wp-block-heading">The currency trap</h3>



<p class="wp-block-paragraph">If the store sells in more than one currency, the money fields split in two. Shopify exposes totals as a set containing both <code>shop_money</code> and <code>presentment_money</code>: the amount in the store&#8217;s base currency, and the amount the customer actually saw and paid.</p>



<p class="wp-block-paragraph">Sum the presentment amounts across a multi-currency store and you get a number with no meaning at all, euros and yen added together as if they were the same unit. For a single reporting figure you want the shop-currency side. Keep the presentment amount and its currency code in the table anyway, because the day someone asks &#8220;how much did we actually sell in Germany&#8221;, you&#8217;ll want it and it is painful to backfill.</p>



<p class="wp-block-paragraph">One caveat worth knowing: orders created through the API rather than through checkout can behave differently from native multi-currency checkout orders. If your store takes orders from an ERP or a marketplace integration, spot-check a few of those specifically.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Storage layout: partition by order date, never by arrival date</h2>



<p class="wp-block-paragraph">Two layers in S3. Keep them separate and keep them honest about what they are.</p>



<ol class="wp-block-list">
<li><strong>Raw.</strong> Every payload exactly as received, partitioned by ingestion date. Append-only, never edited. This is your audit trail and your rebuild source. Lifecycle it to a colder storage class after a few months, don&#8217;t delete it.</li>

<li><strong>Curated.</strong> One row per order representing current state, in Parquet, partitioned by <em>order date</em>. This is what the dashboard queries. It is derived, disposable and rewritable.</li>
</ol>



<p class="wp-block-paragraph">The partitioning choice on the curated layer is the load-bearing decision in the whole design. If you partition by arrival date, which is what Amazon Data Firehose does by default because it buckets on the moment it writes the file, then a refund that arrives three weeks late lands in today&#8217;s partition. Correcting Monday now means finding and rewriting fragments scattered across twenty other partitions. Partitioned by order date, correcting Monday means overwriting exactly one prefix.</p>



<p class="wp-block-paragraph">Firehose can do this with dynamic partitioning, which routes records by keys inside the payload rather than by write time. If you&#8217;re not using Firehose, extract the order date in your Lambda and write the prefix yourself.</p>



<h3 class="wp-block-heading">Use partition projection so Athena stops guessing</h3>



<p class="wp-block-paragraph">The default Glue Data Catalog approach means running a crawler or issuing <code>MSCK REPAIR TABLE</code> to register new partitions. Forget one and you get a query that silently returns nothing for recent days. Nobody notices until Monday.</p>



<p class="wp-block-paragraph">Partition projection removes the metastore lookup entirely. You tell Athena the shape of the partition keys and it calculates the prefixes at query time:</p>



<pre class="wp-block-code"><code>CREATE EXTERNAL TABLE shop_orders (
  order_id             bigint,
  order_number         string,
  created_at           timestamp,
  financial_status     string,
  cancelled_at         timestamp,
  is_test              boolean,
  total_shop           decimal(12,2),
  shop_currency        string,
  total_presentment    decimal(12,2),
  presentment_currency string
)
PARTITIONED BY (order_date string)
STORED AS PARQUET
LOCATION 's3://your-bucket/curated/orders/'
TBLPROPERTIES (
  'projection.enabled' = 'true',
  'projection.order_date.type' = 'date',
  'projection.order_date.format' = 'yyyy-MM-dd',
  'projection.order_date.range' = '2019-01-01,NOW',
  'projection.order_date.interval' = '1',
  'projection.order_date.interval.unit' = 'DAYS',
  'storage.location.template' =
    's3://your-bucket/curated/orders/order_date=${order_date}/'
);</code></pre>



<p class="wp-block-paragraph">Set the range start to your store&#8217;s actual first order month. Projection generates every prefix in the range, so a range starting a decade too early makes wide scans slower for no benefit.</p>



<p class="wp-block-paragraph">Parquet matters here for the same reason. Athena bills on bytes scanned, so a columnar format with good compression cuts the bill directly, and a dashboard that only ever selects six columns from a forty-column table never touches the rest.</p>



<h3 class="wp-block-heading">Net sales in one query</h3>



<p class="wp-block-paragraph">With refunds in their own table keyed by order and carrying their own date, attributing them back to the original order day is a left join and a subtraction:</p>



<pre class="wp-block-code"><code>SELECT
    o.order_date,
    SUM(o.total_shop)                                AS gross_shop,
    SUM(COALESCE(r.refunded_shop, 0))                AS refunded_shop,
    SUM(o.total_shop - COALESCE(r.refunded_shop, 0)) AS net_shop
FROM shop_orders o
LEFT JOIN (
    SELECT order_id, SUM(amount_shop) AS refunded_shop
    FROM shop_refunds
    GROUP BY order_id
) r ON r.order_id = o.order_id
WHERE o.order_date BETWEEN '2025-01-01' AND '2025-01-31'
  AND o.is_test = false
  AND o.cancelled_at IS NULL
GROUP BY o.order_date
ORDER BY o.order_date;</code></pre>



<p class="wp-block-paragraph">Note the two filters doing quiet work at the bottom. Those two lines are the difference between a number finance accepts and a number they don&#8217;t.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Choosing the dashboard layer</h2>



<p class="wp-block-paragraph">Once the data is correct, this part is genuinely a preference. All of these work.</p>



<ul class="wp-block-list">
<li><strong>Amazon QuickSight</strong>, now delivered as part of Amazon Quick Suite, is the least-friction option if you&#8217;re already in AWS. Its in-memory SPICE layer means viewers aren&#8217;t firing an Athena query per chart interaction, which controls both latency and scan cost. Per-viewer pricing tends to be the deciding factor either way, so model it for your actual audience size before committing.</li>

<li><strong>Grafana</strong> with the Athena data source is a good fit if you&#8217;re already running Grafana for infrastructure and want commercial and operational panels on one screen. Grafana Cloud removes the hosting question if you&#8217;d rather not run it.</li>

<li><strong>Power BI</strong> makes sense when the finance team already lives in Microsoft 365 and models in DAX. The cross-cloud hop is real but manageable.</li>

<li><strong>Metabase</strong> or a self-hosted alternative on a small VPS from a provider like InterServer or Hetzner is the pragmatic answer for a handful of internal viewers, where per-seat BI licensing costs more than the entire pipeline.</li>
</ul>



<p class="wp-block-paragraph">The honest trade-off: managed BI costs more per month and saves you from becoming the person who patches the reporting server. Self-hosting inverts that. Neither is wrong, but pick deliberately rather than by inertia.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting: symptom to cause</h2>



<h3 class="wp-block-heading">Totals are consistently higher than the Shopify admin</h3>



<p class="wp-block-paragraph">Almost always refunds, cancellations or test orders. Check in that order. If the gap grows with the age of the reporting window, it&#8217;s refunds. If it&#8217;s a fixed offset on specific days, look for test orders or a QA run.</p>



<h3 class="wp-block-heading">Totals are lower, and recent days are missing rows</h3>



<p class="wp-block-paragraph">Either partitions aren&#8217;t registered, which projection fixes permanently, or the webhook subscription has been dropped after repeated delivery failures. Check the subscription still exists before you go digging through Lambda logs. A nightly bulk pull would have masked this, which is another argument for having one.</p>



<h3 class="wp-block-heading">Orders appear twice</h3>



<p class="wp-block-paragraph">Shopify&#8217;s delivery model is at-least-once, not exactly-once, and you may also have more than one subscription on the same topic. Deduplicate on the delivery ID header before you touch anything else, and make the write itself idempotent so a duplicate is a no-op rather than a second row.</p>



<h3 class="wp-block-heading">Numbers are right on the daily view, wrong on the monthly</h3>



<p class="wp-block-paragraph">Timezone. Order timestamps carry an offset; your partition key is a date string. If you derive the date in UTC and the store reports in a local timezone, orders near midnight land on the wrong day. That averages out over a month, which is exactly why the discrepancy hides until month boundaries.</p>



<h3 class="wp-block-heading">Athena costs jumped without more data</h3>



<p class="wp-block-paragraph">Someone built a dashboard with a filter that doesn&#8217;t hit the partition column, so every panel refresh scans the full table. Look at bytes scanned per query and check whether the BI tool is caching results or re-querying on every interaction.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating orders as immutable events and never revisiting a past day.</li>

<li>Partitioning on arrival time because that&#8217;s the default, then discovering corrections are expensive.</li>

<li>Verifying the HMAC against a re-serialised body instead of the raw bytes.</li>

<li>Doing real work inside the webhook handler, generating duplicates under load.</li>

<li>Summing presentment amounts across currencies.</li>

<li>Registering the event bus ARN with Shopify instead of the partner event source ARN.</li>

<li>Relying on webhooks alone with no scheduled reconciliation.</li>

<li>Shipping a &#8220;revenue&#8221; number without ever defining what it includes.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Keep raw and curated layers separate. Raw is append-only; curated is rewritable.</li>

<li>Make every partition idempotently rebuildable from raw. Test that path deliberately, before you need it.</li>

<li>Buffer through SQS with a dead-letter queue. Free replay, free isolation of bad payloads.</li>

<li>Run a nightly bulk pull over a rolling window and overwrite those partitions.</li>

<li>Use partition projection. It removes an entire category of silent failure.</li>

<li>Alarm on the absence of events, not just on errors. A CloudWatch alarm on zero orders processed in an hour during business hours catches broken subscriptions the same day.</li>

<li>Store the API secret in Secrets Manager and scope the Lambda role to the exact prefixes it writes.</li>

<li>Publish a reconciliation panel comparing your total to the admin&#8217;s for the same window. Surfacing the gap builds more trust than hiding it.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need a data warehouse, or is S3 and Athena enough?</h3>



<p class="wp-block-paragraph">For a single store&#8217;s order data, S3 with Athena is almost certainly enough, and it&#8217;s cheaper because you pay per query rather than for a running cluster. Redshift starts to earn its place when you&#8217;re joining Shopify data against several other large sources, or when concurrent query load makes Athena&#8217;s queue times noticeable.</p>



<h3 class="wp-block-heading">How near-real-time can this be?</h3>



<p class="wp-block-paragraph">Events land within seconds. The practical floor is your buffering window, since writing one tiny file per order gives you a small-files problem that ruins query performance. A few minutes of buffering is the usual compromise. If you genuinely need sub-minute order counts, put a live counter in DynamoDB alongside the analytical pipeline rather than trying to make the data lake do both jobs.</p>



<h3 class="wp-block-heading">Which webhook topics should I subscribe to?</h3>



<p class="wp-block-paragraph">At minimum, order creation, order update, order cancellation and refund creation. Update and refund topics are the ones people skip, and they&#8217;re exactly the ones carrying the corrections. Subscribe to fewer topics than you think you need and add rather than subscribing to everything, since every extra topic is volume you pay to store and process.</p>



<h3 class="wp-block-heading">Can I skip AWS and use a connector tool?</h3>



<p class="wp-block-paragraph">Yes, and for many stores that&#8217;s the right answer. A managed connector into a hosted warehouse gets you a working dashboard in an afternoon. You&#8217;re paying a monthly fee to avoid owning any of this, and trading away control over the data model. Building it on AWS wins when you need Shopify data joined to systems the connector doesn&#8217;t cover, or when row-based connector pricing outgrows the infrastructure cost.</p>



<h3 class="wp-block-heading">How do I backfill historical orders?</h3>



<p class="wp-block-paragraph">Use a GraphQL bulk operation rather than paginating the API. Submit the query, poll for completion, then stream the JSONL result into your raw bucket and run the same transformation your live pipeline uses. If backfill and live processing use different code paths, they will diverge, and you&#8217;ll spend an afternoon working out which one is lying.</p>



<h3 class="wp-block-heading">What does a setup like this cost to run?</h3>



<p class="wp-block-paragraph">For a typical single store, the pipeline itself is small money: Lambda invocations, a few gigabytes in S3, and Athena billed on bytes scanned, which partitioning and Parquet keep low. The BI seats are usually the largest line item, which is why the dashboard layer decision deserves more thought than the ingestion one. Model it against current published rates rather than trusting any figure you read in a blog post, including this one.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">A Shopify sales dashboard with AWS doesn&#8217;t fail because the pipeline breaks. It fails because the pipeline keeps working perfectly on data that has since changed underneath it.</p>



<p class="wp-block-paragraph">Design for correction from the first commit. Partition by order date, keep the raw layer so you can always rebuild, run a scheduled pull to catch what the stream missed, and agree on what revenue means before anyone builds a chart. Do that and the Monday morning screenshot comparison becomes a non-event, which is the highest praise a reporting pipeline ever gets.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with your Shopify data pipeline on AWS?</h2>



<p class="wp-block-paragraph">I design and build ecommerce data pipelines and reporting stacks on AWS. Typical engagements look like:</p>



<ul class="wp-block-list">
<li>Working out why an existing Shopify dashboard disagrees with the admin, and fixing the root cause rather than patching the query</li>

<li>Building the ingestion layer end to end: EventBridge or API Gateway, Lambda, SQS with dead-letter handling, and a scheduled GraphQL bulk reconciliation job</li>

<li>Designing the S3 layout, Glue schema and Athena tables so past days can be recomputed cheaply and partitions never go missing</li>

<li>Migrating REST Admin API integrations to GraphQL before the deadline forces the issue</li>

<li>Building the dashboard itself in QuickSight, Grafana or Metabase, including the metric definitions finance will actually sign off on</li>

<li>Cutting Athena scan costs and BI licensing on a reporting stack that has grown more expensive than anyone planned</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;re in the middle of one of these, send me the actual thing: the Athena query, the S3 prefix layout, the two totals that don&#8217;t match. It&#8217;s a much faster conversation than describing it in the abstract.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/shopify-sales-dashboard-aws/">Shopify Sales Dashboard with AWS: Build One That Actually Reconciles</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a Jira Analytics Pipeline with AWS Lambda and Athena (Without Double-Counting Everything)</title>
		<link>https://john-nessime.com/blog/devops/jira-analytics-pipeline-aws-lambda-athena/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 04 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Business Intelligence]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Engineering Metrics]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Jira]]></category>
		<category><![CDATA[Partition Projection]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[SQL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=125</guid>

					<description><![CDATA[<p>Jira's built-in reports stop at the board boundary. This guide walks through a Jira analytics pipeline built on AWS Lambda, S3 and Athena, organised around the four failure families that actually bite: the removed search endpoint, silently truncated changelogs, incremental loads that duplicate rows, and an S3 layout that quietly inflates your query bill.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/jira-analytics-pipeline-aws-lambda-athena/">Building a Jira Analytics Pipeline with AWS Lambda and Athena (Without Double-Counting Everything)</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone in a delivery review asks why cycle time went up last quarter. You open the Jira dashboard, and it can tell you what is in progress right now, roughly. It cannot tell you what &#8220;in progress&#8221; meant three months ago, how long each ticket sat in code review, or whether the increase came from one team or all six. The control chart resets when the board configuration changes, the sprint report only knows about sprints, and the CSV export tops out somewhere unhelpful.</p>



<p class="wp-block-paragraph">That is usually the moment someone says &#8220;let&#8217;s just pull it into a warehouse.&#8221; A <strong>Jira analytics pipeline</strong> built on AWS Lambda, S3 and Athena is a reasonable answer to that, and it is genuinely cheap to run. It is also easy to build a version that looks correct for two weeks and then quietly reports numbers that are thirty percent wrong.</p>



<p class="wp-block-paragraph">This post covers the extraction and modelling problems that actually cost you time: the search endpoint Atlassian removed, the change history that truncates without erroring, the incremental load pattern that duplicates rows across partitions, and the S3 layout decisions that decide whether Athena costs you pennies or hundreds. Code where it clarifies something, and honest notes on what I would skip.</p>



<h2 class="wp-block-heading">Where Jira&#8217;s own reporting genuinely stops</h2>



<p class="wp-block-paragraph">Give the built-in tooling its due first. Jira&#8217;s velocity, burndown and control charts are fine for a single team inspecting its own recent work, they need no infrastructure, and they update instantly. Marketplace apps like eazyBI and Custom Charts cover a lot of ground without you writing a line of Python. If your question is &#8220;how did this sprint go,&#8221; you do not need a pipeline.</p>



<p class="wp-block-paragraph">Where it stops is anything that crosses a boundary. Comparing lead time across projects that use different workflows. Joining ticket data to deploy events from your CI system or incident data from PagerDuty. Retaining a consistent view of history after someone renames a status or archives a board. Answering a question nobody anticipated when the board was configured. Those need the raw data somewhere you control, in a shape you decide.</p>



<h2 class="wp-block-heading">The shape of the pipeline</h2>



<p class="wp-block-paragraph">The architecture is unremarkable, which is the point:</p>



<ol class="wp-block-list">
<li>EventBridge Scheduler triggers a Lambda function on a schedule.</li>

<li>Lambda reads a Jira API token from Secrets Manager and pages through the Jira Cloud REST API.</li>

<li>It writes Parquet files to S3, partitioned by load date.</li>

<li>The Glue Data Catalog holds the table definitions, with partition projection so nothing has to crawl.</li>

<li>Athena queries S3 directly. Grafana, Power BI, Metabase or QuickSight sit on top of Athena.</li>
</ol>



<p class="wp-block-paragraph">No cluster, no always-on database, nothing to patch. The whole thing costs about as much as a small EC2 instance for a mid-sized Jira site, and most of that is S3 storage. Terraform or CloudFormation to define it, GitHub Actions to deploy it.</p>



<p class="wp-block-paragraph">The complexity is not in the wiring. It is in four places, and they are worth taking in order.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family 1: the search endpoint you were probably going to use is gone</h2>



<p class="wp-block-paragraph">Almost every Jira extraction tutorial and a good number of client libraries still reach for <code>GET /rest/api/3/search</code>. Atlassian removed it from Jira Cloud. It returns 410 Gone. If you are copying a script from a blog post that predates the change, this is the first thing that breaks, and at least it breaks loudly.</p>



<p class="wp-block-paragraph">The replacement is <code>/rest/api/3/search/jql</code>, available as both GET and POST. Use POST for anything real, because JQL strings get long and you avoid URL encoding entirely. Three behavioural changes matter more than the URL:</p>



<h3 class="wp-block-heading">Pagination is cursor-based, and there is no total</h3>



<p class="wp-block-paragraph"><code>startAt</code> is gone. You get a <code>nextPageToken</code> back and hand it to the next request. There is no <code>total</code> in the response, which means any progress bar, any &#8220;expected N rows&#8221; sanity check, and any loop that terminated on <code>startAt &gt;= total</code> needs rewriting.</p>



<p class="wp-block-paragraph">If you only need a count, there is a separate operation, <code>POST /rest/api/3/search/approximate-count</code>, which takes a JQL body and returns an approximate figure without paging through results. It is genuinely useful as a reconciliation check: run it before extraction, compare against the row count you actually wrote, and alert on a large gap.</p>



<p class="wp-block-paragraph">There have been persistent community reports of <code>isLast</code> behaving unreliably on this endpoint, including tokens that chain without ever terminating. Do not trust <code>isLast</code> as your loop condition. Terminate on the absence of <code>nextPageToken</code>, and put a hard page cap in as a circuit breaker so a bad token cannot burn your entire Lambda budget in one invocation.</p>



<pre class="wp-block-code"><code>def search_issues(session, base_url, jql, fields, max_pages=2000):
    """Page through /search/jql. Terminates on missing nextPageToken,
    not on isLast, which has been reported as unreliable."""
    token = None
    for _ in range(max_pages):
        body = {"jql": jql, "fields": fields, "maxResults": 100}
        if token:
            body["nextPageToken"] = token
        r = session.post(f"{base_url}/rest/api/3/search/jql",
                         json=body, timeout=60)
        r.raise_for_status()
        page = r.json()
        for issue in page.get("issues", []):
            yield issue
        token = page.get("nextPageToken")
        if not token:
            return
    raise RuntimeError("page cap hit, refusing to loop further")</code></pre>



<h3 class="wp-block-heading">You have to ask for fields explicitly</h3>



<p class="wp-block-paragraph">The new endpoint does not hand you every field by default. Omit <code>fields</code> and you get essentially nothing back. This is the failure that looks like success: the pipeline runs, files land in S3, row counts look plausible, and every analytical column is null.</p>



<p class="wp-block-paragraph">Be explicit and be narrow. Every field you request costs response size, and Jira sites accumulate hundreds of custom fields nobody uses. Name what you need:</p>



<pre class="wp-block-code"><code>FIELDS = [
    "summary", "status", "issuetype", "project", "priority",
    "assignee", "reporter", "created", "updated", "resolutiondate",
    "labels", "components", "parent",
    "customfield_10016",   # story points on this site, verify yours
]</code></pre>



<p class="wp-block-paragraph">Custom field IDs are per-site. Do not hardcode one you read in someone else&#8217;s blog post. Pull <code>/rest/api/3/field</code> once, find the field by name, and either store the mapping in config or resolve it at runtime and log what it resolved to. When a Jira admin rebuilds a field, an ID-based pipeline goes null and a name-resolving pipeline keeps working.</p>



<h3 class="wp-block-heading">Rate limits and the fifteen-minute wall</h3>



<p class="wp-block-paragraph">Jira Cloud applies cost-based rate limiting and returns HTTP 429 when you exceed it, typically with a <code>Retry-After</code> header. Respect that header rather than inventing your own backoff. A naive retry loop that ignores it turns a brief throttle into a sustained one.</p>



<p class="wp-block-paragraph">Lambda&#8217;s hard ceiling is fifteen minutes. A full historical backfill of a large Jira site will not finish in one invocation, and the ugly failure mode is a function that times out at minute fifteen having written half its data with no record of where it stopped. Two ways out:</p>



<ul class="wp-block-list">
<li><strong>Shard the work.</strong> Fan out one Lambda invocation per project key, or per month of created date. Each one is small, independently retryable, and finishes well inside the limit.</li>

<li><strong>Checkpoint and continue.</strong> Persist the current <code>nextPageToken</code> to DynamoDB or S3 after each page. When the function is close to its deadline, stop cleanly and let Step Functions re-invoke it from the checkpoint.</li>
</ul>



<p class="wp-block-paragraph">Sharding is simpler and I reach for it first. Checkpointing is what you need when a single project is itself too large. Either way, watch Lambda&#8217;s ephemeral storage: the default <code>/tmp</code> allocation is 512 MB, and buffering a large Parquet write there will fail before your API calls do. Raise it or stream to S3 in chunks.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family 2: the change history is the whole point, and it truncates silently</h2>



<p class="wp-block-paragraph">Current issue state answers almost none of the interesting questions. How long a ticket spent waiting for review, how many times it bounced back from QA, when it actually entered development rather than when someone remembered to drag the card, all of that lives in the changelog. Without it you have a list of tickets. With it you have a process.</p>



<p class="wp-block-paragraph">Here is the trap. Requesting an issue with <code>expand=changelog</code> returns a capped number of history entries, commonly the first hundred, and it does not tell you it truncated. Well-worn tickets with lots of field edits blow past that easily. Your data does not error, it just quietly loses the later transitions, which are usually the ones near completion. Cycle time comes out looking better than reality.</p>



<p class="wp-block-paragraph">Two correct approaches:</p>



<ul class="wp-block-list">
<li><code>GET /rest/api/3/issue/{issueIdOrKey}/changelog</code> and page it properly. Correct, but it is one request per issue, which is brutal against rate limits on a large site.</li>

<li><code>POST /rest/api/3/changelog/bulkfetch</code>, which accepts <code>issueIdsOrKeys</code> and an optional <code>fieldIds</code> filter, with the same <code>nextPageToken</code> pagination. Far fewer round trips.</li>
</ul>



<p class="wp-block-paragraph">Bulk fetch is the one I would use, with a caveat: it has carried an experimental designation, so pin your expectations and keep the per-issue path available as a fallback. Filter <code>fieldIds</code> to <code>status</code> if status history is all you model. That cuts the response size enormously, because most changelog volume is description edits and label churn nobody analyses.</p>



<p class="wp-block-paragraph">Store changelog as its own narrow table, one row per field change, not nested inside the issue record. Athena can handle nested structures, but flat is dramatically easier to reason about in SQL and much cheaper to scan.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family 3: the incremental load that duplicates everything</h2>



<p class="wp-block-paragraph">This is the one that bites hardest, because nothing fails. Everything runs green for weeks and the numbers are wrong the entire time.</p>



<p class="wp-block-paragraph">The obvious incremental design is a watermark: track the last successful run, then pull <code>updated &gt;= watermark</code> on each run and append the results to S3. It is the standard pattern and it works for immutable event data.</p>



<p class="wp-block-paragraph">Jira issues are not immutable. A ticket created in January and touched every week appears in every weekly extract. If you append each run into its own partition, that one issue now exists as a row in twenty partitions. Athena is doing exactly what you asked: <code>SELECT count(*) FROM jira_issues WHERE status = 'Open'</code> counts twenty things. Your open bug count is inflated, your throughput is inflated, and the inflation is proportional to how much a ticket gets edited, which correlates with how contentious it was. The busiest work is the most over-counted.</p>



<p class="wp-block-paragraph">Nobody catches this from the dashboard, because the numbers move in the right direction and look approximately sane. It surfaces months later when someone reconciles against a JQL query in Jira and the two disagree.</p>



<h3 class="wp-block-heading">Pick one of three fixes, deliberately</h3>



<p class="wp-block-paragraph"><strong>Full snapshot per load, dedupe at read time.</strong> Keep every version, partition by <code>load_date</code>, and always read through a view that takes the latest row per issue key. Storage is cheap, history is free, and you can answer &#8220;what did the board look like in March&#8221; without any extra machinery. The cost is that every query pays for the deduplication.</p>



<pre class="wp-block-code"><code>CREATE OR REPLACE VIEW jira_issues_current AS
SELECT * FROM (
  SELECT
    i.*,
    ROW_NUMBER() OVER (
      PARTITION BY issue_key
      ORDER BY load_date DESC, updated DESC
    ) AS rn
  FROM jira_issue_snapshot i
  WHERE load_date &gt;= date_format(current_date - interval '7' day, '%Y-%m-%d')
) WHERE rn = 1;</code></pre>



<p class="wp-block-paragraph">The <code>load_date</code> filter inside the view matters. Without it the deduplication window scans the entire table on every query, which is the single most common way a cheap Athena setup becomes an expensive one.</p>



<p class="wp-block-paragraph"><strong>Overwrite the affected partitions.</strong> Partition by something stable, usually issue created month, and rewrite whole partitions when any issue in them changes. Clean reads, no dedupe cost, but you now own read-modify-write logic in Lambda and a concurrency problem if two runs overlap.</p>



<p class="wp-block-paragraph"><strong>Use an ACID table format.</strong> Apache Iceberg gives you real <code>MERGE INTO</code> semantics on S3, and Athena supports it natively. This is the right answer if you are already running Iceberg elsewhere or if the pipeline will grow to a dozen sources. It is not worth adopting solely to load one Jira site, because you inherit compaction and snapshot expiry as ongoing maintenance.</p>



<p class="wp-block-paragraph">For a single Jira site feeding a handful of dashboards, snapshot plus a dedupe view is what I would build. It has the fewest moving parts and it gives you point-in-time history as a side effect, which you will want the first time someone asks a retrospective question.</p>



<h3 class="wp-block-heading">Deletions and moves</h3>



<p class="wp-block-paragraph">A JQL watermark query never returns deleted issues, so they persist in your data forever. Same for issues moved out of scope or into an archived project. Periodically reconcile: pull the full set of issue keys with a minimal <code>fields</code> list, compare against what you hold, and mark the difference. Monthly is usually enough. Skip this and your historical counts drift upward permanently.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family 4: the S3 layout that decides your Athena bill</h2>



<p class="wp-block-paragraph">Athena&#8217;s standard pricing model bills on bytes scanned, rounded up, with a small per-query minimum. DDL statements are free. That means your storage layout, not your SQL, is what determines cost.</p>



<p class="wp-block-paragraph">Three things do almost all the work:</p>



<ul class="wp-block-list">
<li><strong>Write Parquet, not JSON.</strong> Columnar storage lets Athena read only the columns your query touches. A dashboard selecting five columns from a forty-column table scans a small fraction of what the equivalent JSON would. Snappy or Zstd compression on top of that.</li>

<li><strong>Avoid tiny files.</strong> A Lambda that runs hourly and writes one small file per invocation produces thousands of objects. Athena spends more time opening files than reading them. Batch writes so files land in the low hundreds of megabytes, or run a periodic compaction job.</li>

<li><strong>Use partition projection.</strong> This is the one people skip and then wonder why queries have a fixed few-second overhead.</li>
</ul>



<p class="wp-block-paragraph">Partition projection lets Athena calculate partition locations from table properties instead of calling <code>GetPartitions</code> against the Glue Data Catalog. On a table with a couple of years of daily partitions, that lookup is real latency on every single query, and it grows as the table does. Projection removes it, and removes your need for a Glue crawler entirely, which is one less scheduled job and one less IAM role.</p>



<pre class="wp-block-code"><code>CREATE EXTERNAL TABLE jira_issue_snapshot (
  issue_id        string,
  issue_key       string,
  project_key     string,
  issue_type      string,
  status          string,
  status_category string,
  assignee_id     string,
  created         timestamp,
  updated         timestamp,
  resolutiondate  timestamp,
  story_points    double
)
PARTITIONED BY (load_date string)
STORED AS PARQUET
LOCATION 's3://example-jira-lake/issue_snapshot/'
TBLPROPERTIES (
  'projection.enabled'                = 'true',
  'projection.load_date.type'         = 'date',
  'projection.load_date.format'       = 'yyyy-MM-dd',
  'projection.load_date.range'        = 'NOW-3YEARS,NOW',
  'projection.load_date.interval'     = '1',
  'projection.load_date.interval.unit'= 'DAYS',
  'storage.location.template'         =
    's3://example-jira-lake/issue_snapshot/load_date=${load_date}'
);</code></pre>



<p class="wp-block-paragraph">One caution that surprises people: projection describes partitions Athena will look for, not partitions that exist. Set a range wider than your data and queries without a <code>load_date</code> filter will probe empty prefixes. Keep the range tight to what you actually hold.</p>



<p class="wp-block-paragraph">Finally, set <code>BytesScannedCutoffPerQuery</code> on the Athena workgroup. It kills any query that exceeds a scan threshold. One analyst running <code>SELECT *</code> against three years of data in a BI tool&#8217;s preview pane is the classic surprise line item, and this stops it at the source.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Modelling time in status</h2>



<p class="wp-block-paragraph">Once status changelog rows are landed flat, the core metric is a window function. Each transition&#8217;s duration is the gap to the next transition on the same issue:</p>



<pre class="wp-block-code"><code>WITH transitions AS (
  SELECT
    issue_key,
    to_status,
    changed_at,
    LEAD(changed_at) OVER (
      PARTITION BY issue_key ORDER BY changed_at
    ) AS next_changed_at
  FROM jira_changelog
  WHERE field_id = 'status'
    AND load_date &gt;= date_format(current_date - interval '90' day, '%Y-%m-%d')
)
SELECT
  issue_key,
  to_status,
  SUM(date_diff('second', changed_at,
                COALESCE(next_changed_at, current_timestamp))) / 3600.0
    AS hours_in_status
FROM transitions
GROUP BY issue_key, to_status;</code></pre>



<p class="wp-block-paragraph">The <code>COALESCE</code> handles the current status, which has no successor transition. Two modelling decisions to make consciously: whether to subtract non-working hours, and how to treat a ticket that moves backwards through the workflow. Both are business questions, not technical ones, and both should be settled in a documented view rather than reimplemented in each dashboard.</p>



<p class="wp-block-paragraph">Map raw status names to a stable category early. Teams rename statuses constantly, and a metric keyed on the literal string breaks the moment someone changes &#8220;In Review&#8221; to &#8220;Peer Review&#8221;. A small mapping table joined at query time keeps history comparable.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Every analytical column is null.</strong> You did not pass <code>fields</code>, or you passed a custom field ID from another site. Log the resolved field list on every run.</li>

<li><strong>Counts higher than the same JQL in Jira.</strong> Duplicate rows across partitions. Check whether your query goes through the dedupe view or straight at the base table.</li>

<li><strong>Extraction loops forever.</strong> Do not terminate on <code>isLast</code>. Terminate on missing <code>nextPageToken</code> and keep a page cap.</li>

<li><strong>HTTP 410 from the API.</strong> You are still calling the removed <code>/rest/api/3/search</code>. Check your client library version too, not just your own code.</li>

<li><strong>Cycle times suspiciously low.</strong> Truncated changelog. Move to the dedicated changelog endpoint or bulk fetch.</li>

<li><strong>Athena returns zero rows but the files are there.</strong> Partition projection range does not cover the partition, or the S3 prefix does not match <code>storage.location.template</code>. Compare a real object key against the template character by character.</li>

<li><strong>Queries slow down as the table grows.</strong> Either you have no projection and Glue lookups dominate, or you have accumulated small files.</li>

<li><strong>Authentication failures overnight with no deploy.</strong> API token expired or was revoked. Alert on the specific status code rather than on &#8220;run failed&#8221;.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Appending incremental extracts without a deduplication strategy.</li>

<li>Building on current issue state and adding changelog later. Retrofitting history is far more work than including it from the start.</li>

<li>Hardcoding custom field IDs copied from documentation.</li>

<li>Storing the API token in a Lambda environment variable instead of Secrets Manager.</li>

<li>Running an hourly schedule when the dashboard is read once a day. You pay in small files and API quota for freshness nobody uses.</li>

<li>Letting BI tools query base tables directly instead of curated views.</li>

<li>Never reconciling against Jira. A scheduled check comparing approximate count to your row count catches drift within a day.</li>
</ul>



<h2 class="wp-block-heading">Best practices for a Jira analytics pipeline that survives</h2>



<ul class="wp-block-list">
<li>Land raw API responses to S3 before transforming. When your parsing is wrong, and it will be, you replay from raw rather than re-hammering the API.</li>

<li>Make the load idempotent. Re-running for the same date should produce the same result, not a second copy.</li>

<li>Define the pipeline in Terraform or CloudFormation and deploy it from CI. The IAM policy for Lambda and Athena is fiddly and you do not want to rebuild it from memory.</li>

<li>Alert on a run that succeeds with zero rows, not just on runs that error. Silent empty loads are the more common failure.</li>

<li>Scope the Jira token to a service account with read access to exactly the projects you need.</li>

<li>Put an S3 lifecycle policy on the raw zone. It grows faster than you expect and nobody queries last year&#8217;s raw JSON.</li>

<li>Expose curated views, not tables, to Grafana, Metabase, Power BI or QuickSight, and enable Athena&#8217;s query result reuse for repeated dashboard loads.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Should I use Lambda or Glue for Jira extraction?</h3>



<p class="wp-block-paragraph">Lambda, for an API-driven pull. The work is mostly waiting on HTTP responses, which Spark&#8217;s distributed compute does nothing for, and you would be paying for a Glue job&#8217;s minimum billing on something that is idle. Glue earns its place downstream, if you have heavy joins across several sources. Athena CTAS often covers that too.</p>



<h3 class="wp-block-heading">How often should the pipeline run?</h3>



<p class="wp-block-paragraph">Match the decision cadence, not the data cadence. Delivery metrics are reviewed weekly or in sprint ceremonies, so daily is almost always enough and produces far better file sizes. Reserve hourly for something with a real-time consumer, and know that you are buying that freshness with small-file overhead and API quota.</p>



<h3 class="wp-block-heading">Can I use webhooks instead of polling?</h3>



<p class="wp-block-paragraph">You can, and for near-real-time reaction it is the right tool. For analytics it is a poor primary source, because a missed delivery leaves a permanent hole you have no way to detect. The pattern that works is webhooks for freshness plus a scheduled reconciliation pull as the source of truth. If you only build one, build the scheduled pull.</p>



<h3 class="wp-block-heading">Does this work with Jira Data Center or Server?</h3>



<p class="wp-block-paragraph">The AWS half is identical. The extraction half is not. Cursor pagination and the <code>/search/jql</code> endpoint are Cloud-only changes, so self-hosted instances still use the older offset-based <code>/rest/api/2/search</code>. Write the extractor behind an interface if you need to support both, and expect the auth model to differ as well.</p>



<h3 class="wp-block-heading">How much does an Athena-based Jira pipeline cost to run?</h3>



<p class="wp-block-paragraph">The mechanism matters more than any figure I could quote. You pay for S3 storage, Lambda invocation time, and Athena per byte scanned with a small per-query minimum. Jira issue data compresses extremely well as Parquet, so storage stays small. The variable is dashboard query volume, which is why partitioning and workgroup scan limits matter. Model it against current AWS rates and your own expected query count.</p>



<h3 class="wp-block-heading">Why not just use eazyBI or a Jira reporting app?</h3>



<p class="wp-block-paragraph">Often you should. If your questions stay inside Jira, a Marketplace app gets you there in an afternoon with no infrastructure. The case for a pipeline is joining Jira to data that lives elsewhere, retaining history the app does not, or needing your data in a warehouse you already run. Build the pipeline when the app has actually failed you, not in anticipation.</p>



<h3 class="wp-block-heading">Should I load into Redshift instead of querying S3?</h3>



<p class="wp-block-paragraph">Only if you have concurrent BI users hitting the same tables constantly and Athena&#8217;s per-query latency is a real complaint. Jira data volumes are small, the query pattern is bursty, and Athena&#8217;s zero-idle-cost model fits that far better. Redshift makes sense as a consolidation layer across many sources, not for one issue tracker.</p>



<h2 class="wp-block-heading">The one thing to carry away</h2>



<p class="wp-block-paragraph">The hard part of a Jira analytics pipeline is not the AWS wiring. Lambda, S3, Glue and Athena will be working within a day. The hard part is that Jira issues are mutable, so an incremental load that appends is an incremental load that duplicates, and it does so without a single error in your logs.</p>



<p class="wp-block-paragraph">Decide your deduplication strategy before you write the first extract, not after someone reconciles a dashboard against JQL and finds a gap. Snapshot with a read-time dedupe view, partition projection so queries stay fast as history accumulates, and a scheduled reconciliation check that alerts on drift. Get those three right and the rest is plumbing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need this built or fixed?</h2>



<p class="wp-block-paragraph">I design and run serverless data pipelines on AWS, and Jira extraction has more sharp edges than most sources. Things I can help with directly:</p>



<ul class="wp-block-list">
<li>Migrating an extractor off the removed <code>/rest/api/3/search</code> endpoint to cursor-based <code>/search/jql</code> without losing rows in the cutover</li>

<li>Auditing an existing Jira pipeline for duplicate rows and reconciling your numbers back against JQL</li>

<li>Building changelog-based cycle time and time-in-status models that survive workflow renames</li>

<li>Restructuring an S3 layer with Parquet, partition projection and file compaction to cut Athena scan costs</li>

<li>Packaging the whole thing as Terraform or CloudFormation with CI deployment and least-privilege IAM</li>

<li>Connecting Athena to Grafana, Power BI, Metabase or QuickSight with curated views instead of raw tables</li>
</ul>



<p class="wp-block-paragraph">Send me your extractor code, a Glue table definition, or an Athena query that is scanning more than it should, and I will tell you what I would change.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/jira-analytics-pipeline-aws-lambda-athena/">Building a Jira Analytics Pipeline with AWS Lambda and Athena (Without Double-Counting Everything)</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Redshift Cost Optimization for SaaS Analytics: The Levers That Actually Move the Bill</title>
		<link>https://john-nessime.com/blog/devops/redshift-cost-optimization-saas-analytics/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 04 Aug 2026 09:18:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Business Intelligence]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Warehouse]]></category>
		<category><![CDATA[Embedded Analytics]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Multi-Tenant]]></category>
		<category><![CDATA[Row-Level Security]]></category>
		<category><![CDATA[SQL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=123</guid>

					<description><![CDATA[<p>In a SaaS analytics product, the Redshift bill tracks how often queries arrive, not how much data they touch. Here is how the meter actually works, why connection pools bill you while nobody is using the product, how to attribute spend to a tenant, and which isolation choices quietly cost more than they save.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/redshift-cost-optimization-saas-analytics/">Redshift Cost Optimization for SaaS Analytics: The Levers That Actually Move the Bill</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually comes from whoever owns the AWS bill, and it is never dramatic. &#8220;Redshift is up again this month. Did we onboard someone big?&#8221; Nobody onboarded anyone. Nobody shipped a new dashboard. Query volume looks flat on the Grafana board. The bill moved anyway.</p>



<p class="wp-block-paragraph">That gap between what you think you are paying for and what you are actually paying for is what makes Redshift cost optimization awkward in a SaaS analytics product. You are not running one nightly batch against a warehouse that sleeps the rest of the day. You are serving hundreds of small, latency-sensitive queries that fire whenever a customer opens a dashboard, plus ingestion, plus whatever your BI layer and your connection pool are doing when nobody is watching.</p>



<p class="wp-block-paragraph">This post covers the levers that genuinely move that number: how the meter works, why idle-looking connections still bill, how to work out which tenant is expensive, and which isolation choices cost more than they save. Where the popular advice is wrong for SaaS specifically, I will say so.</p>



<h2 class="wp-block-heading">How Amazon Redshift actually charges you</h2>



<p class="wp-block-paragraph">Three buckets, and they behave very differently.</p>



<ul class="wp-block-list"><li><strong>Compute.</strong> On Redshift Serverless this is RPU-hours, metered per second. On provisioned clusters it is node-hours, plus separate line items for concurrency scaling and Spectrum.</li><li><strong>Storage.</strong> Redshift Managed Storage, billed by GB per month, independent of compute. Snapshots are storage too.</li><li><strong>Everything else.</strong> Cross-region data sharing and snapshot replication, machine learning, data transfer outside the usual in-region S3 paths.</li></ul>



<p class="wp-block-paragraph">In a SaaS analytics workload compute dominates, often overwhelmingly. And the important part: compute is a function of how long the warehouse is awake and at what capacity, not how many rows you touched. Two teams can scan identical data volumes and get bills that differ by a factor of five, purely because of how their queries arrive.</p>



<h2 class="wp-block-heading">The billing mechanic that catches SaaS teams out</h2>



<p class="wp-block-paragraph">Read the serverless billing notes properly once and a lot of mysterious spend stops being mysterious. The parts that matter:</p>



<ul class="wp-block-list"><li>The minimum charge is 60 seconds of resource usage, metered per second beyond that. This is a minimum for the warehouse, not for each individual query.</li><li>Usage is recorded when a transaction <em>completes</em>, rolls back, or is stopped. A transaction that runs for hours shows up in your usage view only at the end.</li><li>Cancel a query before it finishes and you still pay for the time it ran.</li><li>Querying system tables is billed like any other query. Your monitoring loop is a workload.</li><li>After a burst, capacity can stay elevated for a period after the load drops. Scale-down is not instant.</li></ul>



<p class="wp-block-paragraph">Put those together and you reach a conclusion that irritates most engineers: on serverless, ten small queries crammed into one minute are cheaper than the same ten queries spread across ten minutes. Every wake-up costs you a minimum billing window multiplied by your base capacity. That is the opposite of the instinct you have from tuning an OLTP service, where you smooth load out to protect tail latency.</p>



<p class="wp-block-paragraph">Before you change anything, get the real numbers out of the warehouse rather than out of Cost Explorer, which lags and aggregates.</p>



<pre class="wp-block-code"><code>-- Daily billed RPU-seconds converted to RPU-hours.
-- Multiply by your region's on-demand RPU-hour rate for dollars.
SELECT trunc(start_time) AS day,
       sum(charged_seconds) / 3600::double precision AS rpu_hours
FROM   sys_serverless_usage
GROUP  BY 1
ORDER  BY 1 DESC;</code></pre>



<p class="wp-block-paragraph"><code>charged_seconds</code> is the column to build cost reporting on. <code>compute_seconds</code> is informative but it is not what the invoice is derived from, and the two can disagree within a given interval. Two constraints worth knowing before you wire this into a dashboard: the view holds roughly a week of history, and it is visible only to superusers. If you want month-over-month trends, UNLOAD it to S3 on a schedule and query the archive with Amazon Athena instead.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Lever one: connections that look idle and are not</h2>



<p class="wp-block-paragraph">This is the one that bites hardest and shows up last, because there is nothing to see. AWS documents it plainly: Redshift Serverless treats all incoming queries as billable user activity, including lightweight health-check queries sent by connection pools. It does not matter whether the statement came from your application, a JDBC driver, or a pooling framework doing its job.</p>



<p class="wp-block-paragraph">So a pool that fires <code>SELECT 1</code> every thirty seconds to validate connections is a warehouse that never gets to sleep. Your product has no users at 3am and you are still paying the minimum window, over and over, multiplied by base capacity. HikariCP, Apache Commons DBCP and PgBouncer all have some form of this behaviour, and the defaults are tuned for OLTP databases where a validation query costs nothing.</p>



<p class="wp-block-paragraph">Open transactions are the same problem wearing a different hat. A <code>BEGIN</code> without a matching <code>COMMIT</code> or <code>ROLLBACK</code> keeps consuming RPUs until the session ends. Session timeouts exist precisely because this happens.</p>



<p class="wp-block-paragraph">What I would check, in this order:</p>



<ol class="wp-block-list"><li>Disable the pool&#8217;s validation or heartbeat query entirely if the driver allows it. If it does not, stretch the interval as far as your failure tolerance permits.</li><li>Drop idle pool size to something honest. A pool sized for peak that stays warm overnight is pure waste on this pricing model.</li><li>Fix any code path that opens a transaction and returns early on error without ending it.</li><li>Set a session timeout per application role so a leaked connection cannot bill indefinitely.</li></ol>



<pre class="wp-block-code"><code>-- Cap idle sessions for the application role.
-- Value is in seconds; the documented range is 60 to 1,728,000.
ALTER USER analytics_app SESSION TIMEOUT 1800;

-- Cap how many connections a single role can hold open at once.
ALTER USER analytics_app CONNECTION LIMIT 40;

-- What is connected right now, and with what timeout.
SELECT * FROM stv_sessions;</code></pre>



<p class="wp-block-paragraph">Session timeout changes apply to new sessions only, so recycle the pool afterwards or you will conclude the setting does nothing.</p>



<h2 class="wp-block-heading">Lever two: base capacity, max capacity and usage limits are three different things</h2>



<p class="wp-block-paragraph">These get conflated constantly, and two of them will not save you a cent on their own.</p>



<ul class="wp-block-list"><li><strong>Base capacity (base RPU).</strong> The floor. It multiplies every billed second, including that 60-second minimum. Halving base capacity roughly halves the cost of a warehouse dominated by short queries. It also halves the compute those queries get, so watch p95 latency alongside the bill.</li><li><strong>Max capacity (MaxRPU).</strong> A ceiling on how far automatic scaling can go. It caps compute available to the workgroup, it does not stop queries and it does not interrupt anything running. Useful as a guard rail against a runaway scan, useless as a budget.</li><li><strong>Usage limits.</strong> An actual budget, expressed in RPU-hours over a daily, weekly or monthly period. The breach actions are: log to a system table, raise an SNS alert, or turn off user queries.</li></ul>



<p class="wp-block-paragraph">Only the third one can stop you spending money, and only the third one can take your product down at 2pm on a Tuesday. Set it to alert first, live with it for a full billing cycle so you learn the shape of a normal week, then decide whether you are genuinely willing to have queries turned off. In a customer-facing SaaS product the answer is usually no, and the limit stays as an alarm feeding PagerDuty or whatever you already page from.</p>



<p class="wp-block-paragraph">There is also the price-performance target, the slider that hands scaling decisions to AWS in exchange for a stated cost or speed preference. AWS recommends it for mid-range base capacities and advises against it at the very bottom and very top of the RPU scale, so check the current guidance against your base setting before enabling it. It is worth trying on a staging workgroup with a replayed query mix; it is not worth switching on blind in production.</p>



<p class="wp-block-paragraph">On provisioned clusters the equivalent controls are per-feature usage limits: concurrency scaling measured in time, Spectrum measured in data scanned, cross-region data sharing, and extra compute for automatic optimization. Each takes a breach action of log, emit a metric, or disable the feature. Concurrency scaling also earns free credits as the main cluster runs, which is why a moderately bursty provisioned cluster often shows no concurrency scaling charge at all until it suddenly does.</p>



<h2 class="wp-block-heading">Lever three: Redshift cost optimization starts with knowing which tenant is expensive</h2>



<p class="wp-block-paragraph">Be clear-eyed about what is possible here. On serverless you cannot get an exact dollar figure per query, because billing happens at the warehouse level and the minimum charge is shared across whatever else was running in that window. What you can build is a defensible apportionment, and that is enough to find the customer whose scheduled export is quietly eating your margin.</p>



<p class="wp-block-paragraph">Start by labelling every statement your API issues on a tenant&#8217;s behalf.</p>



<pre class="wp-block-code"><code>-- Set in the pool's per-checkout init SQL, or per request.
SET query_group TO 'tenant_4417';

SELECT metric_date, sum(events)
FROM   fact_events
WHERE  tenant_id = 4417
  AND  metric_date &gt;= dateadd(day, -30, current_date)
GROUP  BY 1;

RESET query_group;</code></pre>



<p class="wp-block-paragraph">The label lands in the query log and surfaces as <code>query_label</code> in the SYS monitoring views. Keep it short: the older query log views truncate the label to 30 characters, so a tenant slug beats a UUID with prefixes bolted on.</p>



<pre class="wp-block-code"><code>-- Seven days of activity grouped by tenant label.
-- Note: time columns in the SYS views are microseconds;
-- confirm units before converting anything to money.
SELECT trim(query_label)   AS tenant,
       count(*)            AS queries,
       sum(execution_time) AS exec_time,
       sum(queue_time)     AS queue_time
FROM   sys_query_history
WHERE  start_time &gt; dateadd(day, -7, sysdate)
  AND  query_label LIKE 'tenant_%'
GROUP  BY 1
ORDER  BY exec_time DESC;</code></pre>



<p class="wp-block-paragraph">Three columns in that view earn their keep beyond the obvious ones. <code>result_cache_hit</code> tells you which dashboard queries are already free, which is often a bigger share than people expect. The split between <code>queue_time</code> and <code>execution_time</code> tells you whether you have a tuning problem or a capacity problem, and those have opposite fixes. And <code>user_query_hash</code> groups repeated queries with different literals, which is exactly what an embedded dashboard produces, so it is the fastest way to find the one panel that fifty tenants are running badly.</p>



<p class="wp-block-paragraph">From there, apportion the day&#8217;s <code>charged_seconds</code> by each tenant&#8217;s share of execution time. It is an approximation and you should label it as one when you show it to finance. It is still the difference between &#8220;Redshift costs us a lot&#8221; and &#8220;eleven percent of our warehouse spend is one customer pulling an unbounded date range every fifteen minutes.&#8221;</p>



<h2 class="wp-block-heading">Lever four: the isolation model you picked is a cost decision</h2>



<p class="wp-block-paragraph">AWS&#8217;s SaaS guidance describes three partitioning models, and each one has a distinct cost signature on Redshift.</p>



<ul class="wp-block-list"><li><strong>Pool.</strong> All tenants share tables with a tenant identifier column. Cheapest by a wide margin, one warehouse to keep warm, one set of statistics. You pay for it in noisy-neighbour risk and in the access-control work you now have to do yourself.</li><li><strong>Bridge.</strong> Separate schemas or databases inside one cluster. Sounds like a compromise, behaves like neither. AWS&#8217;s own whitepaper is fairly blunt that the isolation profile does not usually justify it, since cluster-level access grants reach across the databases anyway.</li><li><strong>Silo.</strong> A warehouse per tenant. Clean boundaries and per-tenant cost visibility for free. On serverless it is also the most expensive thing you can do, because every workgroup carries its own base capacity floor and its own 60-second minimums. Twenty small tenants means twenty warehouses waking up independently.</li></ul>



<p class="wp-block-paragraph">Data sharing sits between these and is the pattern I reach for when workload interference is the real problem. One producer handles ingestion and transformation; consumers read the shared data without copying it, and a consumer&#8217;s load does not touch the producer. Genuinely useful for separating a heavy ETL window from customer-facing reads. But be honest about the arithmetic: every consumer is its own billable warehouse. Data sharing buys you performance isolation, not cheaper compute.</p>



<p class="wp-block-paragraph">In a pooled model, the thing I set up first is a sort key that leads with the tenant identifier followed by the time column everyone filters on. That lets Redshift prune blocks before it reads them instead of scanning broadly and filtering afterwards. Combine it with row-level security so the tenant predicate cannot be forgotten by an application bug, and you have removed both the largest cost driver and the scariest failure mode in one change.</p>



<h2 class="wp-block-heading">Lever five: scan less, refresh less</h2>



<p class="wp-block-paragraph">Classic warehouse hygiene still applies, it just pays differently here. Shorter queries mean fewer billed seconds at your base capacity.</p>



<ul class="wp-block-list"><li><strong>Sort keys that match your real predicates.</strong> Not the ones from the design doc. Pull the top twenty query hashes and read their WHERE clauses.</li><li><strong>Materialized views for the panels every tenant loads.</strong> Real savings on the read path, but refresh is compute you pay for. A view refreshed every five minutes and read twice an hour is a net loss.</li><li><strong>Let the result cache work.</strong> Identical query text against unchanged data is free. Anything your BI layer does that injects a timestamp or a random parameter into otherwise identical SQL is throwing that away. Worth checking in Amazon QuickSight, Metabase or whatever sits in front.</li><li><strong>Tune zero-ETL refresh intervals.</strong> The refresh interval on the target database is adjustable via <code>ALTER DATABASE</code>. Shorter is fresher and more expensive. For reporting and historical analysis, a longer interval is usually the right call and nobody notices.</li><li><strong>Keep cold history out of managed storage.</strong> Partitioned Parquet or Apache Iceberg tables in S3, catalogued in AWS Glue, queried through the lake. On serverless those queries bill at the same RPU rate rather than as a separate Spectrum line, so the win is in scan efficiency and storage cost, not in dodging a charge.</li></ul>



<p class="wp-block-paragraph">One reassuring detail: the automatic optimization work Redshift does in the background is not billed by default. It becomes billable only if you explicitly enable extra compute resources so those operations can run during busy periods. That is a deliberate trade, not an accident, and it is worth knowing before you turn it on.</p>



<h2 class="wp-block-heading">Provisioned or serverless: how I would decide</h2>



<p class="wp-block-paragraph">Both have a genuine case and the honest answer depends on the shape of your load, not on which is newer.</p>



<p class="wp-block-paragraph">Serverless wins when demand is spiky or concentrated in business hours, when you cannot forecast capacity, and for dev and test environments that sit idle most of the week. It also folds concurrency scaling and data-lake queries into a single rate, which removes two line items people routinely forget to model.</p>



<p class="wp-block-paragraph">Provisioned RA3 wins when load is steady around the clock, because a reserved commitment on nodes can beat accumulated on-demand RPU-hours, and because you get the full workload management surface: queues, query priority, query monitoring rules with the complete set of controls. If you need to guarantee that a tenant&#8217;s export can never starve the interactive path, that machinery is more expressive than a price-performance slider.</p>



<p class="wp-block-paragraph">Commitment discounts now exist on both sides, including reservations for serverless managed at the payer account level. Rates and terms change, so price it against your own numbers rather than a blog post.</p>



<p class="wp-block-paragraph">The tell is simple. Pull a week of <code>charged_seconds</code> bucketed by hour and plot it. A flat line means you are paying serverless rates for provisioned behaviour. A sawtooth with long dead zones means the opposite.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting: the bill moved and nothing shipped</h2>



<ol class="wp-block-list"><li><strong>Get hourly billed seconds first.</strong> Aggregate <code>charged_seconds</code> by hour from the usage view. If the increase is spread evenly across all 24 hours, it is background activity: a pool, a monitor, a health check. If it is concentrated, it is a workload.</li><li><strong>Check for anything running or queued right now.</strong> A single stuck statement explains a lot of otherwise inexplicable spend.</li><li><strong>Look for transactions that never ended.</strong> A deploy that changed error handling can leave transactions open without a single failed request in your logs.</li><li><strong>Compare query counts against query cost.</strong> Flat count with rising cost points at base capacity changes, scale-down lag, or data growth making the same queries slower.</li><li><strong>Group by <code>user_query_hash</code> and diff against last week.</strong> New shapes appearing means a shipped change. Old shapes getting slower means data or statistics.</li><li><strong>Only then look at storage.</strong> Managed storage grows quietly and it is rarely the cause of a sudden jump, but it is often the cause of a slow one.</li></ol>



<pre class="wp-block-code"><code>-- Anything currently running or waiting.
SELECT user_id, query_id, transaction_id, session_id, status,
       trim(database_name) AS database_name,
       start_time, queue_time, execution_time
FROM   sys_query_history
WHERE  status IN ('running','queued')
ORDER  BY start_time;</code></pre>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list"><li>Treating max capacity as a spending cap. It caps compute, not cost, and it will not stop a workload that simply runs for a long time.</li><li>Optimising individual slow queries while ignoring a connection pool that wakes the warehouse every thirty seconds all night.</li><li>Smoothing scheduled jobs out across the hour to be gentle on the warehouse. On serverless this is backwards; batching into fewer windows costs less.</li><li>Building cost dashboards on <code>compute_seconds</code> instead of <code>charged_seconds</code>, then wondering why the totals never reconcile with the invoice.</li><li>Giving every tenant their own workgroup for isolation, then discovering that base capacity floors and minimum charges multiply by tenant count.</li><li>Setting a usage limit to &#8220;turn off user queries&#8221; on the first day, before anyone knows what a normal week looks like.</li><li>Leaving the monitoring loop itself unbounded. Polling system views every few seconds is a workload that bills like any other.</li></ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list"><li>Label every tenant-originated query with <code>query_group</code> from day one. Retrofitting attribution is far more painful than adding a SET statement to your pool&#8217;s init SQL.</li><li>UNLOAD the serverless usage view to S3 on a schedule. Seven days of retention is not enough to argue about a monthly invoice.</li><li>Keep at least one usage limit configured as an alert, permanently, even if you never set a hard cap.</li><li>Review base capacity quarterly against p95 latency, not just against cost. The right number moves as your workload changes.</li><li>Put a hard date bound on every customer-facing query in the application layer. Unbounded ranges are the single most common source of surprise spend in embedded analytics.</li><li>Model concurrency scaling and data-lake charges explicitly if you are on provisioned. They are the line items people forget until they appear.</li><li>Tag workgroups and clusters consistently so cost tooling, whether that is AWS Cost Explorer or something like CloudZero or Vantage, can split spend by environment without guesswork.</li></ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Does Redshift Serverless really charge me when nobody is using the product?</h3>



<p class="wp-block-paragraph">Idle time itself is not billed, but anything that sends a query is. AWS states explicitly that health-check queries from connection pools count as billable user activity. If your pool validates connections on a timer overnight, you are paying minimum billing windows all night. Check the pool before you conclude the pricing model is broken.</p>



<h3 class="wp-block-heading">How do I calculate the cost of a single query?</h3>



<p class="wp-block-paragraph">You cannot, exactly. Serverless bills the warehouse, and the 60-second minimum is shared with whatever else ran in that window. The workable approach is apportionment: take <code>charged_seconds</code> for a period and divide it by each labelled tenant&#8217;s share of execution time from the query history view. Useful for finding outliers, not precise enough for per-customer invoicing.</p>



<h3 class="wp-block-heading">Should I lower base capacity to save money?</h3>



<p class="wp-block-paragraph">Often yes, and it is the single highest-leverage change for a workload made of many short queries, because base capacity multiplies every billed second including the minimum. The catch is that it also reduces the compute each query gets. Change it in one step, watch p95 latency and queue time together for a full week, then decide whether to go further.</p>



<h3 class="wp-block-heading">Is a warehouse per tenant a good idea?</h3>



<p class="wp-block-paragraph">Only when tenants are large enough to keep a warehouse genuinely busy, or when a contract requires that level of separation. For a long tail of small tenants it is the most expensive option available, since each warehouse carries its own capacity floor and its own minimum charges. Pooled tables with row-level security and a tenant-leading sort key gets you most of the isolation for a fraction of the compute.</p>



<h3 class="wp-block-heading">Does concurrency scaling cost extra?</h3>



<p class="wp-block-paragraph">On Redshift Serverless, no, scaling is included in the RPU rate. On provisioned clusters it is a separate charge, offset by credits that accrue while the main cluster runs. That difference catches out teams migrating between the two, in both directions.</p>



<h3 class="wp-block-heading">Will a usage limit take my product down?</h3>



<p class="wp-block-paragraph">It will if you configure the breach action to turn off user queries. The logging and alerting actions are safe and are what you want in a customer-facing system. Treat the hard stop as a deliberate business decision about which is worse, an unexpected invoice or an outage, rather than as a default setting.</p>



<h3 class="wp-block-heading">Why does my cost report never match the AWS invoice?</h3>



<p class="wp-block-paragraph">Usually one of three things: using <code>compute_seconds</code> rather than <code>charged_seconds</code>, forgetting that usage is recorded only when a transaction completes so long transactions land in a later interval, or leaving storage and cross-region transfer out of the model entirely.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Redshift cost optimization for a SaaS analytics product is mostly not a query tuning exercise. It is a question of how often something wakes the warehouse up and at what capacity. Query tuning matters, sort keys matter, materialized views matter, but a connection pool with default settings will quietly outspend all of them combined.</p>



<p class="wp-block-paragraph">So start at the meter. Pull hourly billed seconds, look at the overnight hours when your product has no users, and see whether the line goes to zero. If it does not, you have found your first and cheapest win before touching a single line of SQL.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help getting your Redshift bill under control?</h2>



<p class="wp-block-paragraph">I work with SaaS and data teams on exactly this problem, usually somewhere between the warehouse and the application that is hammering it. Things I can help with:</p>



<ul class="wp-block-list"><li>Auditing an existing Redshift Serverless or RA3 workload and producing a ranked list of what is actually driving spend, with the numbers pulled from your own system views.</li><li>Building per-tenant cost attribution: query labelling, a usage archive in S3, and a dashboard your product and finance teams can both read.</li><li>Fixing the connection and session layer, including pool configuration, validation queries, session timeouts and transaction hygiene.</li><li>Right-sizing base and max capacity against measured latency, and setting usage limits and alerts that warn without risking an outage.</li><li>Reviewing multi-tenant data models: sort and distribution keys, row-level security, and whether data sharing or a pooled model fits your tenant mix.</li><li>Deciding between provisioned and serverless with a workload profile behind the recommendation rather than a rule of thumb.</li></ul>



<p class="wp-block-paragraph">If you have a week of usage data, a suspicious hourly cost chart, or a pool configuration you are not sure about, send it over and I will tell you what I see in it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/redshift-cost-optimization-saas-analytics/">Redshift Cost Optimization for SaaS Analytics: The Levers That Actually Move the Bill</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Embedded Analytics on AWS: The Four Decisions That Bite Later</title>
		<link>https://john-nessime.com/blog/devops/embedded-analytics-on-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 04 Aug 2026 06:07:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Business Intelligence]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Warehouse]]></category>
		<category><![CDATA[Embedded Analytics]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[Multi-Tenant]]></category>
		<category><![CDATA[QuickSight]]></category>
		<category><![CDATA[Row-Level Security]]></category>
		<category><![CDATA[SPICE]]></category>
		<category><![CDATA[SQL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=120</guid>

					<description><![CDATA[<p>Rendering a dashboard inside your app is the easy part. Tenant isolation, session cost and query mode are what break. A practical walkthrough of the four decisions behind embedded analytics on AWS, the API constraints that lock you in, and the errors you will actually see.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/embedded-analytics-on-aws/">Embedded Analytics on AWS: The Four Decisions That Bite Later</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Getting a dashboard to render inside your own application is the easy part. You publish it, call the embed API, drop the iframe in, and it shows up. The hard question arrives about a day later, usually from someone in security or from the first customer who logs in: how exactly does tenant B not see tenant A&#8217;s rows?</p>



<p class="wp-block-paragraph">That is where embedded analytics on AWS stops being a front-end task and becomes an architecture decision. The awkward part is that the choice you make first, how the viewer is identified, quietly decides which isolation mechanisms remain available to you afterwards. Get that order backwards and you rebuild the data layer, not the iframe.</p>



<p class="wp-block-paragraph">This post covers the four decisions that determine whether the build holds: identity model, tenant isolation, query mode, and session economics. Then the embed handshake itself, the errors you will actually see in the browser console, and what to check first when it fails.</p>



<h2 class="wp-block-heading">Before anything else: the product got renamed</h2>



<p class="wp-block-paragraph">Amazon QuickSight was folded into a broader platform called Amazon Quick Suite, and the BI product inside it is now called Amazon Quick Sight. AWS documentation has since moved again under an &#8220;Amazon Quick&#8221; umbrella. You will land on all three naming conventions depending on which search result you click, which makes finding the right doc page genuinely annoying.</p>



<p class="wp-block-paragraph">The practical upshot: the APIs, SDKs and IAM action names did not change. You are still calling <code>quicksight:GenerateEmbedUrlForRegisteredUser</code> against ARNs in the <code>quicksight</code> namespace, and the JavaScript SDK is still published as <code>amazon-quicksight-embedding-sdk</code>. Nothing in your code breaks. Only your bookmarks do. I mention it because half the confusion in a first embedded build comes from following a doc page that describes a UI menu that has since been reorganised.</p>



<h2 class="wp-block-heading">Decision one: registered users or anonymous sessions</h2>



<p class="wp-block-paragraph">Two API operations generate embed URLs. <code>GenerateEmbedUrlForRegisteredUser</code> issues a session for a user who exists inside the BI account. <code>GenerateEmbedUrlForAnonymousUser</code> issues a session for someone who does not, and never will.</p>



<p class="wp-block-paragraph">This reads like a convenience choice. It is not. Row-level security using session tags, the mechanism most SaaS products want, is supported <em>only</em> for anonymous embedding. It does not work with <code>GenerateEmbedUrlForRegisteredUser</code>, it does not work with the older <code>GetDashboardEmbedUrl</code> operation, and it is not supported with the IAM identity type. That constraint is documented, easy to miss, and it is the single most expensive thing to discover late.</p>



<p class="wp-block-paragraph">So the fork is really this. If you register every viewer, you get per-user features (bookmarks, threshold alerts, scheduled snapshots) and you enforce isolation with username or group rules on the dataset. You also inherit the job of provisioning, deprovisioning and reconciling a user directory that mirrors your own. If you go anonymous, you skip all of that and filter with session tags at embed time, but per-user features are off the table because there is no persistent user to hang them on.</p>



<p class="wp-block-paragraph">The registered-user request body is small. Everything interesting is in <code>ExperienceConfiguration</code>:</p>



<pre class="wp-block-code"><code>POST /accounts/&lt;aws-account-id&gt;/embed-url/registered-user

{
  "UserArn": "arn:aws:quicksight:&lt;region&gt;:&lt;account&gt;:user/default/&lt;user&gt;",
  "SessionLifetimeInMinutes": 60,
  "AllowedDomains": ["https://app.example.com"],
  "ExperienceConfiguration": {
    "Dashboard": {
      "InitialDashboardId": "&lt;dashboard-id&gt;",
      "FeatureConfigurations": {
        "Bookmarks": { "Enabled": true }
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">One trap on the anonymous path that deserves its own sentence. Anonymous sessions belong to a namespace, and any dashboard shared with that namespace is reachable by a session in it, whether or not you listed the dashboard in <code>AuthorizedResourceArns</code>. If you were treating that parameter as your allowlist, it is not. Namespace membership is the real boundary.</p>



<h2 class="wp-block-heading">Decision two: where tenant isolation actually lives</h2>



<p class="wp-block-paragraph">There are three places you can put the filter, and only one of them scales.</p>



<ul class="wp-block-list">
<li><strong>A dashboard per tenant.</strong> Works for five customers. Becomes a deployment problem at fifty and a change-management disaster at five hundred, because every visual fix is now a fan-out.</li>



<li><strong>A dataset per tenant, filtered in SQL.</strong> Better isolation guarantees, genuinely defensible in a compliance review, but you multiply refresh jobs and in-memory footprint by tenant count.</li>



<li><strong>One dashboard, one dataset, row-level security.</strong> The standard answer. One artifact to maintain, filtering applied per session.</li>
</ul>



<p class="wp-block-paragraph">With anonymous embedding, RLS is driven by tags. You declare tag keys against columns on the dataset, then supply values at embed time. The filter is evaluated server-side against the session, so a viewer poking at the iframe cannot lift it.</p>



<pre class="wp-block-code"><code>POST /accounts/&lt;aws-account-id&gt;/embed-url/anonymous-user

{
  "Namespace": "default",
  "SessionLifetimeInMinutes": 60,
  "AuthorizedResourceArns": [
    "arn:aws:quicksight:&lt;region&gt;:&lt;account&gt;:dashboard/&lt;dashboard-id&gt;"
  ],
  "SessionTags": [
    { "Key": "tenant_id", "Value": "acme-corp" },
    { "Key": "region",    "Value": "emea" }
  ],
  "AllowedDomains": ["https://app.example.com"],
  "ExperienceConfiguration": {
    "Dashboard": { "InitialDashboardId": "&lt;dashboard-id&gt;" }
  }
}</code></pre>



<p class="wp-block-paragraph">The value in <code>SessionTags</code> must come from your server-side session, never from a request parameter, a cookie your client can write, or a JWT claim you have not verified. This is the whole security boundary. Tag rules support combining conditions, so a manager who should see several sites is expressible without a second dashboard.</p>



<p class="wp-block-paragraph">One quiet limit worth knowing before it bites: when RLS is applied to in-memory datasets, each field has a maximum length in Unicode characters, and fields exceeding it are truncated during ingestion rather than rejected. If your tenant identifiers are long opaque strings, test that a truncated value cannot collide with another tenant&#8217;s. Silent truncation plus a prefix collision is exactly the kind of bug that produces a cross-tenant data leak with no error anywhere in the logs.</p>



<h2 class="wp-block-heading">Decision three: SPICE or direct query against your AWS data</h2>



<p class="wp-block-paragraph">Every dataset runs in one of two modes, and the difference shows up on a bill somewhere else in your account.</p>



<p class="wp-block-paragraph"><strong>Direct query</strong> sends a live query to the source each time a visual renders. Against Amazon Athena that means an S3 scan per dashboard open, billed by bytes scanned. Against Amazon Redshift it means a concurrent query slot per viewer. Freshness is perfect. The failure mode is that dashboard load is now coupled to warehouse load, and your analytics traffic competes with everything else running there. Two hundred people opening a dashboard at 9am is two hundred queries, and Redshift concurrency is finite.</p>



<p class="wp-block-paragraph"><strong>SPICE</strong> imports a snapshot into an in-memory engine and serves every viewer from it. One scan on refresh, then arbitrarily many reads. For an embedded product where the same aggregate is served to thousands of sessions, this is usually the right call, and the Athena cost difference between &#8220;scan once per refresh&#8221; and &#8220;scan once per pageview&#8221; is not subtle. What you give up is freshness, bounded by your refresh schedule, plus a capacity dimension to manage and incremental refresh to configure if the dataset is large.</p>



<p class="wp-block-paragraph">The pattern I reach for first on a data-lake backend is a hybrid: recent partitions in SPICE with an incremental refresh on a look-back window, historical data left on direct query for the rare deep query. It costs more design effort up front and it is the thing most teams skip, but it is the only shape that keeps both the bill and the load time flat as history grows.</p>



<p class="wp-block-paragraph">Whichever you pick, note that visual generation has a timeout, and data-source-specific timeouts apply on top of it. A query that is merely slow in a console tab renders as a broken visual in a customer&#8217;s browser. Model your worst partition, not your average one.</p>



<h2 class="wp-block-heading">Decision four: what a session actually costs</h2>



<p class="wp-block-paragraph">I am not going to quote figures, because AWS changes them and you should read the current pricing page. The mechanism is what matters, and it is genuinely different from seat-based BI licensing.</p>



<ul class="wp-block-list">
<li>A reader session is a fixed 30-minute window. Not a pageview, not a query. Reopening the dashboard twenty minutes later is still the same session.</li>



<li><strong>Per-user pricing</strong> charges per session with a monthly cap per reader. Predictable when the same people return daily.</li>



<li><strong>Capacity pricing</strong> buys sessions in bulk with no user provisioning at all. This is the model built for embedding, and it is the one that pairs with anonymous sessions.</li>



<li>Capacity pricing is also the prerequisite for programmatic dashboard refresh, so if near-real-time rendering is a product requirement, that decision is already made for you.</li>



<li>Annual commitments to capacity unlock removing the &#8220;Powered by&#8221; attribution footer. If white-labelling is a contractual requirement, factor that in early rather than discovering it during a customer demo.</li>



<li>Enabling certain Pro-tier and generative Q&amp;A capabilities triggers an account-level monthly infrastructure fee that exists whether or not anyone uses the feature.</li>
</ul>



<p class="wp-block-paragraph">The cost failure mode nobody plans for is architectural rather than commercial. If you embed the dashboard on a tab that loads by default, you bill a session for every user who lands on that page and looks at something else. Lazy-load the iframe on explicit interaction. That one change is often the largest single lever on the bill, and it costs an afternoon.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The embed handshake, and the three things that break it</h2>



<p class="wp-block-paragraph">The flow is short. Your backend authenticates the user with your own identity system, calls the embed URL API with the right tags or user ARN, returns the URL to the browser, and the SDK mounts an iframe against it.</p>



<ol class="wp-block-list">
<li>The generated URL carries a temporary bearer token valid for five minutes, and it is single use once redeemed. Generate it per page load from your backend. Never cache it, never put it in a build artifact, never log it.</li>



<li>Session lifetime is separate from URL validity, set with <code>SessionLifetimeInMinutes</code>, and ranges from fifteen minutes to ten hours with ten hours as the default. Ten hours is almost never what you want for a customer-facing product. Match it to your own session, or shorter.</li>



<li>Domains must be allowed explicitly. An administrator configures static domains in the admin menu, and <code>AllowedDomains</code> on the API call can override that with up to three domains or subdomains per request. Add an <code>AllowedEmbeddingDomains</code> condition to the IAM policy of the calling role, or any developer with that permission can list any domain on the internet.</li>
</ol>



<p class="wp-block-paragraph">On the browser side, the v2 SDK creates an embedding context (which appends its own zero-pixel iframe to <code>body</code> for message passing) and then mounts the experience:</p>



<pre class="wp-block-code"><code>import { createEmbeddingContext } from 'amazon-quicksight-embedding-sdk';

const context = await createEmbeddingContext();

await context.embedDashboard(
  {
    url: embedUrl,                        // fetched from your backend, just now
    container: '#analytics',
    height: '600px',                      // acts as loading height below
    resizeHeightOnSizeChangedEvent: true,
  },
  {
    toolbarOptions: { export: false, undoRedo: false, reset: false },
    attributionOptions: { overlayContent: true },
    onMessage: async (event) =&gt; {
      if (event.eventName === 'ERROR_OCCURRED') {
        console.error(event.message.errorCode);
      }
    },
  }
);</code></pre>



<p class="wp-block-paragraph">Two details in there earn their place. <code>resizeHeightOnSizeChangedEvent</code> turns the <code>height</code> value into a loading placeholder and lets the frame grow to fit content, which is what stops the dashboard rendering into a 600px letterbox with an inner scrollbar. And <code>overlayContent</code> tells the layout to overlay the attribution footer rather than reserve extra height at the bottom for it.</p>



<h2 class="wp-block-heading">Troubleshooting embedded analytics on AWS</h2>



<p class="wp-block-paragraph">Almost every failure lands in one of these. Read the error code out of the <code>ERROR_OCCURRED</code> message before doing anything else.</p>



<ul class="wp-block-list">
<li><strong><code>Forbidden</code></strong> means the URL&#8217;s authentication code expired. You held the URL longer than five minutes, or you served it from a cache, or a retry redeemed it twice. Fix the generation path, not the permissions.</li>



<li><strong><code>Unauthorized</code></strong> means the session obtained from that code expired. Different problem, different fix: your <code>SessionLifetimeInMinutes</code> is shorter than how long people keep the tab open. Handle it by re-fetching a fresh URL and re-mounting rather than letting the frame sit there dead.</li>



<li><strong>Frame never appears at all.</strong> Check the <code>onChange</code> handler for <code>NO_CONTAINER</code> or <code>INVALID_CONTAINER</code>, which usually means you mounted before your target element existed, and for <code>INVALID_URL</code>, which means the URL shape does not match the experience method you called.</li>



<li><strong>Frame appears, dashboard does not.</strong> Nine times out of ten this is the domain allowlist. The request domain has to match what was allowed, including scheme and any subdomain, and a staging hostname that nobody added is the usual culprit.</li>



<li><strong>Modals render off-screen.</strong> A known consequence of auto-resizing height: an export dialog can open above the visible viewport. Listen for <code>MODAL_OPENED</code> and scroll the parent page to the frame position.</li>



<li><strong>Toolbar features silently missing.</strong> Bookmarks, threshold alerts and scheduling require both the SDK toolbar flag and the matching entry under <code>FeatureConfigurations</code> in the embed URL request, and they only exist on the registered-user path. Setting the client flag alone does nothing.</li>



<li><strong>First render is slow, later ones are fine.</strong> Direct query against a cold warehouse. Compare the same query in Athena or Redshift directly to confirm before blaming the BI layer.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Choosing registered-user embedding for the identity story, then discovering session-tag RLS is unavailable on that path.</li>



<li>Treating <code>AuthorizedResourceArns</code> as the security boundary instead of namespace membership.</li>



<li>Deriving a session tag value from anything the client can influence.</li>



<li>Generating the embed URL at build time, or caching it in a CDN, and then not understanding the <code>Forbidden</code> errors.</li>



<li>Leaving session lifetime at the ten-hour default in a customer-facing app.</li>



<li>Putting the dashboard on a default-loaded tab and paying for sessions nobody asked for.</li>



<li>Building the first version on direct query against Athena because it is quicker to wire up, then meeting the scan bill.</li>



<li>Forgetting that embedding and row-level security sit in the Enterprise tier, so a Standard-tier proof of concept proves nothing.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Decide the identity model before you build a single dataset. Everything downstream inherits it.</li>



<li>Put the embed URL call behind one server-side endpoint that reads tenant scope from your own session and nowhere else. One function, one place to audit.</li>



<li>Constrain the calling IAM role with an <code>AllowedEmbeddingDomains</code> condition and scope resources to specific namespaces rather than a wildcard.</li>



<li>Write an automated test that requests tenant A&#8217;s embed URL and asserts tenant B&#8217;s rows are absent. Run it on every dataset change, because RLS breaks silently.</li>



<li>Default to SPICE with a refresh schedule matched to a stated freshness SLA, and only reach for direct query where the SLA genuinely demands it.</li>



<li>Track refresh failures as a first-class alert in CloudWatch or whatever you already run, whether that is Grafana, Datadog or something in-house. A stale dashboard that still renders is worse than one that errors, because nobody notices.</li>



<li>Lazy-load the iframe on user intent, not on page mount.</li>



<li>Keep the embedded surface read-only unless authoring is a real product requirement. Console embedding is a much larger permissions surface than dashboard embedding.</li>
</ul>



<h2 class="wp-block-heading">Is managed BI even the right call?</h2>



<p class="wp-block-paragraph">Worth asking honestly, because the answer is not always yes. The case for the AWS-native route is real: no connector layer to maintain against Athena, Redshift, S3 and Aurora, IAM you already understand, and a usage-based cost model that beats per-seat licensing when your viewers are bursty. If most of your data already sits in AWS, that adds up.</p>



<p class="wp-block-paragraph">The case against is equally real. Visual customisation is limited compared to charting directly against your own API, the attribution footer needs a commitment to remove, and if you want full control of the front end you may be better served by Apache Superset or Metabase self-hosted, or by Grafana where the workload is closer to operational metrics than customer-facing BI. Those come with an operational burden you now own. That is the trade: you either run the BI layer or you rent it, and renting it means living inside its constraints.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do my users need AWS accounts to view an embedded dashboard?</h3>



<p class="wp-block-paragraph">No. With anonymous embedding they need no AWS account and no BI user record at all. Your application authenticates them however you already do, and your backend maps that identity to session tags when it requests the embed URL.</p>



<h3 class="wp-block-heading">Can I use row-level security with registered-user embedding?</h3>



<p class="wp-block-paragraph">Yes, but only with username or group based rules, not with session tags. Tag-based RLS is restricted to the anonymous embedding path. If you need tags, you need anonymous sessions.</p>



<h3 class="wp-block-heading">How long does an embed URL stay valid?</h3>



<p class="wp-block-paragraph">The URL itself carries a bearer token valid for five minutes and usable once. The session it opens is separate and lasts between fifteen minutes and ten hours depending on <code>SessionLifetimeInMinutes</code>, defaulting to ten hours.</p>



<h3 class="wp-block-heading">Should I use SPICE or direct query for embedded analytics on AWS?</h3>



<p class="wp-block-paragraph">SPICE for anything with many viewers per refresh, which describes most embedded products. Direct query where the data must be current to the second, or where the dataset exceeds what you want to hold in memory. A hybrid split by data age is often the right answer and is under-used.</p>



<h3 class="wp-block-heading">Why do I get a Forbidden error when the dashboard worked yesterday?</h3>



<p class="wp-block-paragraph"><code>Forbidden</code> points at the URL, not at permissions. The most common causes are caching the URL, generating it more than five minutes before use, or a client retry redeeming the same single-use token twice. If it is <code>Unauthorized</code> instead, the session expired and you need a fresh URL.</p>



<h3 class="wp-block-heading">Can I white-label the embedded dashboard completely?</h3>



<p class="wp-block-paragraph">Largely. Themes control colours and typography, the SDK hides toolbar controls, and parameters let your own UI drive the dashboard. Removing the attribution footer entirely is tied to an annual capacity commitment, so confirm that against current terms before you promise it to a customer.</p>



<h3 class="wp-block-heading">Does natural-language querying work in an embedded context?</h3>



<p class="wp-block-paragraph">Yes. The SDK exposes a generative Q&amp;A experience alongside dashboards and visuals, driven by curated topics rather than raw tables. It is billed on its own capacity dimension and gates behind the Pro tiers, so treat it as a separate cost decision rather than a free addition.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Embedded analytics on AWS is not a rendering problem. The iframe is the last five percent. The part that decides whether the build survives contact with a second customer is the identity model, because it silently determines which isolation mechanism you are allowed to use, and that in turn shapes your dataset design, your refresh strategy and your bill.</p>



<p class="wp-block-paragraph">Pick that first. Write the cross-tenant test before you write the dashboard. Everything else is recoverable in an afternoon.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with an embedded analytics build?</h2>



<p class="wp-block-paragraph">This is the kind of work I do. Things I can help with directly:</p>



<ul class="wp-block-list">
<li>Reviewing an existing embed integration for cross-tenant leakage, including the session-tag path and the namespace boundary.</li>



<li>Designing the identity and row-level security model before you commit to a dataset layout.</li>



<li>Cutting Athena scan and Redshift concurrency cost by moving the right datasets into SPICE with incremental refresh.</li>



<li>Building the backend embed-URL service with scoped IAM roles, domain conditions and sane session lifetimes.</li>



<li>Setting up refresh failure alerting so a stale dashboard does not quietly serve last week&#8217;s numbers.</li>



<li>Automated cross-tenant isolation tests wired into CI, so an RLS regression fails the build instead of the customer.</li>
</ul>



<p class="wp-block-paragraph">Send me the actual thing: your embed URL request payload with secrets stripped, the browser console error, or the dataset RLS rules. It is much faster to reason about a real payload than a description of one.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/embedded-analytics-on-aws/">Embedded Analytics on AWS: The Four Decisions That Bite Later</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building an AI Construction Claims Platform on AWS That Holds Up Under Scrutiny</title>
		<link>https://john-nessime.com/blog/case-studies/ai-construction-claims-platform-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 03 Aug 2026 14:05:46 +0000</pubDate>
				<category><![CDATA[Case Studies]]></category>
		<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon S3 Vectors]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Bedrock Guardrails]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Construction Technology]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Embeddings]]></category>
		<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Legal Tech]]></category>
		<category><![CDATA[Metadata Filtering]]></category>
		<category><![CDATA[Primavera P6]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[SQL]]></category>
		<category><![CDATA[Vector Database]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=130</guid>

					<description><![CDATA[<p>Semantic search finds the most persuasive document, not the earliest one. Here is how to architect an AI construction claims and dispute intelligence platform on AWS so retrieval respects the contractual clock, schedule data stays out of the vector index, every answer resolves to a page, and privileged material never shares a retrieval path with project records.</p>
<p>The post <a href="https://john-nessime.com/blog/case-studies/ai-construction-claims-platform-aws/">Building an AI Construction Claims Platform on AWS That Holds Up Under Scrutiny</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone hands you a shared drive and asks a question that sounds trivial: &#8220;Did we give notice of the delay event inside the contractual period, or didn&#8217;t we?&#8221;</p>



<p class="wp-block-paragraph">The answer is in there. It is one email, or one line in a site diary, sitting among forty thousand other files. Nobody can read forty thousand files, so the instinct is to point a language model at the pile and ask it. That instinct is right. The naive implementation of it is where the money goes.</p>



<p class="wp-block-paragraph">Here is the failure mode that bites hardest, and it is invisible until an expert challenges you on it. You build retrieval over the document set, ask about notice of delay, and the system confidently returns a letter that discusses the delay event in great detail. It is a good letter. It is also dated eleven months after the event, written by the claims consultant during preparation of the claim itself. It scored highest precisely because it was written to argue the point. The contemporaneous notice, the thing you actually needed, was four badly typed lines in a routine progress email that mentioned the word &#8220;delay&#8221; once.</p>



<p class="wp-block-paragraph">Semantic similarity has no concept of a deadline. That single gap is the difference between an <strong>AI construction claims platform</strong> that shortens a disclosure exercise and one that quietly manufactures a wrong answer with a citation attached to it.</p>



<p class="wp-block-paragraph">This post covers how to build that platform on AWS: how to lay out ingestion, how to make retrieval respect the contractual clock, why schedule data must never go anywhere near your vector index, how to keep privileged material out of the same retrieval path as project records, and which AWS building blocks are actually the current ones now that several of the obvious candidates have been moved to maintenance mode.</p>



<h2 class="wp-block-heading">What a claims platform actually has to answer</h2>



<p class="wp-block-paragraph">Before any architecture, be honest about the question shapes. They are not all the same problem and they do not all get solved by retrieval.</p>



<ol class="wp-block-list"><li><strong>Chronology.</strong> What happened, in what order, and on what date was it recorded? This is a retrieval and metadata problem.</li><li><strong>Entitlement.</strong> Which clause applies, and what did it require the parties to do? This is retrieval over the contract plus careful prompting.</li><li><strong>Causation.</strong> Which event moved the critical path, and by how much? This is schedule data and date arithmetic. It is not a language problem at all.</li><li><strong>Quantum.</strong> What did the disruption cost? This is cost and resource data, joined to the events above.</li></ol>



<p class="wp-block-paragraph">Treat all four as &#8220;ask the documents&#8221; and you will get fluent nonsense on two of them. The architecture below splits them deliberately.</p>



<h2 class="wp-block-heading">Failure one: retrieval that finds the best match instead of the first one</h2>



<p class="wp-block-paragraph">Two corpora live in every dispute bundle and they look identical to an embedding model.</p>



<ul class="wp-block-list"><li><strong>Contemporaneous records.</strong> Site diaries, progress emails, minutes, early warnings, RFIs, instructions. Written while the project was running, by people with no idea a dispute was coming.</li><li><strong>Claim-era material.</strong> Narratives, expert reports, without-prejudice correspondence, internal analysis. Written afterwards, specifically to be persuasive about the same events.</li></ul>



<p class="wp-block-paragraph">Claim-era material wins on cosine similarity almost every time, because it is denser in exactly the terms you searched for. If your retriever cannot distinguish them, every answer is contaminated by the argument you were trying to test.</p>



<p class="wp-block-paragraph">The fix is metadata, applied at ingestion, and it is cheap to get right and expensive to retrofit. Amazon Bedrock Knowledge Bases reads a sidecar file that sits next to each document in S3, named with the full original filename plus <code>.metadata.json</code>. So <code>letter-0421.pdf</code> gets <code>letter-0421.pdf.metadata.json</code>. The naming convention is the only link between them; there is no separate registration step.</p>



<pre class="wp-block-code"><code>{
  "metadataAttributes": {
    "doc_date": 20240314,
    "corpus": "contemporaneous",
    "doc_type": "site_correspondence",
    "matter_id": "matter-0007",
    "date_source": "email_header",
    "privileged": false
  }
}</code></pre>



<p class="wp-block-paragraph">Look closely at <code>doc_date</code>. It is an integer, not a string, and that is not a style choice. Bedrock Knowledge Bases metadata attributes support STRING, NUMBER, BOOLEAN and STRING_LIST. The range comparison operators, the ones you need to express &#8220;on or before the notice deadline&#8221;, only apply to NUMBER. Store the date as <code>"2024-03-14"</code> and your filter will not throw an error. It will just quietly match nothing, or match everything, depending on how you wrote it. You will find out weeks later when someone asks why a document they can see in the bundle never appears in results.</p>



<p class="wp-block-paragraph">With the date as a sortable integer, a query filter can express the contractual window directly.</p>



<pre class="wp-block-code"><code>{
  "andAll": [
    { "equals":              { "key": "corpus",   "value": "contemporaneous" } },
    { "equals":              { "key": "matter_id","value": "matter-0007" } },
    { "greaterThanOrEquals": { "key": "doc_date", "value": 20240301 } },
    { "lessThanOrEquals":    { "key": "doc_date", "value": 20240329 } }
  ]
}</code></pre>



<p class="wp-block-paragraph">That is the whole trick. You are no longer asking &#8220;what is the most relevant document about this delay&#8221;. You are asking &#8220;what did the parties actually write during the window in which the contract required them to write it&#8221;. Those are different questions and only one of them is worth anything in a dispute.</p>



<h3 class="wp-block-heading">Where the date comes from matters more than the date</h3>



<p class="wp-block-paragraph">Do not use the S3 object timestamp. It records when someone copied a folder, usually years after the fact and identical across ten thousand files. Derive the date from the document itself: the <code>Date:</code> header on an email, the printed date on a letter, the period covered by a diary entry.</p>



<p class="wp-block-paragraph">Sometimes you cannot, because the scanned undated fax exists in every project archive. Record that honestly with a <code>date_source</code> attribute rather than guessing, and treat unknown-date documents as a separate review pile. An extension of time argument built on an inferred date is an argument you will lose.</p>



<h2 class="wp-block-heading">Failure two: treating the programme like a document</h2>



<p class="wp-block-paragraph">This one is worse, because the output looks right.</p>



<p class="wp-block-paragraph">Oracle Primavera P6 exports XER and PMXML files. Asta Powerproject and Microsoft Project have their own formats. XER in particular is a plain text dump of relational tables, so it goes through a text pipeline without complaint. Chunk it, embed it, and you now have vectors representing fragments of a table of activity codes with no relationships attached.</p>



<p class="wp-block-paragraph">Ask that index how much float activity A1200 had at the March data date and you will get a number. It will be well formatted and it will be invented. Total float is the product of a forward and backward pass across the whole logic network under a specific calendar. It cannot be recovered from a retrieved fragment, and a language model asked to produce it will produce something plausible instead of admitting that.</p>



<p class="wp-block-paragraph">Schedule data goes into a structured store, and the model queries it rather than reasoning about it.</p>



<ol class="wp-block-list"><li>Parse each programme file into tables. <code>PyP6Xer</code> handles XER from Python; MPXJ is a Java library that reads XER, PMXML, Asta Powerproject and MSPDI among others, which matters when the bundle contains four scheduling tools.</li><li>Load activities, logic links, calendars, resource assignments and WBS into Amazon Aurora PostgreSQL for interactive work, or into S3 with AWS Glue and Amazon Athena when you have hundreds of updates and want columnar scans.</li><li>Stamp every row with the <em>data date</em> of the update it came from. This is the single most important column in the whole platform. Without it you have a pile of schedules; with it you have a time series of the project&#8217;s own view of itself.</li><li>Run windows analysis, as-planned versus as-built comparison and float erosion in SQL or Python, deterministically, so the same inputs always give the same numbers.</li><li>Expose the results to the model as a tool it can call, or as generated SQL against a defined schema. The model turns a question into a query and narrates the result. It does not do the arithmetic.</li></ol>



<p class="wp-block-paragraph">A rough shape of the query that makes float erosion visible:</p>



<pre class="wp-block-code"><code>SELECT
    a.activity_id,
    a.data_date,
    a.total_float_days,
    a.total_float_days - LAG(a.total_float_days)
        OVER (PARTITION BY a.activity_id ORDER BY a.data_date)
      AS float_change
FROM   schedule_activities a
WHERE  a.project_id = 'PRJ-01'
  AND  a.data_date BETWEEN DATE '2024-01-01' AND DATE '2024-06-30'
ORDER BY a.activity_id, a.data_date;</code></pre>



<p class="wp-block-paragraph">Nothing clever there, and that is the point. Every number is traceable to a row that came from a named XER file. When an opposing expert asks where a figure came from, the answer is a file name and a query, not &#8220;the model said so&#8221;.</p>



<p class="wp-block-paragraph">Be realistic about effort here. Programme parsing and normalisation across inconsistent updates is the hardest part of the build and the part clients always underestimate. Activity IDs get reused, calendars change mid-project, and someone will have re-baselined without telling anyone. Budget for it.</p>



<h2 class="wp-block-heading">Failure three: an answer with no paper trail</h2>



<p class="wp-block-paragraph">In most RAG applications a citation is a nice touch. In dispute work it <em>is</em> the product. An answer that cannot be traced to a page of a disclosed document is not evidence, it is a rumour with good grammar.</p>



<p class="wp-block-paragraph">Design for that from the ingestion layer, not the presentation layer.</p>



<ul class="wp-block-list"><li><strong>Keep page and position.</strong> Amazon Bedrock Data Automation returns confidence scores and bounding box data alongside extracted fields, and Amazon Textract returns geometry per block. Carry both through the pipeline so a citation resolves to a page and a region, not just a file.</li><li><strong>Route low confidence to humans.</strong> Handwritten site diaries and faxed variation orders will produce low-confidence extractions. Those should land in a review queue by default rather than silently entering the index.</li><li><strong>Reject ungrounded answers.</strong> Amazon Bedrock Guardrails includes contextual grounding checks that score whether a response is supported by the retrieved passages. It reduces confident invention. It does not eliminate it, and anyone who tells you otherwise is selling something.</li><li><strong>Keep an immutable evidential copy.</strong> S3 Versioning plus S3 Object Lock on the landing bucket means the file the platform indexed is provably the file that was disclosed.</li></ul>



<p class="wp-block-paragraph">One design rule underpins all of it: the platform shortlists evidence, it does not decide entitlement. Recognised frameworks for this work, the Society of Construction Law Delay and Disruption Protocol and AACE International&#8217;s Recommended Practice 29R-03 on forensic schedule analysis, both assume a named analyst applying a stated method and exercising judgement. A system that outputs &#8220;the contractor is entitled to 42 days&#8221; is not helping. A system that outputs &#8220;here are the eleven contemporaneous documents inside the notice window, here is the float movement across those updates, here is what is missing&#8221; is doing real work.</p>



<h2 class="wp-block-heading">Failure four: one index for privileged and non-privileged material</h2>



<p class="wp-block-paragraph">Dispute bundles contain legal advice, counsel&#8217;s opinions, without-prejudice correspondence and internal settlement analysis. Those must not be retrievable through the same path as project records.</p>



<p class="wp-block-paragraph">The tempting shortcut is a <code>privileged: false</code> metadata filter on every query. Do not rely on that as your boundary. A metadata filter is a query parameter. One missing filter in one code path, one debug endpoint, one caching layer that drops it, and privileged material surfaces in a general search. The blast radius of that mistake is not a bug report.</p>



<p class="wp-block-paragraph">Separate the indexes physically and separate the IAM roles that can reach them. Amazon S3 Vectors makes this practical: you can set a dedicated customer-managed KMS key per vector index, and you get a large number of indexes per vector bucket, so per-matter and per-sensitivity separation does not become an operational burden. Keep the metadata flag as well, because defence in depth is free, but make the identity boundary the one you actually trust.</p>



<p class="wp-block-paragraph">Amazon Macie is worth pointing at the landing bucket to find personal data you did not expect, particularly in HR records and accident reports that get swept into project archives.</p>



<h2 class="wp-block-heading">Choosing the AWS building blocks, including what not to build on</h2>



<p class="wp-block-paragraph">A lot of published architectures for this kind of platform are now pointing at services AWS has stopped developing. Two matter here, and the dates are the point.</p>



<ul class="wp-block-list"><li><strong>Amazon Kendra</strong> entered maintenance mode on 30 June 2026 and stops accepting new customers on 30 July 2026. Existing customers keep support and security fixes but no new capability. AWS directs new enterprise search and RAG work to Amazon Bedrock Knowledge Bases. If a tutorial or a proposal you are reading starts with a Kendra index, it predates that change.</li><li><strong>Amazon Bedrock Agents</strong> moved to maintenance mode in the same round of service availability changes, with Amazon Bedrock AgentCore as the successor for agentic orchestration. Check the current AWS service availability page before you commit an orchestration layer.</li></ul>



<p class="wp-block-paragraph">For the retrieval layer itself, Bedrock Knowledge Bases now comes in two shapes and the choice is a real trade-off rather than a marketing tier.</p>



<h3 class="wp-block-heading">Managed Knowledge Base</h3>



<p class="wp-block-paragraph">AWS manages the vector store, embeddings model, re-ranker and retrieval orchestration as a single primitive, with native connectors for Amazon S3, SharePoint, Confluence, Google Drive, OneDrive and a web crawler, plus automatic parsing strategy selection and a retriever that decomposes multi-step queries. The connectors pull source permissions along with content, which matters when the document set lives in the client&#8217;s SharePoint rather than a bucket you control.</p>



<p class="wp-block-paragraph">Where it wins: you get a working retrieval layer in an afternoon instead of a fortnight, and the parsing tuning that normally eats the first weeks of a build is done for you. For a first matter, or a proof of value before a client commits budget, this is the one I would reach for.</p>



<h3 class="wp-block-heading">Custom Knowledge Base</h3>



<p class="wp-block-paragraph">You bring your own vector store and control chunking, embedding model and index layout.</p>



<p class="wp-block-paragraph">Where it wins: claims work has awkward chunking requirements. A two-page letter split mid-sentence at a page boundary produces a chunk where the notice sentence has lost its date and its addressee. Controlling chunk boundaries around document structure, and controlling which index a document lands in, are both easier when you own the store. Where it doesn&#8217;t: you now own embedding model upgrades, re-indexing, sync failures and capacity, which is real ongoing work for a small team.</p>



<p class="wp-block-paragraph">Start managed, build a retrieval evaluation set of real questions with known correct documents, and only move to custom when that set demonstrates the problem is chunking. Most teams migrate on a hunch and discover the problem was metadata all along.</p>



<h3 class="wp-block-heading">Where the vector storage bill actually comes from</h3>



<p class="wp-block-paragraph">Rates change, so learn the billing mechanism rather than a number. Amazon S3 Vectors charges on three axes: upload volume by logical gigabyte, storage by logical gigabyte, and queries by data processed, where data processed scales with the size of the index being searched. Note that filtering does not reduce the data processed by a query.</p>



<p class="wp-block-paragraph">That shape suits claims work unusually well. A dispute archive is enormous and cold: millions of chunks, queried by a handful of analysts a few hundred times a day, so you pay mostly for storage, which is the cheap axis. Compare that against Amazon OpenSearch Serverless, which prices on provisioned compute units and therefore rewards high query volume against a smaller index, or Aurora PostgreSQL with pgvector when you already need Aurora for the schedule tables and would rather run one system than two.</p>



<p class="wp-block-paragraph">The practical lever is to split indexes per matter. Query cost scales with index size, so one giant index across every dispute you have ever run makes every query more expensive than it needs to be, on top of being a bad idea for confidentiality.</p>



<h2 class="wp-block-heading">A reference pipeline</h2>



<ol class="wp-block-list"><li>Everything lands in S3 under a per-matter prefix, with Versioning and Object Lock enabled on the evidential copy.</li><li>S3 event notifications trigger AWS Step Functions. Use Step Functions rather than a chain of Lambdas so that a failed extraction on page 300 of a 400-page bundle is visible and resumable.</li><li>Classify and split. Scanned bundles arrive as one PDF containing forty separate documents. Splitting them correctly is a prerequisite for dating them correctly.</li><li>Extract text with Amazon Bedrock Data Automation or Amazon Textract, keeping confidence scores and geometry.</li><li>Derive the document date and write the <code>.metadata.json</code> sidecar. Anything undated goes to the review queue.</li><li>Route by type: correspondence to the knowledge base, programme files to the XER parser and the relational store, cost data to its own tables.</li><li>Sync the knowledge base, then run your retrieval evaluation set before anyone uses it. A sync that succeeds is not the same as an index that answers correctly.</li><li>Serve through an API that refuses to return an answer without citations, and log every query with the filters that were applied.</li></ol>



<p class="wp-block-paragraph">Define the whole thing in Terraform or OpenTofu from the start. Matters are per-client and short-lived, and standing one up should be a variable file, not an afternoon in the console. Point Amazon CloudWatch, or Grafana Cloud if you already run Grafana elsewhere, at the Step Functions execution metrics so a silently failing extraction stage does not go unnoticed for a week.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Date filters return nothing, and no error</h3>



<p class="wp-block-paragraph">Almost always the date was stored as a string. Range operators need NUMBER. Convert to an integer in <code>YYYYMMDD</code> form and re-sync the affected documents.</p>



<h3 class="wp-block-heading">A document is in the bucket but never appears in results</h3>



<p class="wp-block-paragraph">Check the sidecar filename first. It must be the complete original filename with <code>.metadata.json</code> appended, extension included. <code>report.pdf.metadata.json</code> works; <code>report.metadata.json</code> is a file the ingestion job will happily ignore. After that, check whether a filter in the query path is excluding it.</p>



<h3 class="wp-block-heading">Answers cite the right document but the wrong passage</h3>



<p class="wp-block-paragraph">Chunking split the document somewhere structurally meaningful. Look at the raw chunks for that file. If the notice sentence and its date are in different chunks, no amount of prompt tuning fixes it. That is the signal to take control of chunking.</p>



<h3 class="wp-block-heading">Float figures do not match the client&#8217;s own analysis</h3>



<p class="wp-block-paragraph">Check calendars before you check logic. Different activity calendars, a changed default calendar, or an update where someone applied a progress override will move float without any logic change. Reconcile activity counts between your parsed tables and the source file before trusting anything downstream.</p>



<h3 class="wp-block-heading">Query costs jumped without more usage</h3>



<p class="wp-block-paragraph">An index grew. With storage-side vector search, query cost tracks the size of the index being scanned, so ingesting a large new bundle raises the price of every subsequent query against that index. Split by matter.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list"><li>Using the file&#8217;s storage timestamp as the document date. It records the migration, not the event.</li><li>Indexing claim narratives and contemporaneous records into the same corpus with no way to tell them apart.</li><li>Embedding programme exports because they happen to be text files.</li><li>Treating a metadata filter as a privilege boundary instead of an optimisation.</li><li>Letting the model state entitlement conclusions rather than assembling and citing evidence.</li><li>Building on services that have moved to maintenance mode because the tutorial you followed predates the change.</li><li>Shipping without a retrieval evaluation set, so you have no way to know whether a change made things better or worse.</li><li>One index for every matter, which is both a cost problem and a confidentiality problem.</li></ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list"><li>Make the document date a first-class, numeric, filterable attribute, and record where it came from.</li><li>Keep an immutable evidential copy separate from the working copy the pipeline mutates.</li><li>Separate structured schedule and cost data from unstructured documents, and let the model query the former rather than reason about it.</li><li>Build a retrieval evaluation set from real questions with known correct documents before you tune anything.</li><li>Enforce citations at the API layer, so an uncited answer is impossible rather than discouraged.</li><li>Isolate privileged material by index and by IAM role, with metadata as a second layer.</li><li>Log every query with its filters, so you can reconstruct how any given answer was reached.</li><li>Define infrastructure as code so a new matter is a deployment, not a project.</li></ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Can an AI construction claims platform replace a delay expert?</h3>



<p class="wp-block-paragraph">No, and building toward that goal produces something unusable. Established forensic frameworks assume a named analyst applying a stated method whose reasoning can be tested. The platform&#8217;s value is compressing weeks of document review into hours and making the schedule data queryable, so the expert spends their time on judgement rather than searching.</p>



<h3 class="wp-block-heading">Should I use Amazon Kendra for the search layer?</h3>



<p class="wp-block-paragraph">Not for a new build. Kendra entered maintenance mode on 30 June 2026 and closed to new customers on 30 July 2026, with AWS pointing to Bedrock Knowledge Bases for equivalent and more current capability. Existing Kendra deployments continue to be supported, so this is a migration assessment rather than an emergency, but starting there now means starting on a service with no roadmap.</p>



<h3 class="wp-block-heading">How do I stop the model inventing float and delay figures?</h3>



<p class="wp-block-paragraph">Do not give it the chance. Keep schedule data in a relational or columnar store and have the model generate queries against a defined schema, or call a tool that runs a fixed calculation. The arithmetic happens in SQL or Python where it is deterministic and reproducible; the model only turns questions into queries and results into sentences.</p>



<h3 class="wp-block-heading">Which vector store should I choose for a claims archive?</h3>



<p class="wp-block-paragraph">Match the store to your query pattern. Large, cold archives queried by a few analysts favour storage-priced options like Amazon S3 Vectors, where you mostly pay to keep the data. Smaller indexes hit constantly favour compute-priced options like Amazon OpenSearch Serverless. If you already run Aurora PostgreSQL for schedule data, pgvector alongside it is a legitimate way to avoid operating a second system.</p>



<h3 class="wp-block-heading">How do I handle scanned and handwritten site records?</h3>



<p class="wp-block-paragraph">Extract them with confidence scores retained, set a threshold, and route everything below it to human review before indexing. Handwritten diaries are frequently the most probative documents in a delay claim and also the least reliable to read automatically, so the review queue is not an edge case. Plan capacity for it.</p>



<h3 class="wp-block-heading">Where do documents come from if they are not already in S3?</h3>



<p class="wp-block-paragraph">Most project records live in a common data environment such as Procore, Autodesk Construction Cloud, Aconex or a client SharePoint tenancy. Bedrock Managed Knowledge Base has native connectors for SharePoint, Confluence, Google Drive and OneDrive that ingest permissions alongside content. For platforms without a native connector, export to S3 and keep the export manifest as part of the disclosure record.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">An <strong>AI construction claims platform</strong> lives or dies on whether it understands time. Every hard requirement in this build traces back to that: numeric dates so you can filter to a contractual window, a data date on every schedule row so float movement is measurable, a corpus flag so contemporaneous records are not drowned out by material written to argue about them, and citations that resolve to a page so any answer can be checked.</p>



<p class="wp-block-paragraph">Get the temporal metadata right at ingestion and the rest of the architecture is ordinary AWS work. Get it wrong and you have built a very expensive way to retrieve the most persuasive document instead of the true one.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help building this on AWS?</h2>



<p class="wp-block-paragraph">I design and build document and data platforms on AWS, and this kind of system sits squarely in that work. Things I can help with:</p>



<ul class="wp-block-list"><li>Designing the ingestion pipeline: S3 landing zones with Object Lock, Step Functions orchestration, splitting and classifying scanned bundles, and confidence-based routing to human review.</li><li>Getting the temporal metadata model right, including date derivation, sidecar generation and filter design against Amazon Bedrock Knowledge Bases.</li><li>Parsing Primavera P6 XER and PMXML exports into queryable tables in Aurora PostgreSQL or S3 with Glue and Athena, with a data date on every row.</li><li>Choosing and sizing the vector layer across Amazon S3 Vectors, OpenSearch Serverless and pgvector, based on your actual query pattern rather than a benchmark.</li><li>Building index and IAM separation for privileged material, plus KMS key strategy and Macie scanning of landing buckets.</li><li>Setting up retrieval evaluation, citation enforcement, query audit logging and CloudWatch or Grafana dashboards over the pipeline so failures surface early.</li></ul>



<p class="wp-block-paragraph">If you are partway into something like this already, send me a sample metadata sidecar, a Step Functions execution history, or a query that returns the wrong document, and I will tell you what I think is going on.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/case-studies/ai-construction-claims-platform-aws/">Building an AI Construction Claims Platform on AWS That Holds Up Under Scrutiny</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
