<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Python | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/python/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/python/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Thu, 29 Oct 2026 06:58:01 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Python | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/python/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>The Delay Was in the Data Three Weeks Ago: Building a Construction Early Warning Dashboard</title>
		<link>https://john-nessime.com/blog/data-engineering/construction-early-warning-dashboard/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 25 Oct 2026 06:06:42 +0000</pubDate>
				<category><![CDATA[Construction Technology]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Capital Projects]]></category>
		<category><![CDATA[Delay Analysis]]></category>
		<category><![CDATA[Earned Schedule]]></category>
		<category><![CDATA[Grafana]]></category>
		<category><![CDATA[Last Planner System]]></category>
		<category><![CDATA[Leading Indicators]]></category>
		<category><![CDATA[Owner Dashboard]]></category>
		<category><![CDATA[PostgreSQL]]></category>
		<category><![CDATA[Primavera P6]]></category>
		<category><![CDATA[Procore]]></category>
		<category><![CDATA[Project Controls]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Total Float]]></category>
		<category><![CDATA[XER Files]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=1033</guid>

					<description><![CDATA[<p>Schedule delay rarely arrives as a surprise. It sits in float erosion, aging RFIs and slipping weekly commitments long before the monthly update moves the finish date. Here is how to build a dashboard that catches it while there is still float left to spend.</p>
<p>The post <a href="https://john-nessime.com/blog/data-engineering/construction-early-warning-dashboard/">The Delay Was in the Data Three Weeks Ago: Building a Construction Early Warning Dashboard</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The monthly schedule update lands on a Thursday. The planner has re-run the logic, and substantial completion has moved three weeks to the right. The project manager asks the obvious question in the next meeting: why is this the first time anyone is hearing about it?</p>



<p class="wp-block-paragraph">The uncomfortable answer is that people did know. The superintendent knew the drywall crew had been short for two weeks. The project engineer knew a structural RFI had been sitting with the design team well past its due date. The foreman knew the weekly commitments kept slipping. None of it was hidden. It just never reached one screen, in one place, while there was still float left to spend.</p>



<p class="wp-block-paragraph">This post walks through building a <strong>construction early warning dashboard</strong> that catches those signals while they are still cheap to act on. It covers why schedule delay shows up late, the one data decision that makes early warning possible, the signals worth tracking, how to pull data out of Primavera P6, Microsoft Project and Procore, and how to alert without training everyone to ignore the alerts.</p>



<h2 class="wp-block-heading">Why Construction Delay Shows Up Three Weeks Late</h2>



<p class="wp-block-paragraph">Delay rarely arrives as one event. It accumulates in small amounts across activities that do not look critical yet. Three things keep it invisible.</p>



<ol class="wp-block-list">
<li><strong>The schedule is updated in batches.</strong> If the CPM schedule is statused monthly, or even biweekly, the finish date is only recalculated at that cadence. Everything that happened in between lands at once.</li>



<li><strong>Progress is self-reported.</strong> Percent complete is an estimate from whoever statuses the activity. Optimism is not dishonesty, but it compounds. An activity that sits at a comfortable-looking percentage for three updates is telling you something.</li>



<li><strong>The data is overwritten in place.</strong> This is the one that bites hardest. Most schedule tools, and most dashboards built on top of them, show the current value of float, the current forecast finish and the current status. Last month&#8217;s values are gone unless someone saved the file.</li>
</ol>



<p class="wp-block-paragraph">That third point is why so many project dashboards look fine right up until they don&#8217;t. A current-state dashboard tells you where you are. Delay warnings live in the rate of change. An activity with twelve days of float looks healthy. The same activity with twelve days of float, down from thirty over the last two updates, is the next critical path.</p>



<h2 class="wp-block-heading">The Design Decision That Makes Early Warning Possible: Snapshot Everything</h2>



<p class="wp-block-paragraph">Before any chart or alert, decide that every schedule update, every RFI log pull and every weekly plan gets stored as a dated snapshot. You append. You never overwrite. Storage for this is trivial; a few hundred activities per update is nothing for any database.</p>



<p class="wp-block-paragraph">PostgreSQL is what I reach for first here. It runs happily on a small managed instance like Amazon RDS or DigitalOcean Managed Databases, or on a modest VPS from Hetzner, Contabo or Interserver if you are comfortable handling backups yourself. A minimal activity snapshot table looks like this:</p>



<pre class="wp-block-code"><code>CREATE TABLE activity_snapshot (
    project_code     text        NOT NULL,
    data_date        date        NOT NULL,
    task_code        text        NOT NULL,
    task_name        text,
    total_float_days numeric,
    early_start      date,
    early_finish     date,
    is_complete      boolean     NOT NULL DEFAULT false,
    loaded_at        timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (project_code, data_date, task_code)
);</code></pre>



<p class="wp-block-paragraph">The primary key does the important work. Each activity appears once per data date, so loading the same export twice fails loudly instead of silently doubling the history. The <code>data_date</code> is the schedule&#8217;s status date, not the day you loaded the file, because trends must be plotted against when the schedule says it was true.</p>



<p class="wp-block-paragraph">Keying on <code>task_code</code>, the activity ID the planner sees, rather than an internal database ID matters too. Internal IDs can change when a schedule is re-imported. Activity IDs usually survive, and when they don&#8217;t, you want to know (more on that in troubleshooting).</p>



<h2 class="wp-block-heading">Early Warning Signals That Move Before the Finish Date Does</h2>



<p class="wp-block-paragraph">Each of these is a leading indicator of schedule delay. None is perfect alone. Together they give you weeks of notice instead of a surprise in the monthly update.</p>



<h3 class="wp-block-heading">Float Erosion on Near-Critical Paths</h3>



<p class="wp-block-paragraph">Watching the critical path is necessary but late. By the time an activity is critical, its float is already gone. The useful signal is near-critical work losing float quickly. This query compares each open activity to its own previous snapshot:</p>



<pre class="wp-block-code"><code>WITH history AS (
    SELECT
        project_code,
        task_code,
        task_name,
        data_date,
        total_float_days,
        LAG(total_float_days) OVER (
            PARTITION BY project_code, task_code
            ORDER BY data_date
        ) AS prev_float_days
    FROM activity_snapshot
    WHERE NOT is_complete
)
SELECT
    project_code,
    task_code,
    task_name,
    prev_float_days,
    total_float_days,
    prev_float_days - total_float_days AS float_lost_days
FROM history
WHERE data_date = (SELECT max(data_date) FROM activity_snapshot)
  AND prev_float_days IS NOT NULL
  AND total_float_days &lt;= 20
  AND prev_float_days - total_float_days &gt;= 5
ORDER BY float_lost_days DESC;</code></pre>



<p class="wp-block-paragraph">The thresholds (twenty days remaining, five days lost) are starting points, not rules. Tune them to your update cadence and project length. What matters is that the query looks at movement, which a current-state report cannot.</p>



<h3 class="wp-block-heading">Constraint Aging: RFIs and Submittals Against Need-By Dates</h3>



<p class="wp-block-paragraph">An overdue RFI on its own is noise. Every project has them. An overdue RFI that blocks an activity starting in ten days is a delay you can see coming.</p>



<p class="wp-block-paragraph">The hard part is the link. RFI and submittal logs in Procore or Autodesk Construction Cloud rarely point at schedule activity IDs. Someone has to maintain a small mapping table from RFI or submittal to activity, or at least to area and spec section. I would not skip it. Without the link you get a list of late paperwork; with it you get &#8220;this answer is needed in eight days and the ball has been in the architect&#8217;s court for three weeks.&#8221;</p>



<p class="wp-block-paragraph">The signal to chart is days of slack between the expected response and the need-by date of the linked activity. When it goes negative, the RFI is now on the schedule&#8217;s path whether the schedule knows it or not.</p>



<h3 class="wp-block-heading">Commitment Reliability: PPC and Reasons for Variance</h3>



<p class="wp-block-paragraph">If the field runs the Last Planner System, you already have a strong signal. Percent Plan Complete (PPC) is the share of weekly commitments finished as promised. Each week&#8217;s PPC is a lagging number, but the trend across weeks leads the CPM schedule, because missed promises show up in the field long before they show up as recalculated dates.</p>



<p class="wp-block-paragraph">Store reasons for variance as categories, not free text. A PPC dip caused by weather tells you something different from three straight weeks of &#8220;material not on site.&#8221; The second one is a procurement problem that will keep producing delay until someone fixes it.</p>



<h3 class="wp-block-heading">Look-Ahead Churn</h3>



<p class="wp-block-paragraph">Count how many times an activity&#8217;s planned start has moved across snapshots of the three- or six-week look-ahead. An activity pushed out once is normal planning. One that has been pushed four updates running is being deferred, not planned, and the reason is usually a constraint nobody has named yet.</p>



<h3 class="wp-block-heading">Earned Value, Read Carefully</h3>



<p class="wp-block-paragraph">If you track earned value, the Schedule Performance Index (SPI = EV / PV) belongs on the dashboard, with a caveat many dashboards skip. Classic SPI drifts back toward 1.0 as the project nears completion, even when the job is late, because earned value and planned value both end at the budget at completion. A late-stage SPI that looks like recovery may just be arithmetic.</p>



<p class="wp-block-paragraph">Earned Schedule fixes this by measuring in time rather than money. It asks at what point in the baseline your current earned value was supposed to be reached, then divides that by the actual time elapsed to get SPI(t). It needs no new data collection, only different math on the same numbers. If you only plot one schedule index late in a job, plot SPI(t).</p>



<h3 class="wp-block-heading">Claimed Progress Versus Manpower</h3>



<p class="wp-block-paragraph">Daily logs record who was on site. When an activity&#8217;s reported percent complete keeps rising while the manpower logged against that trade stays well below the plan, one of those numbers is wrong. Usually it is the percent complete. This comparison is crude, and it needs a consistent trade mapping, but it is one of the few checks that uses data nobody is trying to make look good.</p>



<h2 class="wp-block-heading">Getting the Data Out of P6, Microsoft Project and Procore</h2>



<p class="wp-block-paragraph">A construction project dashboard lives or dies on boring extraction work. Plan for it.</p>



<h3 class="wp-block-heading">Primavera P6 XER Exports</h3>



<p class="wp-block-paragraph">An XER file is plain tab-delimited text. Lines beginning with <code>%T</code> name a table, <code>%F</code> lists its fields, and <code>%R</code> lines are rows. Activities live in the <code>TASK</code> table. This reader pulls one table into a list of dictionaries:</p>



<pre class="wp-block-code"><code>def read_xer_table(path, table_name):
    rows, fields, current = [], None, None
    # XER exports often use a Windows code page, not UTF-8
    with open(path, encoding="cp1252", errors="replace") as fh:
        for line in fh:
            parts = line.rstrip("rn").split("t")
            tag = parts[0]
            if tag == "%T":
                current = parts[1]
            elif tag == "%F" and current == table_name:
                fields = parts[1:]
            elif tag == "%R" and current == table_name:
                rows.append(dict(zip(fields, parts[1:])))
    return rows

tasks = read_xer_table("update.xer", "TASK")
for t in tasks[:5]:
    print(t["task_code"], t["task_name"], t["total_float_hr_cnt"])</code></pre>



<p class="wp-block-paragraph">Two details catch people. Float and durations in XER are stored in hours (<code>total_float_hr_cnt</code>), so converting to days means dividing by the hours per day of each activity&#8217;s calendar, not a blanket eight. And an activity with a value in <code>act_end_date</code> has an actual finish, which is a clean way to set <code>is_complete</code> without relying on status labels.</p>



<p class="wp-block-paragraph">Use the float P6 stored in the export rather than recomputing the critical path yourself. P6&#8217;s result reflects its scheduling options, constraints and calendars. Your recompute probably won&#8217;t.</p>



<h3 class="wp-block-heading">Microsoft Project</h3>



<p class="wp-block-paragraph">Microsoft Project can save a schedule as XML, which carries tasks, dates and total slack. The same snapshot logic applies. Save one file per status date, load it with its data date, and never replace the previous one.</p>



<h3 class="wp-block-heading">Procore and Other Project Management Platforms</h3>



<p class="wp-block-paragraph">Procore&#8217;s REST API exposes RFIs and submittals per project, including status, due dates and ball-in-court. Two warnings. Most useful endpoints are scoped to a single project, so a portfolio view means iterating over the project list. And submittal status values can be configured per company, so map them explicitly rather than hard-coding what you saw on one account. Autodesk Construction Cloud has its own RFI API with a similar shape.</p>



<h3 class="wp-block-heading">The Spreadsheet Nobody Wants to Admit Exists</h3>



<p class="wp-block-paragraph">The weekly work plan is very often an Excel file or a Smartsheet. Accept that. A scheduled job that reads a fixed-layout sheet from a shared folder is less elegant than an API and far more likely to get used. Fighting the field&#8217;s tools is how early warning systems die.</p>



<h2 class="wp-block-heading">Building the Dashboard and Alerts People Don&#8217;t Mute</h2>



<p class="wp-block-paragraph">For visualization I lean toward Grafana, self-hosted or on Grafana Cloud, because its alert rules can run SQL directly against PostgreSQL and route results to email, Slack or Microsoft Teams. You can see the style of dashboard in this <a href="https://monitor.john-nessime.com/public-dashboards/4c1463601230401ab55159059364b739" target="_blank" rel="noreferrer noopener">live Grafana demo</a>.</p>



<p class="wp-block-paragraph">Power BI deserves a fair hearing. Many contractors already license it, project controls teams know it, and its reporting and drill-down are excellent for monthly reviews. Where it is weaker is rule-based alerting on arbitrary queries. If your organization lives in Microsoft 365, a sensible split is Power BI for the reporting pack and a small scheduled job or Grafana instance for the alerts. Metabase is another reasonable middle ground for smaller teams.</p>



<p class="wp-block-paragraph">Whatever the tool, the alert design matters more than the charts:</p>



<ul class="wp-block-list">
<li><strong>Send a weekly digest, not a page.</strong> Schedule risk is not an outage. A Monday summary of every signal that crossed a threshold gets read. Twenty individual notifications get filtered.</li>



<li><strong>Name the owner in the alert.</strong> &#8220;RFI 142 blocks level 3 slab pour, need-by in 8 days, ball in court: structural engineer&#8221; is actionable. &#8220;Constraint risk: high&#8221; is not.</li>



<li><strong>Compare the project to itself.</strong> Portfolio-wide thresholds punish complex jobs and let simple ones coast. A drop from its own recent average is the better trigger.</li>



<li><strong>Alert on stale data.</strong> If no schedule update has loaded in longer than your cadence, that is itself a warning. An early warning dashboard showing old data is worse than none.</li>
</ul>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Float values look wrong compared to P6</h3>



<p class="wp-block-paragraph">Check the hours-to-days conversion first. Then check which float definition the schedule uses. A project-level &#8220;must finish by&#8221; date or activity constraints can produce negative float that looks like a bug but is correct. Also confirm whether the planner defines critical as longest path or as float below a threshold.</p>



<h3 class="wp-block-heading">Trend lines break after a re-baseline or renumbering</h3>



<p class="wp-block-paragraph">If activity IDs change, <code>LAG()</code> finds no previous row and the activity appears brand new. Add a check that counts activities present in the previous snapshot but missing from the current one. A sudden jump means the history needs a mapping table, not a fresh start.</p>



<h3 class="wp-block-heading">PPC sits at or near 100 percent every week</h3>



<p class="wp-block-paragraph">That usually means commitments are being made at the end of the week, or tasks are so loose they cannot fail. A PPC that never moves carries no signal. Look at how commitments are recorded before trusting the number.</p>



<h3 class="wp-block-heading">The dashboard is green but the field says you are behind</h3>



<p class="wp-block-paragraph">Check how percent complete is calculated. P6 supports duration, physical and units percent complete types, and they can tell very different stories about the same activity. Believe the field, then find which input is out of step.</p>



<h2 class="wp-block-heading">Common Mistakes</h2>



<ul class="wp-block-list">
<li>Building on the live schedule only, with no history, so every trend has to be reconstructed by hand.</li>



<li>Watching only the critical path and missing the near-critical work that becomes critical next month.</li>



<li>Trusting classic SPI in the final third of a project, when it drifts toward 1.0 regardless of performance.</li>



<li>Tracking overdue RFIs without linking them to the activities they block.</li>



<li>Recomputing float yourself instead of using what the scheduling tool calculated.</li>



<li>Sending so many alerts that the project team creates an inbox rule to hide them.</li>
</ul>



<h2 class="wp-block-heading">Best Practices</h2>



<ul class="wp-block-list">
<li>Append every export with its data date. Never overwrite history.</li>



<li>Start with two signals, float erosion and constraint aging, and add more once those are trusted.</li>



<li>Agree with the planner on update cadence. The dashboard cannot be fresher than its inputs.</li>



<li>Keep the RFI-to-activity mapping small and owned by one named person.</li>



<li>Review thresholds after the first month. Early numbers are guesses.</li>



<li>Show each signal with its source and data date, so nobody argues about whether it is current.</li>
</ul>



<h2 class="wp-block-heading">Frequently Asked Questions</h2>



<h3 class="wp-block-heading">What are leading indicators of construction delay?</h3>



<p class="wp-block-paragraph">The most useful are float erosion on near-critical activities, RFIs and submittals aging toward the start of the work they block, a falling PPC trend, activities repeatedly pushed out of the look-ahead, and reported progress outpacing logged manpower.</p>



<h3 class="wp-block-heading">How often should the schedule be updated for early warning to work?</h3>



<p class="wp-block-paragraph">More often than monthly if you can manage it. Weekly or biweekly updates give the trend queries enough points to show movement. Signals from RFIs and weekly plans can refresh daily regardless of schedule cadence.</p>



<h3 class="wp-block-heading">Is SPI a good early warning metric?</h3>



<p class="wp-block-paragraph">Early and mid-project, yes, alongside float and critical path analysis. Late in a project, classic SPI drifts toward 1.0 even on late jobs, so the time-based SPI(t) from Earned Schedule is more trustworthy.</p>



<h3 class="wp-block-heading">Can I build this with Power BI instead of Grafana?</h3>



<p class="wp-block-paragraph">Yes for the reporting side, and it is often the better choice where the team already uses it. The snapshot database and trend logic are the same either way. Rule-based alerting is where you may want a separate tool.</p>



<h3 class="wp-block-heading">Do I need Primavera P6 for a construction early warning dashboard?</h3>



<p class="wp-block-paragraph">No. Any scheduling tool that exports activities with dates and float works, including Microsoft Project. What you need is a consistent export on every update and somewhere to keep the history.</p>



<h3 class="wp-block-heading">How do I link RFIs to schedule activities?</h3>



<p class="wp-block-paragraph">Usually with a small mapping table maintained by the project engineer: RFI or submittal number, activity ID, and need-by date. Mapping by area and spec section is a workable first step if activity-level links are too much at the start.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">The project manager who learns about delay three weeks late usually isn&#8217;t short of data. The problem is that the data is spread across a schedule, an RFI log and a weekly plan, and each one only remembers its latest state. A construction early warning dashboard works because it keeps the history and watches the movement: float draining from work that isn&#8217;t critical yet, answers arriving after the work that needs them, promises slipping week after week. Snapshot first. Everything else is a query on top.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need Help Building a Construction Early Warning Dashboard?</h2>



<p class="wp-block-paragraph">I build and host the data side of this for construction teams and project controls groups. Here is where I can help:</p>



<ul class="wp-block-list">
<li>Setting up a snapshot database that stores every P6 or Microsoft Project update with its data date.</li>



<li>Writing XER and XML loaders that handle hours-based float, calendars and renumbered activities.</li>



<li>Pulling RFIs and submittals from Procore or Autodesk Construction Cloud and linking them to activities.</li>



<li>Building Grafana dashboards for float erosion, constraint aging, PPC and SPI(t) trends.</li>



<li>Designing weekly digest alerts with named owners, sent to email, Slack or Teams.</li>



<li>Hosting and maintaining the dashboard so your team reads it instead of running it.</li>
</ul>



<p class="wp-block-paragraph">If you want to scope it first, the <a href="https://john-nessime.com/tools/engagement-builder/">Engagement Builder</a> lays out phases and effort. Or send me one recent XER export and your RFI log, and I will tell you which signals your data can already support.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://john-nessime.com/projects/hosted-monitoring/">See Hosted Monitoring</a></div>



<div class="wp-block-button is-style-outline is-style-outline--1"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/data-engineering/construction-early-warning-dashboard/">The Delay Was in the Data Three Weeks Ago: Building a Construction Early Warning Dashboard</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Turning External Documents Into Structured Data With Amazon Textract</title>
		<link>https://john-nessime.com/blog/technical-guides/amazon-textract-structured-data/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 22 Sep 2026 06:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon SNS]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Bedrock Data Automation]]></category>
		<category><![CDATA[Boto3]]></category>
		<category><![CDATA[Confidence Scoring]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Human In The Loop]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[OCR]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[Serverless]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=390</guid>

					<description><![CDATA[<p>Amazon Textract almost never fails loudly. A working guide to turning external PDFs and scans into structured data: picking the right operation per document family, parsing the block graph, routing on per-field confidence, and catching the limits that silently truncate your records.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/amazon-textract-structured-data/">Turning External Documents Into Structured Data With Amazon Textract</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A vendor sends over a year of delivery notes. It arrives as one PDF, several hundred pages, scanned on a machine that was clearly overdue for service. Someone asks how long it will take to get the reference numbers into the warehouse table, and the honest answer is that the OCR call is the easy part.</p>



<p class="wp-block-paragraph">Document pipelines are not hard because optical character recognition is inaccurate. Amazon Textract reads text well. They are hard because everything after the API call belongs to you, and because Textract almost never fails loudly. You get a 200 response, a list of blocks, a confidence score, and a record with an empty field where the purchase order number should have been. Nothing alerts. The row lands in the warehouse with a null, and someone in finance notices three weeks later while reconciling.</p>



<p class="wp-block-paragraph">That silent gap is the whole problem. Getting reliable Amazon Textract structured data out of files you did not design has less to do with OCR accuracy than with everything around it: which operation to call for which document family, why the response is a graph rather than a record, how to route on confidence instead of trusting it, and the limits that quietly truncate your data. All of it assumes documents you cannot ask anyone to change.</p>



<h2 class="wp-block-heading">Why external documents break pipelines that passed testing</h2>



<p class="wp-block-paragraph">Internal documents are generated by a template you control. External documents vary along three axes at once, and each one has its own failure mode:</p>



<ul class="wp-block-list">
<li><strong>Container.</strong> One logical document per file, or forty stapled into one PDF, or a single page split across three JPEGs.</li>

<li><strong>Text origin.</strong> Born-digital PDFs with a real text layer, flatbed scans, and photographs taken at an angle in bad light. Same file extension, wildly different accuracy.</li>

<li><strong>Layout stability.</strong> The vendor redesigns their invoice template and moves the PO number from the header to a footer block. Nobody tells you.</li>
</ul>



<p class="wp-block-paragraph">You cannot fix any of that upstream. You can only detect it at ingest and route accordingly. The sections below are organized by the failure family each one produces.</p>



<h2 class="wp-block-heading">Failure family one: the file is out of spec before Textract sees it</h2>



<p class="wp-block-paragraph">Textract accepts JPEG, PNG, PDF, and TIFF. It does not support XFA-based PDFs, and it will not open a password-protected PDF. Images are accepted up to 10,000 pixels on any side. For PDFs, the maximum page dimensions are 40 inches and 9,000 points.</p>



<p class="wp-block-paragraph">Two more constraints catch people out. Text detection covers English, French, German, Italian, Portuguese, and Spanish, and several capabilities are narrower still: handwriting, invoices and receipts, identity documents, and Queries are English only. Vertical text, common in Japanese and Chinese documents, is not supported at all. Rotation is fine, including heavy in-plane rotation. Vertical writing direction is not.</p>



<p class="wp-block-paragraph">Build an ingest gate that checks encryption, page count, byte size, and dimensions before it spends an API call, and give each rejection a reason code a human can act on. Then catch exceptions properly instead of wrapping the call in a bare retry. <code>BadDocumentException</code> means Textract could not read the file, and retrying fails identically. <code>DocumentTooLargeException</code> means route to the asynchronous path or split the file. <code>InvalidS3ObjectException</code> is usually IAM or a bucket-region mismatch, not a document problem. Only <code>InternalServerErrorException</code> deserves a retry with backoff.</p>



<h2 class="wp-block-heading">Failure family two: synchronous versus asynchronous is a correctness gate</h2>



<p class="wp-block-paragraph">This is the one that silently eats data, so it goes early.</p>



<p class="wp-block-paragraph">The synchronous operations, <code>DetectDocumentText</code> and <code>AnalyzeDocument</code>, cap at 10 MB in memory. For PDF and TIFF input they also cap at <strong>one page</strong>. Not one page per call with automatic paging. One page, full stop. Hand a synchronous call a 40-page PDF and you get page one, and a pipeline that reports success.</p>



<p class="wp-block-paragraph">The asynchronous operations are a different shape. PDF and TIFF go up to 500 MB and 3,000 pages, while JPEG and PNG stay at 10 MB. You start a job, Textract works in the background, and you collect results by job ID.</p>



<p class="wp-block-paragraph">A synchronous call looks like this. The document lives in S3, and you name the features you want:</p>



<pre class="wp-block-code"><code>aws textract analyze-document 
  --document '{"S3Object":{"Bucket":"my-doc-bucket","Name":"inbound/note-0042.png"}}' 
  --feature-types '["FORMS","TABLES"]' 
  --region us-east-1</code></pre>



<p class="wp-block-paragraph">The asynchronous version returns a job identifier immediately and publishes completion to an SNS topic. That is what you want in production, because polling a job inside a Lambda function means paying for a function that sleeps:</p>



<pre class="wp-block-code"><code>import boto3

textract = boto3.client("textract")

start = textract.start_document_analysis(
    DocumentLocation={
        "S3Object": {"Bucket": "my-doc-bucket", "Name": "inbound/batch-0042.pdf"}
    },
    FeatureTypes=["FORMS", "TABLES"],
    NotificationChannel={
        "SNSTopicArn": "arn:aws:sns:us-east-1:111122223333:textract-jobs",
        "RoleArn": "arn:aws:iam::111122223333:role/TextractSnsPublishRole",
    },
)

job_id = start["JobId"]</code></pre>



<p class="wp-block-paragraph">Results come back paginated, and forgetting the continuation token is a second, quieter way to lose most of a file:</p>



<pre class="wp-block-code"><code>blocks = []
next_token = None

while True:
    kwargs = {"JobId": job_id}
    if next_token:
        kwargs["NextToken"] = next_token

    page = textract.get_document_analysis(**kwargs)
    blocks.extend(page["Blocks"])

    next_token = page.get("NextToken")
    if not next_token:
        break</code></pre>



<p class="wp-block-paragraph">Check <code>JobStatus</code> before you trust <code>Blocks</code>. A job can come back partially successful, which means some pages processed and some did not. If you treat that as success you will publish a document that is missing its middle.</p>



<h2 class="wp-block-heading">Failure family three: the response is a graph, not a record</h2>



<p class="wp-block-paragraph">Textract does not return fields. It returns a flat list of <code>Block</code> objects, each with an ID and relationships pointing at other IDs. Reassembling those into a row is your code, and it is where most of the real work lives.</p>



<p class="wp-block-paragraph">For form data, the shape is specific. A key-value pair is two blocks of type <code>KEY_VALUE_SET</code>, distinguished by their <code>EntityTypes</code> array containing either <code>KEY</code> or <code>VALUE</code>. The key block carries a relationship of type <code>VALUE</code> holding the ID of its value block, and a relationship of type <code>CHILD</code> holding the IDs of the <code>WORD</code> blocks that spell out the label. The value block carries only <code>CHILD</code> relationships pointing at its own words. So reading one field means three hops.</p>



<pre class="wp-block-code"><code>blocks_by_id = {b["Id"]: b for b in blocks}

def child_text(block):
    parts = []
    for rel in block.get("Relationships", []):
        if rel["Type"] != "CHILD":
            continue
        for cid in rel["Ids"]:
            child = blocks_by_id[cid]
            if child["BlockType"] == "WORD":
                parts.append(child["Text"])
            elif child["BlockType"] == "SELECTION_ELEMENT":
                parts.append(child["SelectionStatus"])
    return " ".join(parts)

fields = {}
for b in blocks:
    if b["BlockType"] != "KEY_VALUE_SET":
        continue
    if "KEY" not in b.get("EntityTypes", []):
        continue

    value_text = ""
    for rel in b.get("Relationships", []):
        if rel["Type"] == "VALUE":
            for vid in rel["Ids"]:
                value_text = child_text(blocks_by_id[vid])

    fields[child_text(b)] = (value_text, b["Confidence"])</code></pre>



<p class="wp-block-paragraph">Two things about that snippet. The <code>SELECTION_ELEMENT</code> branch matters because checkboxes and radio buttons are values too, and their content is a selection status rather than text. And the confidence on the key block is not a separate key confidence: Textract returns the same value for both halves of a pair. If you planned to compare label confidence against value confidence to catch mismatches, that signal does not exist.</p>



<p class="wp-block-paragraph">Tables follow the same pattern with different block types, and cells carry row and column indices plus span values for merged cells. Layout adds another set, including title, header, footer, section header, page number, list, and figure blocks, and it sequences text into human reading order. That last property is why Layout is worth requesting when the output is headed for a language model rather than a database: chunking on reading order beats chunking on raw line order.</p>



<p class="wp-block-paragraph">Before hand-rolling any of this, look at the AWS-maintained parsers. The <code>amazon-textract-response-parser</code> package and the higher-level <code>amazon-textract-textractor</code> library both turn the block graph into objects with sensible properties. Writing your own traversal is a good way to understand the model and a poor way to spend two sprints.</p>



<h2 class="wp-block-heading">Getting Amazon Textract structured data out of the right API</h2>



<p class="wp-block-paragraph">Textract is a family of operations, not one endpoint, and the biggest single accuracy win available is usually picking a narrower one. Features are billed per page and they stack, so requesting <code>TABLES</code> on a corpus with no tables costs money and returns noise.</p>



<ul class="wp-block-list">
<li><strong>DetectDocumentText.</strong> Raw text and geometry. Use it when you only need a search index or a text layer.</li>

<li><strong>AnalyzeDocument with FORMS.</strong> Key-value pairs discovered from labels on the page. Works when the label is physically present and stable.</li>

<li><strong>AnalyzeDocument with TABLES.</strong> Grid reconstruction with row and column indices. The right call for line-item grids and statements.</li>

<li><strong>AnalyzeDocument with QUERIES.</strong> You ask natural language questions and get answers back. This is the fix for layout drift.</li>

<li><strong>AnalyzeDocument with LAYOUT and SIGNATURES.</strong> Reading order and structural elements; signature locations for compliance checks.</li>

<li><strong>AnalyzeExpense.</strong> Purpose-built for invoices and receipts, with normalized field types.</li>

<li><strong>AnalyzeID.</strong> Purpose-built for identity documents.</li>
</ul>



<h3 class="wp-block-heading">Why Queries beats FORMS on documents you don&#8217;t control</h3>



<p class="wp-block-paragraph">FORMS gives you whatever labels the document happens to contain. That means your downstream schema is hostage to a vendor&#8217;s wording. When they change &#8220;P.O. No.&#8221; to &#8220;Order Reference&#8221;, your parser looks for a key that no longer exists and writes a null.</p>



<p class="wp-block-paragraph">Queries inverts this. You supply the question and an alias, and the alias comes back attached to the answer, so your output key is fixed by you rather than by the document:</p>



<pre class="wp-block-code"><code>aws textract analyze-document 
  --document '{"S3Object":{"Bucket":"my-doc-bucket","Name":"inbound/note-0042.png"}}' 
  --feature-types '["QUERIES"]' 
  --queries-config '{"Queries":[
      {"Text":"What is the purchase order number?","Alias":"PO_NUMBER"},
      {"Text":"What is the delivery date?","Alias":"DELIVERY_DATE"}
  ]}' 
  --region us-east-1</code></pre>



<p class="wp-block-paragraph">The alias is the entire point. It is a contract between your schema and the extraction layer that survives a vendor redesign. There are ceilings: 15 queries per page synchronously, 30 per page asynchronously, English only. Phrase questions using words that actually appear on the page, and add positional hints when you know the layout, because the model uses them. If Queries underperforms on your corpus, adapters let you tune the feature against your own examples and pass an adapter ID and version at call time.</p>



<h3 class="wp-block-heading">Invoices are their own problem</h3>



<p class="wp-block-paragraph">For invoices and receipts, reach for AnalyzeExpense first. Its response is shaped around the domain rather than around the page: each <code>ExpenseDocument</code> holds <code>SummaryFields</code> for header-level data and <code>LineItemGroups</code> containing the line items. Fields carry a normalized <code>Type</code> such as <code>VENDOR_NAME</code>, plus a <code>ValueDetection</code> and an optional <code>LabelDetection</code>.</p>



<p class="wp-block-paragraph">That optional label is the thing to notice. AnalyzeExpense can identify a vendor name printed only inside a logo, and pull line items out of a grid with no column headers. FORMS structurally cannot do either, because FORMS needs a key to exist. On receipts that is not a small accuracy delta, it is the difference between a usable field and an empty one.</p>



<h2 class="wp-block-heading">Failure family four: confidence is per field and it is not your risk model</h2>



<p class="wp-block-paragraph">Every extracted element carries a confidence score. Two mistakes follow from that.</p>



<p class="wp-block-paragraph">The first is averaging into a document-level score. A document where every field is strong except the invoice total is not a mostly-good document. It is a document with one field that must not be trusted, surrounded by noise. Score fields, not documents.</p>



<p class="wp-block-paragraph">The second is treating one threshold as universal. A misread address on a delivery note is an annoyance. A misread bank account number is an incident. Set thresholds per field from the cost of being wrong, then route into three lanes:</p>



<ol class="wp-block-list">
<li><strong>Auto-accept.</strong> Above threshold and passing a format check. Straight into the target table.</li>

<li><strong>Review.</strong> Below threshold, or above it but failing a validation rule. A human sees the value and the cropped region it came from.</li>

<li><strong>Reject.</strong> Field absent entirely, or the document failed the ingest gate. Back to the sender with a reason.</li>
</ol>



<p class="wp-block-paragraph">Validation rules do more work than thresholds here. A total that does not equal the sum of its line items is wrong at any confidence. Check-digit rules on account and tax identifiers catch character-level errors the model was perfectly confident about. Confidence tells you how sure the model is. Validation tells you whether the answer is even possible.</p>



<p class="wp-block-paragraph">Textract can hand low-confidence results to Amazon Augmented AI through a human loop configuration on the API call, which saves building a review queue. Whether that beats a small internal tool depends on how much domain context your reviewers need on screen. Either way, capture the corrections. Reviewer edits are the only labeled data you get for free, and they are how you calibrate thresholds against your own corpus instead of guessing.</p>



<h2 class="wp-block-heading">Failure family five: reprocessing you didn&#8217;t budget for</h2>



<p class="wp-block-paragraph">Textract bills per page, per feature. Your parser, meanwhile, will change repeatedly in the first months of a pipeline&#8217;s life, because that is what parsers do.</p>



<p class="wp-block-paragraph">So separate extraction from parsing, permanently. Write the raw response JSON to S3 keyed by a hash of the document bytes plus the operation and feature set requested, then parse from that stored artifact. Reprocessing becomes free, deterministic, and diffable: you can replay a parser change across the whole back catalogue and compare outputs before promoting it. Without that cache, every parser bug fix is a bill.</p>



<pre class="wp-block-code"><code>import hashlib

digest = hashlib.sha256(doc_bytes).hexdigest()
features = "-".join(sorted(["FORMS", "TABLES"]))
key = f"textract-raw/{digest}/analyze-document/{features}.json"</code></pre>



<p class="wp-block-paragraph">Hashing content rather than filename also deduplicates the same scan arriving twice under different names, which happens constantly with email-driven intake. For the asynchronous operations, a client request token gives idempotency at the job level; reusing a token with different parameters raises a mismatch error rather than quietly starting a second job.</p>



<p class="wp-block-paragraph">One caveat: source documents often contain personal data, so a cache is a retention decision. Set S3 lifecycle rules on both the originals and the raw JSON, and if working copies land on a workstation or a build box, delete them properly. Tools like O&amp;O SafeErase exist for exactly that gap between &#8220;moved to trash&#8221; and &#8220;actually gone&#8221;.</p>



<h2 class="wp-block-heading">Where Textract stops being the right tool</h2>



<p class="wp-block-paragraph">Being fair about this matters, because AWS now offers overlapping paths to the same outcome. Amazon Bedrock Data Automation is a managed document processing service that handles classification and extraction behind one API, priced per document rather than per page and per feature. AWS positions it as the default starting point for new builds, and for mixed document types with no dominant format that recommendation is sound. You write far less orchestration.</p>



<p class="wp-block-paragraph">Textract keeps the advantage where the corpus is high volume and standardized, where geometry and per-field confidence are needed for auditability, where regional availability constrains you, or where you want extraction to be deterministic and separately testable. Bounding boxes and confidence scores are not a nice-to-have in a regulated workflow, they are the evidence trail.</p>



<p class="wp-block-paragraph">The hybrid is common and sensible: Textract for extraction, a Bedrock model for reasoning over the extracted text. Just do not send a multimodal model a full-page image because OCR felt inconvenient. It costs considerably more per page and gives you no geometry.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Only the first page came through.</strong> A synchronous call on a multi-page PDF or TIFF. Move to the asynchronous operations, and assert the page count Textract reports against the count you expected.</li>

<li><strong>Output truncates partway through a long document.</strong> Missing continuation token handling on the get operation. Loop until the token is absent.</li>

<li><strong>Fields visible on the page never appear in output.</strong> The label is implied rather than printed, or sits inside a logo. Switch that field to Queries, or to AnalyzeExpense if it is an invoice.</li>

<li><strong>Quality collapsed for one sender only.</strong> Template change. Diff the current layout against a stored sample, which is the argument for keeping raw responses.</li>

<li><strong>Throttling during a backfill.</strong> Transactions per second and concurrent job counts are per-account, per-region quotas. Queue submissions rather than fanning out, and request an increase before the migration.</li>

<li><strong>Table cells land in the wrong columns.</strong> Usually a skewed scan. Deskew and raise contrast before submitting, because image preparation is cheaper than post-processing repairs.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Testing on three clean sample documents supplied by the vendor rather than a random sample of real intake.</li>

<li>Requesting every feature type on every page because it is simpler than routing, then being surprised by the bill.</li>

<li>Using FORMS for invoices when AnalyzeExpense exists and understands unlabeled fields.</li>

<li>Treating a confidence score as an accuracy guarantee instead of pairing it with validation rules.</li>

<li>Discarding the raw JSON after parsing, which turns every parser fix into a re-extraction charge.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Classify before you extract. Route each document to the narrowest API that covers it.</li>

<li>Use Queries with explicit aliases for fields your schema depends on, so vendor wording changes cannot break your column names.</li>

<li>Store raw responses immutably, keyed by content hash, and parse from storage.</li>

<li>Set per-field confidence thresholds from the cost of being wrong, and pair every threshold with a format or arithmetic validation.</li>

<li>Instrument it like any other pipeline: extraction latency, straight-through rate, review queue depth, per-field null rate. A slow rise in nulls on one field is the earliest signal of a template change. Prometheus with Grafana, or Grafana Cloud if you would rather not run it yourself, covers this well.</li>

<li>Keep documents and pipeline in the same region, use VPC endpoints for sensitive data, and apply lifecycle rules to originals and extracted output alike.</li>

<li>Keep a golden set of real documents with known correct answers, and run it on every parser change.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can Amazon Textract process multi-page PDFs?</h3>



<p class="wp-block-paragraph">Yes, but only through the asynchronous operations. Synchronous calls accept a single page for PDF and TIFF input. Asynchronous processing handles PDF and TIFF up to 500 MB and 3,000 pages.</p>



<h3 class="wp-block-heading">What is the difference between FORMS and Queries?</h3>



<p class="wp-block-paragraph">FORMS returns whatever key-value pairs it finds from labels printed on the page, so your output keys follow the document. Queries lets you ask for values by natural language question and attach your own alias to each answer, so output keys stay fixed when the layout changes. For a schema you must keep stable, Queries is the safer contract.</p>



<h3 class="wp-block-heading">How much does Amazon Textract cost?</h3>



<p class="wp-block-paragraph">Billing is per page and varies by operation and feature, and requesting several features on one page adds their costs together. Check the current rate card for your region and multiply by pages, not documents. The practical lever is routing: send each document to the narrowest operation that answers your question, and cache raw responses so parser iteration is free.</p>



<h3 class="wp-block-heading">Does Textract handle handwriting?</h3>



<p class="wp-block-paragraph">It detects handwritten text, in English only. Treat handwritten fields as a separate confidence tier with lower auto-accept thresholds, because the variance across writers is much wider than for printed text.</p>



<h3 class="wp-block-heading">Which languages does Amazon Textract support?</h3>



<p class="wp-block-paragraph">Text detection covers English, French, German, Italian, Portuguese, and Spanish. Handwriting, invoices and receipts, identity documents, and Queries are English only. Vertically written text is not supported.</p>



<h3 class="wp-block-heading">Should I use Textract or Bedrock Data Automation?</h3>



<p class="wp-block-paragraph">Bedrock Data Automation is the lower-effort default for mixed document types, with classification and extraction behind a single managed API. Textract fits better at high volume on standardized documents, where per-field confidence and geometry matter for audit, or where extraction needs to be deterministic and independently testable. Combining them is normal.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">If one thing survives this post, make it this: Amazon Textract structured data extraction fails quietly, not loudly. The API returns success while your record is missing its most important field, because a synchronous call read one page of forty, or because a vendor renamed a label, or because a value came back at low confidence and nothing was watching.</p>



<p class="wp-block-paragraph">So build for the silence. Gate at ingest, use asynchronous operations by default, pick the narrowest API per document family, pin your schema with query aliases, store the raw response, and route on per-field confidence backed by validation rules that know what a possible answer looks like. Do that and the OCR really does become the easy part.</p>



<h2 class="wp-block-heading">Need help turning documents into a pipeline you can trust?</h2>



<p class="wp-block-paragraph">Most of the difficulty in document extraction shows up after the proof of concept, once real intake replaces the sample files. That is the part I work on:</p>



<ul class="wp-block-list">
<li>Designing the ingest gate and classification step so each document family reaches the right Textract operation instead of one call doing everything.</li>

<li>Building the asynchronous path properly: S3 events, SNS completion, paginated retrieval, dead-letter handling, idempotent reprocessing.</li>

<li>Replacing brittle FORMS parsing with query aliases or AnalyzeExpense, and validating the change against a golden set of your own documents.</li>

<li>Setting per-field confidence thresholds and validation rules, then wiring a review queue whose corrections feed back into tuning.</li>

<li>Adding observability that catches template drift early: per-field null rates, straight-through rate, and review queue depth on a dashboard.</li>

<li>Getting the cost curve under control through routing, raw-response caching, and lifecycle rules on stored documents and JSON.</li>
</ul>



<p class="wp-block-paragraph">If you have a pipeline misbehaving, send me a sample document with the fields you need and the raw Textract JSON it produced. That is usually enough to see where it is going wrong.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/amazon-textract-structured-data/">Turning External Documents Into Structured Data With Amazon Textract</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Backfilling Historical API Data into S3 Without Silent Gaps</title>
		<link>https://john-nessime.com/blog/devops/backfill-historical-api-data-into-s3/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 24 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Backfill]]></category>
		<category><![CDATA[Boto3]]></category>
		<category><![CDATA[Conditional Requests]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Cursor Pagination]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Exponential Backoff]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Parquet]]></category>
		<category><![CDATA[Partition Projection]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[S3 Inventory]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=268</guid>

					<description><![CDATA[<p>A backfill that exits zero can still be missing a week of data, and nothing will tell you. This is a practical guide to the failure families behind silent gaps: pagination drift under a mutating source, retries that duplicate pages, prefix layouts designed for writes instead of reads, the seam where backfill meets live ingest, and the storage class rules that make mistakes expensive. Includes deterministic key derivation, S3 conditional writes, Athena partition projection, and a per-window manifest pattern that turns completeness into something you can query.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/backfill-historical-api-data-into-s3/">Backfilling Historical API Data into S3 Without Silent Gaps</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The backfill exits zero. Every window processed, nothing red in the logs, the object count in the bucket looks about right for the range. You close the ticket. Weeks later an analyst asks why the chart dips for one week in the middle of the historical range, and nobody can answer, because as far as every system involved is concerned, nothing failed.</p>



<p class="wp-block-paragraph">That&#8217;s the shape of the problem. When you backfill historical API data into S3, the failure to worry about isn&#8217;t the one that throws. It&#8217;s the one that writes fewer records than it should and reports success anyway. Bytes transferred look plausible. Nothing goes red. The data is quietly wrong, and you find out when someone downstream trusts it.</p>



<p class="wp-block-paragraph">This post covers the failure families that produce that outcome and the design decisions that make them impossible rather than merely unlikely: pagination that drifts under a mutating source, retries that write the same page twice under different keys, a prefix layout chosen while thinking about writes instead of reads, the seam where the backfill meets the live pipeline, and the storage class trap that makes a cheap-looking archive expensive to fix.</p>



<h2 class="wp-block-heading">Why a backfill that finishes cleanly can still be wrong</h2>



<p class="wp-block-paragraph">A live pipeline has a natural error signal. Data stops arriving, a lag metric climbs, someone notices. A backfill has none of that. It runs once, against a range nobody is watching in real time, and then it&#8217;s over. The only evidence it leaves behind is a set of objects in a bucket.</p>



<p class="wp-block-paragraph">So the question you need to answer months later isn&#8217;t &#8220;did the job succeed?&#8221; It&#8217;s &#8220;is this window complete, and how do I know?&#8221; If your design can&#8217;t answer that from artifacts on disk, you don&#8217;t have a backfill. You have a large amount of data of unknown provenance.</p>



<h2 class="wp-block-heading">Failure family 1: the source moved while you paginated</h2>



<p class="wp-block-paragraph">This one bites hardest and shows up least. Most REST APIs paginate historical queries with an offset or page number. You ask for page 1, page 2, page 3. Meanwhile the underlying table is still accepting writes.</p>



<p class="wp-block-paragraph">If a record is inserted into the middle of the result set between your page 4 and page 5 requests, everything after it shifts down by one. The row that would have started page 5 is now the last row of page 4, which you already fetched. You request page 5 and that record is gone. Not errored. Gone. Insertions cause skips, deletions cause duplicates, and the job still exits zero.</p>



<h3 class="wp-block-heading">What to do instead</h3>



<ul class="wp-block-list"><li><strong>Prefer keyset or cursor pagination.</strong> If the API offers an opaque cursor or a &#8220;records after ID X&#8221; parameter, use it. The read position is anchored to a value rather than a count, so inserts elsewhere in the set can&#8217;t shift it.</li><li><strong>Sort by an immutable column.</strong> If you&#8217;re stuck with offsets, sort by something that never changes after insert, usually the primary key or the creation timestamp. Sorting by a last-modified column while records are being modified is the worst case: rows physically move within the ordering as you walk it.</li><li><strong>Bound every request on both ends.</strong> Don&#8217;t ask for &#8220;everything before now.&#8221; Ask for a closed interval with an upper bound already in the past. A closed window can&#8217;t grow while you read it.</li><li><strong>Know which timestamp the filter applies to.</strong> Filtering on created-at gives you records that came into existence in that window. Filtering on updated-at gives you records touched in it, which for a mutable source is a different set entirely.</li></ul>



<p class="wp-block-paragraph">That last point causes arguments. Backfill on created-at, then run incremental syncs on updated-at, and your historical range never picks up edits made to old records. That may be correct for your use case. It is not correct by default, and it&#8217;s worth writing down which one you chose.</p>



<h2 class="wp-block-heading">Failure family 2: the retry that wrote the same page twice</h2>



<p class="wp-block-paragraph">Backfills retry. They have to, because you&#8217;re hammering an API for hours and something will time out. The problem is what the retry writes.</p>



<p class="wp-block-paragraph">If your object key contains a UUID, a run identifier or a wall-clock timestamp, a retried page produces a second object with a different key and the same records inside. You now have duplicates that listing will never reveal, because both objects are legitimately present and neither is obviously a copy. Derive the key from the request instead of the run, so the same window plus the same cursor always produces the same key and a retry overwrites rather than accumulates.</p>



<pre class="wp-block-code"><code>import hashlib
import boto3
from botocore.config import Config
from botocore.exceptions import ClientError

s3 = boto3.client(
    "s3",
    config=Config(
        retries={"total_max_attempts": 8, "mode": "standard"},
        max_pool_connections=32,
    ),
)

def page_key(dataset, window, cursor):
    # Deterministic: the same page always maps to the same key.
    token = hashlib.sha256(cursor.encode()).hexdigest()[:16]
    return f"raw/{dataset}/dt={window}/page-{token}.json.gz"

def put_once(bucket, key, body):
    try:
        s3.put_object(Bucket=bucket, Key=key, Body=body, IfNoneMatch="*")
        return "written"
    except ClientError as err:
        if err.response["Error"]["Code"] == "PreconditionFailed":
            return "already_present"
        raise</code></pre>



<p class="wp-block-paragraph">Two things in there are worth explaining rather than copying.</p>



<p class="wp-block-paragraph"><code>IfNoneMatch="*"</code> is a conditional write. S3 accepts the PUT only if no object exists at that key and returns a 412 <code>PreconditionFailed</code> otherwise. It&#8217;s supported on <code>PutObject</code> and <code>CompleteMultipartUpload</code>. That turns &#8220;did I already write this page?&#8221; into a single request with no read-before-write race. If two workers hit the same key concurrently, S3 can instead return a 409 <code>ConditionalRequestConflict</code>, which AWS documents as retryable, so handle the two codes differently: retry on 409, skip on 412.</p>



<p class="wp-block-paragraph">The same thing from the CLI, useful for testing the behaviour by hand:</p>



<pre class="wp-block-code"><code>aws s3api put-object 
  --bucket your-bucket 
  --key raw/orders/dt=WINDOW/page-abc123.json.gz 
  --body page.json.gz 
  --if-none-match "*"</code></pre>



<p class="wp-block-paragraph">The <code>retries</code> block matters for a reason that trips people up. In botocore, <code>max_attempts</code> means different things depending on where you set it: in the AWS config file or the environment variable it counts total requests including the first, but in a <code>Config</code> object it counts retries on top of the first. <code>total_max_attempts</code> exists only in a <code>Config</code> object and always means total, so using it removes the ambiguity. There&#8217;s also an <code>adaptive</code> mode that layers client-side rate limiting on top of <code>standard</code>; AWS still labels it experimental, so I reach for <code>standard</code> plus my own concurrency cap first.</p>



<h2 class="wp-block-heading">Failure family 3: a layout designed for writing, not reading</h2>



<p class="wp-block-paragraph">The prefix structure you pick during the backfill is the one you&#8217;re stuck with. Reorganising later means rewriting every object, and copy requests aren&#8217;t free.</p>



<h3 class="wp-block-heading">Partition on event time, not ingestion time</h3>



<p class="wp-block-paragraph">The tempting layout is the date you ran the job. It&#8217;s easy, it&#8217;s monotonic, and it&#8217;s useless. Every query anyone writes against historical data filters on when the thing happened, not when you happened to fetch it. If your partitions are ingestion dates, every one of those queries becomes a full scan. Use Hive-style keys so query engines prune without extra configuration:</p>



<pre class="wp-block-code"><code>s3://your-bucket/raw/orders/dt=YYYY-MM-DD/page-&lt;token&gt;.json.gz
s3://your-bucket/curated/orders/dt=YYYY-MM-DD/part-&lt;n&gt;.parquet</code></pre>



<p class="wp-block-paragraph">Keep the raw landing zone and the curated zone separate. Raw is whatever the API gave you, unmodified, so you can re-derive everything if your parsing logic turns out to be wrong. Curated is columnar and typed. Merging them saves nothing and costs you the ability to reprocess.</p>



<h3 class="wp-block-heading">Don&#8217;t let the write path throttle itself</h3>



<p class="wp-block-paragraph">S3 documents at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix, with no limit on the number of prefixes. The word doing the work is <em>partitioned</em>. That rate applies once S3 has split the keyspace out for that prefix, and AWS is explicit that the scaling is gradual rather than instant. During the ramp you can see 503 <code>SlowDown</code> responses even when your sustained rate looks well inside the documented ceiling.</p>



<p class="wp-block-paragraph">For a backfill this is mostly good news, since date-partitioned keys spread writes across many prefixes as you work through the range. It becomes a problem when you parallelise <em>within</em> a single day and drive thousands of writes per second into one <code>dt=</code> prefix. Two mitigations: back off with jitter instead of retrying 503s immediately, and shard workers across windows rather than within them.</p>



<h3 class="wp-block-heading">Register partitions without a scan</h3>



<p class="wp-block-paragraph">A backfill can create thousands of partitions at once. <code>MSCK REPAIR TABLE</code> handles that badly. AWS documents that it only adds partitions and never removes them, that it can take a long time on large tables, and that it can time out leaving the catalog partially updated. Partial success with no error is exactly the failure mode this whole post is about.</p>



<p class="wp-block-paragraph">Partition projection sidesteps the catalog. Athena computes the partition list from table properties instead of looking it up, so a backfilled range is queryable the moment the objects exist.</p>



<pre class="wp-block-code"><code>TBLPROPERTIES (
  'projection.enabled'          = 'true',
  'projection.dt.type'          = 'date',
  'projection.dt.format'        = 'yyyy-MM-dd',
  'projection.dt.range'         = 'START_DATE,NOW',
  'projection.dt.interval'      = '1',
  'projection.dt.interval.unit' = 'DAYS',
  'storage.location.template'   = 's3://your-bucket/curated/orders/dt=${dt}/'
)</code></pre>



<p class="wp-block-paragraph">Replace <code>START_DATE</code> with the earliest date in your backfill. The trade-off is worth stating plainly: projection generates every partition in the range whether or not data exists there, so an over-wide range on a sparse dataset wastes planning effort. And if the S3 layout ever stops matching <code>storage.location.template</code>, queries return nothing rather than erroring. Set the range to your actual data, not to a comfortable margin.</p>



<h2 class="wp-block-heading">Failure family 4: the seam between backfill and live pipeline</h2>



<p class="wp-block-paragraph">Two processes write to the same table: a backfill walking backwards through history, and a live ingest walking forwards. They meet somewhere, and that meeting point is where records go missing.</p>



<p class="wp-block-paragraph">The instinct is to make the handoff exact. Backfill up to timestamp T, start streaming from T. Don&#8217;t. Clock skew between your runner and the API, records that arrive late at the source, and rounding in the API&#8217;s own filtering will all conspire to open a hole at exactly T.</p>



<p class="wp-block-paragraph">Overlap deliberately instead. Run the backfill past the point where live ingest started, by a margin generously larger than any plausible source lag, then deduplicate on a natural key during curation. Duplicates you can remove with a query. Gaps you can only fix by going back to the API, and by then the retention window may have closed.</p>



<h2 class="wp-block-heading">Failure family 5: the storage class you can&#8217;t undo</h2>



<p class="wp-block-paragraph">Historical data feels like archive data, so the instinct is to write it straight into a cold class or transition it immediately. Read the mechanics first, because the billing rules for cold classes are structural and pro-rated against you.</p>



<ul class="wp-block-list"><li>S3 Standard-IA and One Zone-IA carry a 30-day minimum storage duration and a 128 KB minimum billable object size.</li><li>Glacier Instant Retrieval and Glacier Flexible Retrieval carry a 90-day minimum duration; Deep Archive carries 180 days. Delete, overwrite or transition earlier and you&#8217;re charged the remainder.</li><li>Objects in Glacier Flexible Retrieval and Deep Archive are billed for 40 KB of extra per-object metadata on top of the object itself.</li></ul>



<p class="wp-block-paragraph">Read those together and the conclusion is uncomfortable. A backfill that writes millions of small JSON pages into an archive class pays a size floor on every one of them, plus metadata overhead, plus a minimum duration that makes fixing the mistake expensive. Those pages are also the objects you&#8217;re most likely to want to reprocess, which is exactly the overwrite that triggers the early-deletion charge.</p>



<p class="wp-block-paragraph">What I&#8217;d actually do: land raw pages in Standard, compact them into larger Parquet files in the curated zone, verify completeness, and only then apply a lifecycle rule to the raw prefix. Compaction is what makes cold storage sane, because it turns a size floor per tiny object into a rounding error on a large one. It also fixes the small-file problem in Athena, which is a separate cost line.</p>



<p class="wp-block-paragraph">One more cost note: if your backfill runs from EC2 in a private subnet, every byte pulled from the API and pushed to S3 may route through a NAT gateway with per-GB processing charges. A gateway VPC endpoint for S3 removes the S3 half of that. It&#8217;s a five-minute change people usually discover after the bill.</p>



<h2 class="wp-block-heading">Proving the backfill is complete</h2>



<p class="wp-block-paragraph">Here&#8217;s the part most backfills skip, and the part that makes everything above verifiable rather than hopeful. Write a manifest object per window, alongside the data, describing what that window contains.</p>



<pre class="wp-block-code"><code>{
  "dataset": "orders",
  "window": "&lt;dt value&gt;",
  "filter_field": "created_at",
  "range_start_inclusive": "&lt;iso8601&gt;",
  "range_end_exclusive": "&lt;iso8601&gt;",
  "pages_written": "&lt;int&gt;",
  "records_written": "&lt;int&gt;",
  "source_reported_total": "&lt;int or null&gt;",
  "last_cursor": "&lt;opaque token&gt;",
  "schema_fingerprint": "&lt;sha256 of sorted field names&gt;"
}</code></pre>



<p class="wp-block-paragraph">The manifest turns completeness into something you can query. A window with no manifest was never finished. A window whose <code>records_written</code> disagrees with <code>source_reported_total</code> is suspect. A window whose <code>schema_fingerprint</code> differs from its neighbours tells you the API changed shape partway through your range, which is otherwise close to undetectable.</p>



<p class="wp-block-paragraph">Write the manifest last, after every page for that window is confirmed in S3. A manifest written optimistically at the start is worse than no manifest at all, because it asserts a completeness nobody checked.</p>



<h2 class="wp-block-heading">Troubleshooting a backfill that already ran</h2>



<ol class="wp-block-list"><li><strong>Count objects per window and look at the shape.</strong> Use S3 Inventory rather than a live <code>ListObjectsV2</code> walk on a large bucket. Plot objects per <code>dt=</code> partition. Real data has a rhythm, weekday peaks and weekend troughs. A window that breaks the rhythm without a business reason is your gap.</li><li><strong>Reconcile record counts against the source.</strong> Many APIs return a total count on the first page of a filtered query. Pull that for a sample of windows and compare with what&#8217;s in S3. If you kept manifests, this is one query.</li><li><strong>Check for duplicate natural keys within a partition.</strong> A <code>GROUP BY</code> on the record ID with <code>HAVING COUNT(*) &gt; 1</code> finds retry duplicates immediately. Duplicates clustered at window boundaries mean pagination drift; duplicates spread evenly mean your keys weren&#8217;t deterministic.</li><li><strong>Look at boundary hours, not whole days.</strong> Cursor drift and clock skew concentrate at the edges of each request window. Aggregate to the hour and the missing slice usually becomes obvious even when daily totals look fine.</li><li><strong>Verify the layout still matches the table definition.</strong> If you&#8217;re using partition projection and a query returns zero rows, the first suspect is <code>storage.location.template</code> not matching the actual prefix, trailing slashes included.</li></ol>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list"><li>Treating an exit code of zero as evidence of completeness.</li><li>Putting a run ID or timestamp in the object key, which makes retries additive instead of idempotent.</li><li>Partitioning by ingestion date because it&#8217;s what the job naturally knows.</li><li>Making the backfill and live-ingest boundary exact instead of overlapping.</li><li>Writing millions of small objects straight into an archive class and meeting the minimum object size and duration rules afterwards.</li><li>Running <code>MSCK REPAIR TABLE</code> over thousands of new partitions and trusting that it completed.</li><li>Backfilling on a last-modified field while assuming you captured creation events.</li><li>Sharding workers within a single day, concentrating writes on one prefix and inviting 503s.</li></ul>



<h2 class="wp-block-heading">Best practices when you backfill historical API data into S3</h2>



<ul class="wp-block-list"><li>Make every window independently re-runnable. If re-running one day is routine rather than scary, you&#8217;ll actually fix problems when you find them.</li><li>Derive object keys deterministically from window plus cursor, and use conditional writes so a retry is a no-op.</li><li>Bound every API request on both ends, and never let the upper bound be &#8220;now&#8221;.</li><li>Write raw responses untouched before parsing anything. Parsing bugs are common; re-fetching a closed API window often isn&#8217;t possible.</li><li>Emit a manifest per window, written last, and reconcile it against source counts.</li><li>Compact to Parquet before applying any lifecycle transition.</li><li>Run the job somewhere it can survive for hours. A long backfill on a laptop over hotel wifi is a bad plan; a small dedicated VPS from a provider like Contabo or InterServer, or a spot-priced EC2 instance, costs little and won&#8217;t sleep halfway through.</li><li>Watch progress somewhere other than the terminal. Pushing records-per-window into CloudWatch or Grafana gives you a curve you can eyeball, and an unusually flat stretch is a gap you can catch while the source data is still retrievable.</li></ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Should I write raw JSON or convert to Parquet during the backfill?</h3>



<p class="wp-block-paragraph">Both, in two zones. Write the raw response body first so the fetch step has no dependency on your schema assumptions, then convert as a separate job. Converting inline couples an expensive, rate-limited API walk to a parsing step that will eventually throw on an unexpected field, and you don&#8217;t want to re-fetch years of data because of a type error.</p>



<h3 class="wp-block-heading">How large should each object be?</h3>



<p class="wp-block-paragraph">In the raw zone, one object per API page is fine and keeps idempotency simple. In the curated zone, aim for files in the tens to low hundreds of megabytes. Query engines pay a fixed overhead per file, so thousands of tiny Parquet files scan slower and cost more than a few large ones holding identical data.</p>



<h3 class="wp-block-heading">Can I run the backfill in parallel across date windows?</h3>



<p class="wp-block-paragraph">Yes, and that&#8217;s the right axis, because each window is independent and lands on its own prefix. The binding constraint is almost always the source API&#8217;s rate limit rather than S3. Add a global concurrency cap and honour any documented limit or <code>Retry-After</code> header. Getting your credentials throttled or suspended mid-backfill is a worse outcome than finishing slowly.</p>



<h3 class="wp-block-heading">Do I need AWS Glue or Airflow for this?</h3>



<p class="wp-block-paragraph">Not to start. A single script that takes a window as an argument, writes pages and a manifest, and is safe to re-run covers most backfills. Orchestration earns its place when you need retries across hundreds of windows, dependency ordering, or a visible run history. Glue and Step Functions fit AWS-native stacks; Airflow gives you more control and more to operate. Pick after the script works, not before.</p>



<h3 class="wp-block-heading">What if the API caps how far back I can query?</h3>



<p class="wp-block-paragraph">Then the retention boundary is your real deadline and it should drive priority. Fetch the oldest reachable data first, not the most recent, because the newest data is still available tomorrow and the oldest may not be. If the vendor offers a bulk export or reporting endpoint separate from the standard API, it&#8217;s usually faster and less rate-limited than paginating.</p>



<h3 class="wp-block-heading">Does any of this change with S3-compatible storage instead of AWS?</h3>



<p class="wp-block-paragraph">The layout, idempotency and manifest patterns carry over unchanged to Cloudflare R2, Backblaze B2, MinIO and similar. What varies is API-level feature support. Conditional writes with <code>If-None-Match</code>, request-rate behaviour and storage class semantics differ by provider, so verify the specific behaviour you&#8217;re depending on against that provider&#8217;s documentation rather than assuming parity with S3.</p>



<h3 class="wp-block-heading">How do I know the backfill is actually finished?</h3>



<p class="wp-block-paragraph">When every expected window has a manifest, every manifest&#8217;s record count reconciles against the source, and a duplicate check on natural keys returns nothing. Not when the script exits.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A backfill isn&#8217;t finished when the job exits. It&#8217;s finished when you can prove the range is complete without re-reading the source. Deterministic keys, conditional writes, closed windows, manifests and deliberate overlap all exist to make that proof cheap.</p>



<p class="wp-block-paragraph">If you&#8217;re about to backfill historical API data into S3, spend the extra hour on the key derivation and the manifest before you start the run. The alternative is learning about a gap from a chart, months later, with the source retention window already closed behind you.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a hand with a backfill?</h2>



<p class="wp-block-paragraph">This is a large part of what I do. If you&#8217;re staring at a historical range and an API with a rate limit, I can help with:</p>



<ul class="wp-block-list"><li>Designing an idempotent, re-runnable backfill job with deterministic keys and per-window manifests, so a failed day is a one-command fix.</li><li>Choosing a partition layout that serves your actual query patterns, and setting up Athena or Glue tables with partition projection so thousands of backfilled partitions become queryable without catalog scans.</li><li>Auditing a backfill that already ran, reconciling record counts against the source and finding the gaps and duplicates before your dashboards do.</li><li>Handling awkward source APIs: offset-only pagination, aggressive rate limits, undocumented result caps, schema drift partway through a historical range.</li><li>Compaction and lifecycle design, so cold storage actually saves money instead of hitting minimum object size and minimum duration charges.</li><li>Wiring the seam between the backfill and your live pipeline so there&#8217;s no gap at the handover, plus the monitoring that would have caught it.</li></ul>



<p class="wp-block-paragraph">Send me something concrete: a sample API response, your current key layout, or the object counts per partition. That&#8217;s usually enough to spot where the gap is.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/backfill-historical-api-data-into-s3/">Backfilling Historical API Data into S3 Without Silent Gaps</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>As-Planned vs As-Built Analysis: Building a Platform That Survives Cross-Examination</title>
		<link>https://john-nessime.com/blog/technical-guides/as-planned-vs-as-built-analysis-platform/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 24 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Case Studies]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[As-Built Schedule]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[Capital Projects]]></category>
		<category><![CDATA[Construction Technology]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Lineage]]></category>
		<category><![CDATA[Delay Analysis]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Extension of Time]]></category>
		<category><![CDATA[Forensic Schedule Analysis]]></category>
		<category><![CDATA[Legal Tech]]></category>
		<category><![CDATA[OCR]]></category>
		<category><![CDATA[PostgreSQL]]></category>
		<category><![CDATA[Primavera P6]]></category>
		<category><![CDATA[Project Controls]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[XER Files]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=274</guid>

					<description><![CDATA[<p>Most as-planned vs as-built analysis compares the baseline to the last P6 update and calls the result an as-built. It isn't one. This is how to build the data layer underneath a delay analysis: versioned XER ingestion, activity identity across renumbering, a first-appearance table that proves when every actual date entered the record, calendar-safe float, and record linking that proposes candidates instead of asserting cause.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/as-planned-vs-as-built-analysis-platform/">As-Planned vs As-Built Analysis: Building a Platform That Survives Cross-Examination</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question that kills a delay analysis is never about methodology. It is five words from the other side&#8217;s expert: &#8220;which update did that come from?&#8221;</p>



<p class="wp-block-paragraph">Picture the experts&#8217; meeting. Your exhibit shows activity C-2140, structural steel erection to level four, finishing well behind where the baseline put it. He has the same XER files you do. He opens update 19 and points out that C-2140 carries no actual finish. Same in update 20. In update 21 an actual finish appears, and the date on it sits three weeks in the past, inside the window you already closed out.</p>



<p class="wp-block-paragraph">So which file is the record? And if one activity was back-filled a month after the fact, what does that say about the other four thousand?</p>



<p class="wp-block-paragraph">That is the failure an <strong>as-planned vs as-built analysis</strong> has to survive, and no amount of care inside the Excel workbook fixes it. It is a data problem before it is a scheduling problem. This post covers the platform underneath the analysis: how to ingest a long series of Primavera P6 updates, how to prove where every actual date came from, and how to produce output a tribunal can audit instead of having to trust.</p>



<h2 class="wp-block-heading">The as-built is not a file, and treating it as one is the invisible failure</h2>



<p class="wp-block-paragraph">Almost every quick as-planned vs as-built comparison does the same thing: take the approved baseline, take the last update in the sequence, join on activity ID, subtract the dates, sort by variance. It produces a clean-looking table in an afternoon.</p>



<p class="wp-block-paragraph">It is also the weakest possible construction of an as-built, for three reasons that never show up in the output.</p>



<ul class="wp-block-list">
<li><strong>A P6 update is a forecast document.</strong> It was produced to tell the employer when the job would finish, not to record what happened. Actual dates are a by-product of that exercise, entered by a planner working to a monthly deadline.</li>

<li><strong>The last update has been edited the most.</strong> By the end of a long job it has usually been re-baselined, partially renumbered, had calendars swapped, had activities merged or deleted. Every one of those edits is invisible in a two-file comparison.</li>

<li><strong>Actual dates get back-filled.</strong> A date typed in month 21 describing month 19 is not contemporaneous evidence in the same sense as a date typed in month 19. It might still be right. It is a different quality of record, and the difference is exactly what gets probed in cross-examination.</li>
</ul>



<p class="wp-block-paragraph">AACE International&#8217;s Recommended Practice 29R-03 places the as-planned versus as-built family in its observational, static group: you compare a planned network against an as-built without inserting or removing delay events. The SCL Delay and Disruption Protocol, second edition, is blunter about what makes any of it work, pushing hard on agreeing a record-keeping regime up front and never overwriting a programme version. Both point at the same requirement. You need the whole sequence of updates, intact, and you need to know which one told you what.</p>



<p class="wp-block-paragraph">The platform&#8217;s whole job is to make that sequence queryable.</p>



<h2 class="wp-block-heading">A worked example: what thirty-four updates actually look like</h2>



<p class="wp-block-paragraph">Take a job shaped like this. Three years of monthly updates, call it thirty-four XER exports. Around four thousand activities in the current programme. Three baseline revisions, two agreed and one submitted but never accepted. Somewhere near month fourteen the contractor moved most trades from a five-day to a six-day calendar to recover. The numbers are illustrative rather than from any particular project, but the shape is ordinary.</p>



<p class="wp-block-paragraph">Run the naive comparison and you get a variance table. Run the sequence properly and four separate questions fall out, none of which the variance table can answer:</p>



<ol class="wp-block-list">
<li>Which activities existed in the baseline, disappeared mid-job, and came back under a different activity ID?</li>

<li>For each actual date in the final update, in which update did that value first appear, and how far behind the event was it?</li>

<li>When float moved, was it because work moved, or because someone changed a calendar or a duration?</li>

<li>What contemporaneous record sits underneath each actual date, and does it agree?</li>
</ol>



<p class="wp-block-paragraph">Those four questions are the platform. Everything below is how to answer each one without hand-checking four thousand rows thirty-four times.</p>



<h2 class="wp-block-heading">Failure family one: activity identity across versions</h2>



<p class="wp-block-paragraph">An XER file is tab-delimited text. It opens in a text editor. The structure is a header line beginning <code>ERMHDR</code>, then a run of tables, each introduced by <code>%T</code> with the table name, a <code>%F</code> line carrying the field names, and <code>%R</code> lines carrying rows, closed by <code>%E</code>. The tables you care about first are <code>PROJECT</code>, <code>TASK</code>, <code>TASKPRED</code> and <code>CALENDAR</code>.</p>



<pre class="wp-block-code"><code>ERMHDR	19.12	2026-01-15	Project	admin	Admin User	dbxDatabaseNoName	Project Management	USD
%T	TASK
%F	task_id	proj_id	wbs_id	clndr_id	task_code	task_name	status_code	act_start_date	act_end_date	target_start_date	target_end_date	total_float_hr_cnt
%R	1001	100	200	1	C-2140	Structural Steel Erection L4	TK_Complete	...
%E</code></pre>



<p class="wp-block-paragraph">Here is the trap. <code>task_id</code> is the primary key of the P6 database the file was exported from. It is not a stable identifier for an activity across time. If the project was ever copied between databases, restored, or exported from a different P6 instance, the same physical activity can carry a different <code>task_id</code>. Worse, two XERs from different sources can reuse the same <code>task_id</code> for completely unrelated activities.</p>



<p class="wp-block-paragraph"><code>task_code</code> is the Activity ID a human sees, and it is stable right up until someone renumbers. Neither field alone gives you identity. So do not pick one. Build an identity resolution layer and keep the evidence for every decision it makes:</p>



<ul class="wp-block-list">
<li>Use <code>(proj_id, task_code)</code> as the working join key, scoped to a single source database.</li>

<li>Store <code>task_id</code>, <code>task_name</code> and <code>wbs_id</code> on every snapshot row so a rename or a WBS move is detectable after the fact.</li>

<li>Emit an explicit <em>unresolved</em> record when an activity vanishes or appears mid-sequence. Do not guess.</li>
</ul>



<p class="wp-block-paragraph">That last point is the one people get wrong. When C-2140 disappears in update 22 and an activity with the same name but the ID C-2140A appears in update 23, there are three candidate explanations: deletion and replacement, renumbering, or a split. The platform cannot know which. A tool that silently picks one is worse than a tool that flags it, because the analyst never learns there was a question. Surface it, hand it to a human, record the decision and the reason.</p>



<h2 class="wp-block-heading">Failure family two: date provenance, and the table that makes the whole thing worth building</h2>



<p class="wp-block-paragraph">If you build one thing, build this. For every activity and every date field, record the first update in which that value appeared, and every subsequent change to it.</p>



<p class="wp-block-paragraph">Assume a snapshot table with one row per activity per update, appended and never modified. A window function does the work:</p>



<pre class="wp-block-code"><code>-- First appearance of each distinct actual finish value,
-- and how far the reported date lags the update that reported it.
WITH changes AS (
  SELECT
    task_code,
    version_no,
    data_date,
    act_end_date,
    LAG(act_end_date) OVER (
      PARTITION BY task_code ORDER BY version_no
    ) AS prev_act_end
  FROM activity_snapshot
)
SELECT
  task_code,
  version_no          AS first_reported_in_update,
  act_end_date        AS reported_actual_finish,
  data_date           AS update_data_date,
  data_date::date - act_end_date::date AS reporting_lag_days
FROM changes
WHERE act_end_date IS NOT NULL
  AND (prev_act_end IS NULL OR prev_act_end &lt;&gt; act_end_date)
ORDER BY reporting_lag_days DESC;</code></pre>



<p class="wp-block-paragraph">Three things arrive at once. You can answer the question that opened this post, for any activity, in a second: C-2140&#8217;s actual finish first appeared in update 21, carrying a date twenty-three days behind that update&#8217;s data date.</p>



<p class="wp-block-paragraph">The distribution of that lag column is then a finding in its own right. If most activities report within days and one subcontractor&#8217;s work consistently reports a month late, that is not a rounding artefact, it is a records issue you can evidence. And any actual date that changes after it was first stated deserves a hard look. Actuals are supposed to lock. A value that moves has either been corrected, which should have a paper trail, or overwritten, which should not have happened.</p>



<p class="wp-block-paragraph">P6 ships a Schedule Comparison tool, the one that used to be called Claim Digger and now lives inside Visualizer. It does a field-by-field diff and dumps a large HTML table, which is genuinely useful for checking one submitted update against the last. That is what it was built for. It was not built to reason across thirty-four files at once, and pushing its output into Excel does not change that. Deltek&#8217;s Acumen Fuse and Steelray&#8217;s Delay Analyzer go further, with what the vendors describe as half-step analysis, separating progress from scope revisions across successive versions. If your instruction allows the licence and the timetable allows the learning curve, look at them seriously before writing any code. Neither is priced publicly, both target enterprise project controls teams, and neither does the records-linking work in the last section for you.</p>



<h2 class="wp-block-heading">Failure family three: calendars, and float that quietly means nothing</h2>



<p class="wp-block-paragraph">Durations and float in an XER are stored in hours. <code>total_float_hr_cnt</code> holds hours, not days. A value of 40 is five days only if the activity runs an eight-hour calendar. Divide everything by eight and a job with mixed shift patterns produces float numbers that are confidently wrong.</p>



<p class="wp-block-paragraph">The calendar itself lives in the <code>CALENDAR</code> table, in a <code>clndr_data</code> field holding a proprietary parenthesised blob describing work weeks, exceptions and holidays. It is the hardest part of the format to parse correctly, and getting it wrong produces durations and dates that look plausible and are not.</p>



<p class="wp-block-paragraph">Two rules follow, and both are about the read path, not the maths.</p>



<ul class="wp-block-list">
<li>Store float and durations in hours exactly as exported. Convert at presentation time, using that activity&#8217;s own calendar, and print the calendar name next to the number. If you cannot resolve the calendar, print hours and say so.</li>

<li>Track calendar assignment as a versioned attribute, the same as any date. A change from a five-day to a six-day calendar changes every float figure downstream of it, with no change to logic and no change to progress. An activity can go from critical to non-critical because of an administrative edit, and a variance table will attribute that shift to the works.</li>
</ul>



<p class="wp-block-paragraph">This is also why an as-built critical path is not simply the chain with zero float in the last update. Retrospective longest path and as-planned versus as-built windows analysis exist as separate methods precisely because the float figures in a progressed programme are a function of how the programme was maintained.</p>



<h2 class="wp-block-heading">Failure family four: separating movement from editing</h2>



<p class="wp-block-paragraph">Every delta between two consecutive updates falls into one of a small number of categories, and collapsing them into a single &#8220;variance&#8221; number is where analyses lose their defensibility. Classify each change explicitly:</p>



<ul class="wp-block-list">
<li><strong>Progress:</strong> an actual start or finish appeared, or remaining duration reduced consistent with work done.</li>

<li><strong>Duration change:</strong> original or remaining duration edited without corresponding progress.</li>

<li><strong>Logic change:</strong> a relationship added, removed, retyped, or its lag altered.</li>

<li><strong>Calendar change:</strong> activity reassigned, or the calendar definition itself edited.</li>

<li><strong>Constraint change:</strong> a date constraint applied, moved or removed.</li>

<li><strong>Scope change:</strong> activity added or deleted.</li>
</ul>



<p class="wp-block-paragraph">Six buckets, one row per change, with the before value, the after value and the version pair. Once that table exists, a lot of the argument stops being an argument. &#8220;The completion date moved out eighteen days in window 12&#8221; becomes &#8220;of which fourteen days sit with progress on the critical chain and four with two logic edits applied in the same submission.&#8221; The second version is the one that gets tested and holds.</p>



<p class="wp-block-paragraph">Logic changes deserve their own view. Pull <code>TASKPRED</code> for every version, key each relationship on the predecessor and successor activity codes plus the relationship type, and diff the sets. Relationships that appear late in a job, on activities that are already in progress, are worth reading one by one.</p>



<h2 class="wp-block-heading">Failure family five: joining the schedule to the records</h2>



<p class="wp-block-paragraph">The schedule tells you when. It never tells you why. Cause comes from daily reports, site instructions, RFIs, correspondence, minutes, inspection records, weather data, and delivery notes. That material arrives as scanned PDFs and email exports, and it does not carry activity IDs. Nobody has ever written &#8220;C-2140&#8221; on a site diary.</p>



<p class="wp-block-paragraph">So the join key is not the activity ID. It is a fuzzy composite of date, location or area, and trade or discipline. The practical approach:</p>



<ol class="wp-block-list">
<li>Extract structured fields from each record: document date, report date, area, trade, headcount if present, and the narrative text. Text extraction plus a layout-aware OCR pass handles most of it. Handwriting on older site diaries will not fully automate, and you should budget for that.</li>

<li>Normalise area and trade against the schedule&#8217;s own coding. Activity codes and WBS paths are usually the best available crosswalk, since they encode area and discipline already.</li>

<li>Generate <em>candidate</em> links: for each activity&#8217;s as-built window, every record whose date falls inside it and whose area or trade matches.</li>

<li>Have a human confirm, reject or annotate each candidate. Store who decided and when.</li>
</ol>



<p class="wp-block-paragraph">Step four is not optional and it is not a nicety. A machine-proposed link is a search result. Causation is an opinion, and the expert has to own it. Build the tool so it accelerates the search and refuses to state the conclusion, and you get the productivity without handing opposing counsel an argument about black boxes.</p>



<h2 class="wp-block-heading">The data model, in about six tables</h2>



<p class="wp-block-paragraph">The grain that makes everything else easy: one immutable row per activity per schedule version. Nothing is ever updated in place.</p>



<pre class="wp-block-code"><code>CREATE TABLE schedule_version (
  version_id     bigserial PRIMARY KEY,
  version_no     integer     NOT NULL,   -- ordinal in the sequence
  source_file    text        NOT NULL,
  file_sha256    char(64)    NOT NULL,   -- proves the file was not altered
  p6_export_ts   timestamptz,            -- from the ERMHDR line
  data_date      timestamptz NOT NULL,
  ingested_at    timestamptz NOT NULL DEFAULT now(),
  notes          text
);

CREATE TABLE activity_snapshot (
  version_id           bigint  NOT NULL REFERENCES schedule_version,
  task_code            text    NOT NULL,
  task_id              bigint  NOT NULL,
  task_name            text,
  wbs_path             text,
  clndr_id             bigint,
  status_code          text,
  target_start_date    timestamptz,
  target_end_date      timestamptz,
  act_start_date       timestamptz,
  act_end_date         timestamptz,
  total_float_hr_cnt   numeric,
  PRIMARY KEY (version_id, task_code)
);</code></pre>



<p class="wp-block-paragraph">Add <code>relationship_snapshot</code> keyed on <code>(version_id, pred_task_code, succ_task_code, rel_type)</code>, a <code>calendar_snapshot</code> holding the raw <code>clndr_data</code> alongside your parsed working pattern, a <code>record_document</code> table for the contemporaneous material, and a <code>record_link</code> table carrying the human decision on each candidate link.</p>



<p class="wp-block-paragraph">Thirty-four updates at four thousand activities is under a hundred and fifty thousand rows. This is not a big-data problem and it should not be built like one. PostgreSQL on a single machine, with the parsing done in Python, will run every query in this post fast enough that you stop thinking about it.</p>



<p class="wp-block-paragraph">On the parsing side, an XER is simple enough to read with the standard library, and there are open-source parsers such as PyP6Xer if you would rather not write the tokeniser yourself. Whichever route you take, keep the raw file. Hash it on ingest, store the hash in <code>schedule_version</code>, and never write to the original. Chain of custody over the source files is the cheapest credibility you will ever buy.</p>



<h2 class="wp-block-heading">Where to run it, and the confidentiality problem nobody budgets for</h2>



<p class="wp-block-paragraph">A modest VPS is plenty. Contabo and InterServer both sell machines with more than enough memory and disk for a job this size, and the specification matters far less than the question you should ask first: does your confidentiality undertaking actually permit the disclosure material to sit on that machine, in that jurisdiction? Often the answer is no, and the analysis has to run on a workstation inside the client&#8217;s own environment. Find that out before you build anything.</p>



<p class="wp-block-paragraph">Three practical points that follow from the material being privileged rather than ordinary project data:</p>



<ul class="wp-block-list">
<li><strong>Full-disk encryption on anything that holds the files</strong>, including the laptop you take to the hearing. This is table stakes and it is still routinely skipped.</li>

<li><strong>A VPN for untrusted networks.</strong> You will open the data room from a hotel or an airport at some point. NordVPN and Surfshark both handle that case. It protects the transport; it does nothing about where the data comes to rest, so do not let it substitute for the jurisdiction question above.</li>

<li><strong>A disposal plan.</strong> Undertakings usually require destruction or return at the close of the engagement, and deleting a file does not destroy it. Overwrite-based tools such as O&amp;O SafeErase cover the workstation case. If the volume was encrypted from the start, destroying the key is faster and cleaner than overwriting the data, and easier to certify.</li>
</ul>



<p class="wp-block-paragraph">If cloud storage is permitted, write-once object storage with a retention lock suits the ingested originals well. It gives you an immutability story that does not rest on your own discipline.</p>



<h2 class="wp-block-heading">Troubleshooting the ingest</h2>



<ul class="wp-block-list">
<li><strong>The parser dies on a decode error.</strong> XER files are not reliably UTF-8, particularly when they came from a database with a non-English locale. Try a single-byte fallback encoding before assuming the file is corrupt, and log which encoding succeeded against the version record.</li>

<li><strong>Row and field counts do not match.</strong> Activity names can contain characters that confuse a naive split, and some exports wrap fields inconsistently. Parse defensively on field count per <code>%F</code> line rather than assuming.</li>

<li><strong>Duplicate <code>task_code</code> in one file.</strong> One XER can carry several projects plus the shared global data they depend on. Always scope by <code>proj_id</code> and confirm you have the right project before anything else.</li>

<li><strong>Activity counts drop between updates for no reason.</strong> Check whether the export was filtered or taken at a WBS level rather than the full project. A partial export looks exactly like a deletion event in your diff.</li>

<li><strong>Dates shift by a few hours.</strong> P6 stores times, not just dates, and calendar start hours vary. Normalise to the activity calendar&#8217;s working day before comparing, or every finish looks a shift early.</li>

<li><strong>Float figures look wrong across the board.</strong> Check the hours-versus-days conversion and the calendar assignment before checking anything else. It is almost always one of those two.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating the final update as the as-built without stating that assumption anywhere in the report.</li>

<li>Joining on <code>task_id</code> because it is the primary key, and inheriting whatever database history came with the file.</li>

<li>Letting the platform resolve an ambiguous activity match silently, so the analyst never sees the ambiguity existed.</li>

<li>Converting float to days with a fixed divisor, then reporting a single variance figure that mixes progress with programme edits.</li>

<li>Automating the record-to-activity link all the way to a stated cause, which turns an evidence tool into a target.</li>

<li>Modifying source XERs in place, then having no way to demonstrate they are as received.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>Hash every source file on ingest and record the hash next to the version. It takes one line and it answers a question you will otherwise answer badly.</li>

<li>Make every table append-only. Corrections become new rows with a reason, never overwrites.</li>

<li>Build the first-appearance view early. It reframes the analysis from &#8220;what do the files say&#8221; to &#8220;when did the files start saying it&#8221;, which is the more defensible question.</li>

<li>Make every exhibit reproducible from one query, so a challenged figure traces back to a file and a row rather than a spreadsheet cell.</li>

<li>Keep a written record of every judgement call the platform surfaced and a human resolved. That log is often more useful in the hearing than the analysis itself.</li>

<li>Validate against the commercial tools where you can. If Acumen Fuse or Schedule Comparison disagrees with your diff on a given version pair, one of you is wrong and you want to know which before the report goes out.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is as-planned vs as-built analysis still accepted in construction disputes?</h3>



<p class="wp-block-paragraph">Yes, as one of a recognised family of methods. AACE Recommended Practice 29R-03 includes it in its taxonomy as an observational method, and the SCL Protocol&#8217;s second edition deliberately stopped naming a single preferred technique for retrospective analysis, setting out selection factors instead. Acceptance in a given dispute turns on the records available, the contract and the forum, not on the method&#8217;s reputation in the abstract.</p>



<h3 class="wp-block-heading">How many schedule updates do I need before building a platform is worth it?</h3>



<p class="wp-block-paragraph">Roughly: below about eight updates, a careful analyst with a workbook is faster. Above twenty, manual comparison stops being reliable, not because people are careless but because the number of pairwise comparisons grows past what anyone can hold. The crossover in practice sits somewhere in the low teens, and it moves earlier if there are multiple baselines or a renumbering event.</p>



<h3 class="wp-block-heading">Can I do as-planned vs as-built analysis without a Primavera P6 licence?</h3>



<p class="wp-block-paragraph">For the data work, yes. An XER is plain text and parses without P6 installed, and browser-based and open-source viewers exist. You will still want licensed access somewhere in the engagement for anything that requires recalculating the network, because reproducing P6&#8217;s scheduling engine faithfully is not a project you want inside a disputes timetable.</p>



<h3 class="wp-block-heading">What is the difference between as-planned vs as-built and a windows analysis?</h3>



<p class="wp-block-paragraph">The simple form of as-planned versus as-built compares planned dates against actual dates observationally, without necessarily using network logic. A windows analysis divides the project into periods, usually monthly, and examines the contemporaneous critical path and the critical delay in each period before investigating the cause. Windows demands more of the records and generally carries more weight where the programme was maintained properly. This platform supports both, because both need the same version history underneath.</p>



<h3 class="wp-block-heading">What do I do when the contractor&#8217;s updates were not maintained properly?</h3>



<p class="wp-block-paragraph">State it, quantify it, and let it drive method selection. That is exactly what the first-appearance table is for: reporting lag, actuals that moved after being stated, out-of-sequence progress and unexplained logic edits are all measurable. Recommended practice treats recreating a schedule from records as a last resort, used when contemporaneous data is unavailable, and recreated schedules carry less evidential weight. Showing the deficiency with numbers is stronger than asserting it.</p>



<h3 class="wp-block-heading">Should I build this or buy commercial delay analysis software?</h3>



<p class="wp-block-paragraph">Buy for schedule diagnostics and version comparison. That problem is well solved and the established tools have years of edge cases behind them. Build for the parts specific to your instruction: the record linking, the exhibit generation in your house format, and the audit trail. The hybrid is usually right, and the deciding factor is more often the licence and the timescale than the technical fit.</p>



<h3 class="wp-block-heading">Does the platform need to reproduce P6&#8217;s critical path calculation?</h3>



<p class="wp-block-paragraph">No, and it should not try. Read the float and date values P6 already calculated and stored in each export, and treat them as evidence of what the programme said at that moment. Anywhere you genuinely need a recalculated network, do it in the scheduling tool so the result is reproducible by anyone with the same file.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">A credible as-planned vs as-built analysis is not a better comparison between two files. It is a complete, immutable, queryable history of what every schedule update said and when it started saying it. Build the version history and the first-appearance table first, and the exhibits fall out of it. Build the exhibits first and you will spend the hearing defending a spreadsheet whose provenance you cannot reconstruct.</p>



<p class="wp-block-paragraph">The scheduling judgement stays with the expert. The platform&#8217;s job is narrower and duller: make sure that when someone asks where a date came from, the answer takes one query and not one week.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need the data side of this built?</h2>



<p class="wp-block-paragraph">I build the engineering layer underneath delay and quantum work, so the expert spends their time on opinion rather than on data wrangling. That usually looks like:</p>



<ul class="wp-block-list">
<li>Ingesting a full sequence of P6 XER or XML updates into a versioned, append-only PostgreSQL model with hashing and chain of custody on every source file.</li>

<li>Building the first-appearance and reporting-lag views that show when each actual date entered the record, and flagging actuals that moved after being stated.</li>

<li>Classifying every between-update change into progress, duration, logic, calendar, constraint or scope, with before and after values on each row.</li>

<li>Extracting structured fields from daily reports, RFIs and correspondence with OCR and document processing, then generating candidate activity links for an analyst to confirm or reject.</li>

<li>Producing exhibit-grade output in Power BI or as static deliverables, where every figure traces back to a query, a file and a row.</li>

<li>Standing it up on infrastructure that fits the confidentiality regime you work under, whether that is a locked-down VPS or a machine inside the client&#8217;s own environment.</li>
</ul>



<p class="wp-block-paragraph">If you have a stack of XERs and a deadline, send me two consecutive updates and whatever the record-keeping looked like, and I will tell you what the sequence can and cannot support.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/as-planned-vs-as-built-analysis-platform/">As-Planned vs As-Built Analysis: Building a Platform That Survives Cross-Examination</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Customer Sentiment Analysis From CRM and Support Data: Building a Score You Can Actually Trust</title>
		<link>https://john-nessime.com/blog/technical-guides/customer-sentiment-analysis-crm-support-data/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 22 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Comprehend]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[CRM Analytics]]></category>
		<category><![CDATA[Customer Support Analytics]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[EU AI Act]]></category>
		<category><![CDATA[NLP]]></category>
		<category><![CDATA[PII Redaction]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[Sentiment Analysis]]></category>
		<category><![CDATA[SQL]]></category>
		<category><![CDATA[Zendesk]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=250</guid>

					<description><![CDATA[<p>Most customer sentiment analysis pipelines run perfectly and still produce a number nobody should act on. Here are the failure families that cause it, from labels stamped on the wrong message to averages taken over a scale that was never numeric, plus a pipeline shape and a legal boundary worth knowing before you ship.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/customer-sentiment-analysis-crm-support-data/">Customer Sentiment Analysis From CRM and Support Data: Building a Score You Can Actually Trust</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The CX lead pings you on a Thursday afternoon: &#8220;Sentiment has been sitting at 3.2 for four months. Is the pipeline broken?&#8221;</p>



<p class="wp-block-paragraph">You check. Every extraction job ran. Every ticket got a label. No rows dropped, no schema drift, no dead letters. Nothing is broken. The number is just useless, and it has been useless since the day it shipped.</p>



<p class="wp-block-paragraph">That is the normal outcome for customer sentiment analysis built on CRM and support data. The connectors work, the model returns labels, the dashboard renders on schedule, and the output still cannot tell anyone whether customers are getting happier or angrier. These are not pipeline failures. They are measurement failures, which is exactly why your monitoring never catches them.</p>



<p class="wp-block-paragraph">This post walks through the failure families I check first when a sentiment programme is producing numbers nobody trusts: where the label gets stamped, what happens when you average an ordinal scale, who is missing from the data entirely, why a general-purpose model misreads support English, where the CRM join breaks, and one regulatory boundary that catches more teams than it should. Then a pipeline shape that holds up, and the checks worth running before anyone puts this on a slide.</p>



<h2 class="wp-block-heading">What the score is actually measuring</h2>



<p class="wp-block-paragraph">Before anything else, get specific about the unit of analysis. &#8220;Customer sentiment&#8221; is a phrase people use for at least four different measurements:</p>



<ul class="wp-block-list">
<li><strong>Message sentiment</strong> — the tone of one comment in one thread.</li>

<li><strong>Ticket sentiment</strong> — usually the first inbound message, sometimes a rollup.</li>

<li><strong>Contact sentiment</strong> — one human&#8217;s tone across all their tickets.</li>

<li><strong>Account sentiment</strong> — the thing the business actually cares about, and the one nobody computes correctly.</li>
</ul>



<p class="wp-block-paragraph">Most stacks compute the second and label it the fourth. That single substitution causes more bad decisions than any model choice you will make downstream. A model with mediocre accuracy applied at the right unit beats an excellent model applied at the wrong one.</p>



<h2 class="wp-block-heading">Failure one: the label is stamped at the wrong moment</h2>



<p class="wp-block-paragraph">This is the expensive one, and it is invisible from the pipeline side because nothing errors.</p>



<p class="wp-block-paragraph">Zendesk&#8217;s intelligent triage is the clearest example because the documentation is honest about it. Zendesk states that most intelligent triage classifications are based on a ticket&#8217;s first message only, with an admin-configurable option to update topic and sentiment from the latest end-user message. If nobody turned that option on, your &#8220;customer sentiment&#8221; field is a permanent record of how annoyed someone was in their opening paragraph.</p>



<p class="wp-block-paragraph">Think about what that does to a trend line. The opening message of a support ticket is negative almost by definition, because people do not open tickets to say things are going well. What you want to know is whether the interaction <em>ended</em> well. A ticket that opens furious and closes with a thank-you is a support win, and a first-message-only score records it as a loss forever.</p>



<p class="wp-block-paragraph">Two consequences follow that people miss:</p>



<ul class="wp-block-list">
<li>Your sentiment metric is structurally incapable of moving in response to anything the support team does. Hire better agents, cut response time in half, rewrite your macros: the number will not budge, because it was set before any of that happened.</li>

<li>It <em>will</em> move in response to marketing, onboarding and product changes upstream, because those change who opens tickets and how angry they are on arrival. So the metric drifts for reasons the team watching it cannot influence.</li>
</ul>



<p class="wp-block-paragraph">Zendesk also documents that when an agent corrects the sentiment field on a ticket, the correction does not retrain the model. Worth knowing before you build a human-review workflow on the assumption that it feeds back into anything. It gives you a cleaner column, not a better classifier.</p>



<p class="wp-block-paragraph">The fix is to decide your stamping policy explicitly and store it. Score the first inbound message, score the last inbound message, and store both, along with the delta. The delta is the interesting column. Sentiment recovery within a thread is a far better signal of support quality than absolute sentiment ever was.</p>



<h2 class="wp-block-heading">Failure two: averaging a scale that was never numeric</h2>



<p class="wp-block-paragraph">Zendesk exposes sentiment as five ordered values from Very negative to Very positive, and Zendesk Explore ships a prebuilt report for average customer sentiment on a one-to-five scale. Amazon Comprehend returns one of <code>POSITIVE</code>, <code>NEGATIVE</code>, <code>NEUTRAL</code> or <code>MIXED</code> plus per-class confidence scores. Google Cloud Natural Language returns a polarity score and a magnitude. Azure AI Language returns document- and sentence-level labels with confidences.</p>



<p class="wp-block-paragraph">Only some of those are numbers, and none of the label sets are interval scales. The gap between Very negative and Negative is not the same size as the gap between Neutral and Positive, and there is no defensible reason to assume it is. So when you compute <code>AVG(sentiment)</code>, you are doing arithmetic on rank labels and getting a value that has no units.</p>



<p class="wp-block-paragraph">That would be a pedantic complaint if it were only a purity issue. It is not, because support sentiment distributions are usually bimodal. People write in angry or they write in grateful. Very few tickets are genuinely neutral. Take the mean of a two-humped distribution and you land in the valley between the humps, in a bucket where almost nobody actually is. Worse, the mean is stable there: the humps can grow, shrink and swap size while the average barely moves. That is your flat 3.2.</p>



<p class="wp-block-paragraph">Use share metrics instead. They are ordinal-safe, they move, and they are directly actionable.</p>



<pre class="wp-block-code"><code>-- Share of negative tickets, weighted by account value.
-- Ordinal-safe: no arithmetic on the label itself, only counting.
SELECT
    date_trunc('week', t.created_at)                      AS week,
    COUNT(*)                                              AS tickets,
    COUNT(*) FILTER (
        WHERE t.sentiment IN ('very_negative', 'negative')
    )                                                     AS negative_tickets,
    ROUND(
        100.0 * COUNT(*) FILTER (
            WHERE t.sentiment IN ('very_negative', 'negative')
        ) / NULLIF(COUNT(*), 0)
    , 1)                                                  AS pct_negative,
    SUM(a.arr) FILTER (
        WHERE t.sentiment IN ('very_negative', 'negative')
    )                                                     AS arr_behind_negative
FROM   support_tickets  t
JOIN   crm_contacts     c ON c.id = t.contact_id
JOIN   crm_accounts     a ON a.id = c.account_id
WHERE  t.created_at &gt;= date_trunc('week', CURRENT_DATE) - INTERVAL '26 weeks'
GROUP  BY 1
ORDER  BY 1;</code></pre>



<p class="wp-block-paragraph">Three things are happening there worth calling out. <code>FILTER</code> counts rows in a category without touching the label as a number. <code>NULLIF</code> stops a zero-ticket week from throwing a division error and silently killing the job. And the last column is the one executives will actually read, because a percentage of tickets is a support metric while revenue sitting behind negative sentiment is a business metric.</p>



<h2 class="wp-block-heading">Failure three: the people who left never showed up in the data</h2>



<p class="wp-block-paragraph">Support data only contains customers who chose to contact you. That is a selection effect, not a sample.</p>



<p class="wp-block-paragraph">The dangerous version is this: negative sentiment falls, everyone celebrates, and the actual cause is that the angriest cohort stopped writing in because they had already decided to leave. Silence reads as improvement. In a quarter where churn is rising, a falling negative-sentiment share is a warning sign, not a win.</p>



<p class="wp-block-paragraph">You cannot fix this inside the support dataset, which is precisely why the CRM join matters. Two guards worth building:</p>



<ul class="wp-block-list">
<li><strong>Contact rate as a companion metric.</strong> Tickets per active account per month, tracked alongside sentiment. Sentiment improving while contact rate collapses is a disengagement pattern, not a satisfaction one.</li>

<li><strong>Went-quiet detection.</strong> Accounts that used to file tickets regularly and have filed none in a defined window. In most books of business this list is a better churn predictor than the sentiment score itself, and it costs one window function to produce.</li>
</ul>



<h2 class="wp-block-heading">Failure four: the model learned someone else&#8217;s language</h2>



<p class="wp-block-paragraph">General-purpose sentiment APIs are trained largely on review-style and social text. Support English is a different dialect, and the mismatches are systematic rather than random.</p>



<ul class="wp-block-list">
<li><strong>Problem statements are not complaints.</strong> &#8220;The export fails with a 500&#8221; is a factual bug report. Review-trained models frequently score it negative because failure vocabulary dominates.</li>

<li><strong>Politeness masks severity.</strong> A calmly worded message from an enterprise admin saying the production sync has been down since Tuesday is your worst ticket of the week and will score close to neutral.</li>

<li><strong>Domain vocabulary inverts.</strong> &#8220;Killed the process&#8221;, &#8220;the job died&#8221;, &#8220;aborted the run&#8221; are neutral technical descriptions carrying heavy negative weight in general corpora.</li>

<li><strong>Your own text pollutes the corpus.</strong> Auto-acknowledgements, signatures, quoted reply chains and CSAT survey footers all get scored if you feed the raw comment body. Quoted history is the big one: score a five-reply thread naively and you have scored message one five times.</li>

<li><strong>Non-native English reads flat.</strong> Simpler sentence structure and less emotional vocabulary push scores toward neutral for a whole segment of your customer base, which quietly biases any regional comparison.</li>
</ul>



<p class="wp-block-paragraph">Independent evaluations of the major cloud sentiment APIs on general text tend to land well below the accuracy figures in vendor marketing, and the providers frequently disagree with each other on the same input. Treat published benchmark numbers as irrelevant to your decision. The only number that matters is accuracy on a few hundred of your own tickets, hand-labelled by someone who understands your product.</p>



<h3 class="wp-block-heading">Building the evaluation set</h3>



<ol class="wp-block-list">
<li>Pull a stratified random sample of tickets: across channels, across regions, across account tiers. Do not sample only recent tickets, or you bake in whatever was happening that month.</li>

<li>Have two people label independently, using a written rubric with worked examples of each class.</li>

<li>Measure agreement between the humans first. If your two support leads disagree on a third of the tickets, the task is underspecified and no model will fix that. Tighten the rubric and relabel.</li>

<li>Only then score the candidate models against the human labels, and look at the confusion matrix rather than accuracy. Where a model confuses classes tells you whether it is usable.</li>

<li>Freeze that set. It becomes your regression test when a vendor silently updates a model version.</li>
</ol>



<p class="wp-block-paragraph">Step three is the one teams skip, and it is the one that saves the project. Human agreement is the ceiling on model performance. There is no point chasing 90% against labels your own experts only agree on 70% of the time.</p>



<h2 class="wp-block-heading">Failure five: the CRM join is where the value and the breakage both live</h2>



<p class="wp-block-paragraph">A sentiment label with no account attached is trivia. Two hundred irritated free-tier users and one irritated account paying for a third of your renewals are not the same event, and the raw ticket table cannot tell them apart.</p>



<p class="wp-block-paragraph">The join from helpdesk to CRM is almost always on email, and email is a terrible key. Expect all of this:</p>



<ul class="wp-block-list">
<li>Shared inboxes such as ops@ or billing@, mapping to no single contact or to several.</li>

<li>Personal addresses used by people who work at a customer, invisible to any domain-matching logic.</li>

<li>Contacts who left and were deleted in Salesforce, HubSpot or Zoho CRM, orphaning every historical ticket they filed.</li>

<li>Account hierarchies, where the subsidiary files the ticket and the parent holds the contract.</li>

<li>Mergers and renames, which quietly rewrite history if your warehouse dimension is not versioned.</li>
</ul>



<p class="wp-block-paragraph">The engineering answer is boring and effective: treat unmatched tickets as a first-class, monitored quantity rather than something the join silently discards. Track match rate as a data quality metric with an alert threshold. When it drops, something changed in the CRM, and you want to know that before the number on the dashboard shifts for a reason nobody can explain.</p>



<p class="wp-block-paragraph">Also version the account dimension. If you overwrite the tier or the ARR in place, then every historical aggregate silently rewrites itself the next time someone upgrades, and last quarter&#8217;s report stops reproducing.</p>



<h2 class="wp-block-heading">The legal boundary that catches sentiment projects</h2>



<p class="wp-block-paragraph">This one is worth reading carefully if you have EU customers or EU staff, because the line does not fall where most people assume, and a lot of contact-centre software sells both sides of it in one bundle.</p>



<p class="wp-block-paragraph">Article 5(1)(f) of the EU AI Act prohibits AI systems that infer emotions of a person in the workplace or in education settings. The prohibitions in Article 5 have applied since 2 February 2025, and breaches sit in the Act&#8217;s highest penalty band. What matters for a sentiment project is the scope, which is narrow on two axes at once:</p>



<ul class="wp-block-list">
<li>It applies to emotion inference <strong>from biometric data</strong> — faces, voiceprints, physiological signals. The European Commission&#8217;s guidance takes the position that inferring emotion from written text is not based on biometric data and therefore falls outside the prohibition.</li>

<li>It applies to <strong>workplace and education contexts</strong>. Your customers writing into a helpdesk are neither.</li>
</ul>



<p class="wp-block-paragraph">So scoring the text of customer tickets and CRM notes generally sits outside the ban. Running voice-based emotion or stress analysis on your own support agents&#8217; calls, for coaching or quality scoring, is the case the prohibition was written for. Plenty of platforms ship customer-facing text sentiment and agent-facing voice sentiment as adjacent features in the same product, and the pitch does not distinguish them. If you are the deployer, the compliance obligation is yours, and a vendor&#8217;s assurance is not a defence.</p>



<p class="wp-block-paragraph">Separately, Article 50 carries transparency obligations for emotion recognition systems where they remain lawful. And none of this displaces GDPR: ticket bodies are full of personal data, and shipping them to a third-party API is a processing decision that needs a basis, a processor agreement and a residency answer.</p>



<p class="wp-block-paragraph">I am an engineer, not a lawyer. Treat the above as the shape of the question to take to counsel, not as advice.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">A pipeline shape that survives contact</h2>



<p class="wp-block-paragraph">Nothing exotic. The discipline is in what you store, not what you call.</p>



<ol class="wp-block-list">
<li><strong>Extract incrementally</strong> from the helpdesk and CRM on an updated-at cursor, not a full reload. Land raw payloads immutably before you transform anything, so a bad model run is re-runnable without re-hitting the source API.</li>

<li><strong>Clean the text.</strong> Strip quoted reply chains, signatures, automated acknowledgements and survey footers. Keep the cleaned text as its own column so a scoring bug is debuggable later.</li>

<li><strong>Redact personal data</strong> before it leaves your boundary. Regex handles the structured patterns: emails, phone numbers, card and account numbers, order IDs. Named-entity detection handles names and addresses. Amazon Comprehend exposes <code>DetectPiiEntities</code> and <code>ContainsPiiEntities</code> for this if you are already in AWS.</li>

<li><strong>Score in batches</strong>, with the model identifier and version written into every row.</li>

<li><strong>Aggregate at account level</strong> with share metrics and value weighting, against a versioned account dimension.</li>

<li><strong>Alert on movement, not on level.</strong> Nobody acts on &#8220;sentiment is 3.2&#8221;. People act on &#8220;this named account&#8217;s negative share tripled in two weeks&#8221;.</li>
</ol>



<p class="wp-block-paragraph">Step four&#8217;s version column is the one people leave out and regret. Managed sentiment endpoints get retrained. When labels shift on unchanged text, the version column is the difference between a five-minute explanation and a week of forensics.</p>



<p class="wp-block-paragraph">Here is what a batch call looks like against Amazon Comprehend, which is a reasonable default if your warehouse already lives in AWS:</p>



<pre class="wp-block-code"><code>import boto3

comprehend = boto3.client("comprehend", region_name="eu-west-1")

# BatchDetectSentiment accepts a maximum of 25 documents per call,
# and DetectSentiment caps each document at 5 KB of UTF-8 text.
# Chunk to both limits or the call fails for the whole batch.
BATCH_SIZE = 25
MAX_BYTES  = 5000

def score_batch(texts):
    payload = [t.encode("utf-8")[:MAX_BYTES].decode("utf-8", "ignore")
               for t in texts[:BATCH_SIZE]]
    resp = comprehend.batch_detect_sentiment(
        TextList=payload,
        LanguageCode="en",
    )
    # ResultList is index-aligned to the input; ErrorList holds
    # per-document failures. Read both or you lose rows silently.
    return resp["ResultList"], resp["ErrorList"]</code></pre>



<p class="wp-block-paragraph">The truncation on the encoded bytes rather than the character count is deliberate: the limit is measured in bytes, and a ticket full of accented characters or CJK text hits it much earlier than the character count suggests. The <code>ErrorList</code> read matters for the same reason a dead letter queue matters. A partial batch success that you never inspect is data loss that looks like a clean run.</p>



<p class="wp-block-paragraph">For a quick sanity check on a single string before wiring any of this up, the CLI is faster than writing code:</p>



<pre class="wp-block-code"><code>aws comprehend detect-sentiment 
    --region eu-west-1 
    --language-code "en" 
    --text "Still waiting on the refund you promised last week."

{
    "SentimentScore": {
        "Mixed": 0.0033542951568961143,
        "Positive": 0.0086313202530145600,
        "Neutral": 0.1495875907897949,
        "Negative": 0.8384268 
    },
    "Sentiment": "NEGATIVE",
    "LanguageCode": "en"
}</code></pre>



<p class="wp-block-paragraph">Note that the response carries per-class confidences, not just the winning label. Store the whole <code>SentimentScore</code> object. A ticket scored NEGATIVE at 0.51 and one scored NEGATIVE at 0.98 are very different tickets, and the label alone throws that away. Confidence is also what lets you route only the ambiguous cases to a more expensive model or a human, instead of paying for everything.</p>



<h3 class="wp-block-heading">Choosing where the model runs</h3>



<p class="wp-block-paragraph">Three broad options, and the trade-offs are real in both directions.</p>



<ul class="wp-block-list">
<li><strong>Native helpdesk features.</strong> Zendesk intelligent triage, Salesforce Einstein, HubSpot and Zoho CRM all ship some form of built-in sentiment. Zero integration work, labels land on the ticket where agents can see them, and routing rules can consume them directly. You get the vendor&#8217;s definition of sentiment, no control over stamping policy beyond the toggles they expose, and data that is awkward to move.</li>

<li><strong>Managed NLP APIs.</strong> Amazon Comprehend, Azure AI Language and Google Cloud Natural Language. Per-character billing, no infrastructure, useful extras such as aspect-level or targeted sentiment. Comprehend&#8217;s targeted sentiment, which attaches sentiment to specific entities in the text, is English-only, and that constraint bites hard if your support volume is multilingual. Bill by volume of text, so a batch backfill of five years of tickets is a genuinely different cost event from steady-state scoring.</li>

<li><strong>Self-hosted transformers.</strong> A fine-tuned encoder model from the Hugging Face ecosystem, running on your own box. Best accuracy on your domain once tuned, no per-call cost, no text leaving your perimeter, which resolves most residency arguments before they start. In exchange you own model serving, evaluation and drift. A mid-tier dedicated server from a provider like Contabo or Hetzner runs CPU inference for batch workloads perfectly well, and batch is what this workload is. You do not need a GPU to score yesterday&#8217;s tickets overnight.</li>
</ul>



<p class="wp-block-paragraph">If you have no strong residency constraint and a small volume, start with a managed API. The self-hosted path is worth it when you have enough labelled data to fine-tune, or a legal reason the text cannot leave. Not before.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Sentiment is flat regardless of what the team does</h3>



<p class="wp-block-paragraph">Two likely causes. Either you are averaging an ordinal scale over a bimodal distribution, or you are scoring first messages only. Check the distribution shape first, since it takes one query. If it has two humps, switch to share metrics before you touch anything else.</p>



<h3 class="wp-block-heading">Everything scores neutral</h3>



<p class="wp-block-paragraph">Usually a text-cleaning problem. If quoted reply chains, signatures and boilerplate survive into the scored text, the actual message is diluted by noise and the model regresses to neutral. Print the exact string you are sending for twenty tickets and read them. This bug is invisible in aggregate and obvious in the raw payload.</p>



<h3 class="wp-block-heading">The trend shifted overnight with no matching business event</h3>



<p class="wp-block-paragraph">Check the model version column and the CRM match rate on the same day. A managed endpoint retrain and a CRM field change both produce exactly this signature. If you did not store the version, this is the moment you start.</p>



<h3 class="wp-block-heading">One region always looks happier</h3>



<p class="wp-block-paragraph">Before concluding anything about that region, check the language distribution. Non-native English and machine-translated tickets both drift toward neutral, which shows up as &#8220;less negative&#8221; in a share metric. Compare like with like by language, or score in the original language where the provider supports it.</p>



<h3 class="wp-block-heading">Batch scoring jobs fail intermittently</h3>



<p class="wp-block-paragraph">Almost always the document size limit, hit by one oversized ticket with a pasted log file or stack trace in the body. Truncate on encoded byte length before the call and check the per-document error list rather than assuming a successful HTTP response means every document scored.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Reporting an average of an ordinal label set and calling it a score.</li>

<li>Scoring the first message and describing the result as customer satisfaction.</li>

<li>Treating unmatched tickets as an inner-join casualty instead of a monitored metric.</li>

<li>Sending raw ticket bodies to a third-party API with no redaction step.</li>

<li>Discarding confidence scores and keeping only the winning label.</li>

<li>Trusting a vendor accuracy figure over a few hundred of your own hand-labelled tickets.</li>

<li>Overwriting the account dimension in place, so historical reports quietly stop reproducing.</li>

<li>Assuming a helpdesk vendor&#8217;s compliance posture transfers to you as the deployer.</li>
</ul>



<h2 class="wp-block-heading">Best practices for customer sentiment analysis</h2>



<ul class="wp-block-list">
<li>Write down the unit of analysis before writing any code, and make sure the dashboard label matches it.</li>

<li>Store first-message sentiment, last-message sentiment and the delta. Report the delta.</li>

<li>Use share-of-negative and value-weighted share, never a mean over labels.</li>

<li>Keep a frozen, hand-labelled evaluation set and re-run it on a schedule as a regression test.</li>

<li>Persist the model identifier, model version and full confidence distribution on every scored row.</li>

<li>Redact personal data before the text crosses a network boundary, and log what was redacted, not the values.</li>

<li>Monitor CRM match rate, contact rate and language mix alongside the sentiment number itself.</li>

<li>Alert on named-account movement, not on the global level.</li>

<li>Know which side of the biometric line each feature you enable sits on, and put the answer in writing.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is customer sentiment analysis accurate enough to act on?</h3>



<p class="wp-block-paragraph">For prioritisation and trend detection, generally yes. For individual decisions about a specific customer, no. The reliable pattern is to use it as a filter that surfaces conversations for a human to read, rather than as a verdict. Any single ticket&#8217;s label should be treated as a hint, not a fact.</p>



<h3 class="wp-block-heading">Should I use my helpdesk&#8217;s built-in sentiment or build my own pipeline?</h3>



<p class="wp-block-paragraph">Start with the built-in feature. It is free or cheap, it lands where agents work, and it tells you within a month whether anyone actually uses sentiment for anything. Build your own when you need a different stamping policy, cross-system aggregation, account-value weighting, or control over where the text is processed. Those are the four reasons that justify the effort.</p>



<h3 class="wp-block-heading">Can I use an LLM instead of a dedicated sentiment API?</h3>



<p class="wp-block-paragraph">You can, and the quality on nuance, sarcasm and domain vocabulary is usually better because you can describe your product and your customers in the prompt. The costs are latency, price per token at backfill scale, and output stability. Pin the model version, constrain the output to a fixed label set, and set temperature to zero, or you will get labels that drift between runs on identical text. A pragmatic middle ground is a cheap classifier by default with an LLM as a fallback on low-confidence cases.</p>



<h3 class="wp-block-heading">How much historical data do I need before the trend means anything?</h3>



<p class="wp-block-paragraph">Enough to cover your seasonality, which for most businesses means at least a full year. Support volume and tone move with billing cycles, product releases, holidays and renewal periods. A quarter of data will show you a trend that is really just the calendar.</p>



<h3 class="wp-block-heading">Does sentiment analysis on support tickets fall under the EU AI Act ban?</h3>



<p class="wp-block-paragraph">The Article 5(1)(f) prohibition targets emotion inference from biometric data in workplace and education contexts, and Commission guidance treats text-based sentiment as outside that scope. Scoring customer-written text is generally not caught. Voice or facial emotion analysis applied to your own employees is squarely the thing it prohibits. Transparency duties and GDPR obligations still apply regardless, and this is a question for your counsel rather than your engineer.</p>



<h3 class="wp-block-heading">What is a good target for share of negative tickets?</h3>



<p class="wp-block-paragraph">There is no cross-industry benchmark worth quoting, because the number depends entirely on your product category, your support channels and how the model was calibrated. Your own baseline is the only meaningful comparison. Measure for a full cycle, then set targets against that.</p>



<h3 class="wp-block-heading">How do I handle multilingual support volume?</h3>



<p class="wp-block-paragraph">Detect the language, store it as a column, and segment every report by it. Do not translate to English and then score, since translation flattens emotional register and adds a second error source. Check which of your provider&#8217;s features are English-only before designing around them; some advanced modes, including Comprehend&#8217;s targeted sentiment, are narrower than the base sentiment endpoint.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A customer sentiment analysis pipeline that runs cleanly every night and produces a number nobody can act on is not a partially working system. It is a broken one, and the breakage is upstream of anything your monitoring can see.</p>



<p class="wp-block-paragraph">So before optimising a model or shopping for a platform, answer three questions in writing: which message gets scored, what unit the number describes, and which accounts it is weighted by. Get those right with a mediocre classifier and you have something useful. Get them wrong with the best model on the market and you have a very accurate description of how annoyed people were in their opening sentence.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help making this work on your data?</h2>



<p class="wp-block-paragraph">I build and fix the data plumbing behind customer analytics. On this particular problem, that usually means:</p>



<ul class="wp-block-list">
<li>Auditing an existing sentiment metric to find out why it never moves, and rebuilding it on share and delta metrics that do.</li>

<li>Building incremental extraction from Zendesk, Salesforce, HubSpot or Zoho CRM into a warehouse, with cursor state, retries and dead-letter handling that survives an API outage.</li>

<li>Fixing the helpdesk-to-CRM identity join, including shared inboxes, account hierarchies and a versioned account dimension so historical reports keep reproducing.</li>

<li>Standing up a redaction layer so ticket text is scrubbed of personal data before it reaches any third-party model endpoint.</li>

<li>Running a proper evaluation: stratified sample, dual human labelling, inter-rater agreement, confusion matrices across candidate models, and a frozen regression set.</li>

<li>Setting up account-level alerting in Grafana or your BI tool that fires on movement in named accounts rather than on a global average.</li>
</ul>



<p class="wp-block-paragraph">If you have a dashboard that has been flat for months, send me the query behind it and a screenshot of the distribution. That is usually enough to tell you what is wrong before we talk about scope.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/customer-sentiment-analysis-crm-support-data/">Customer Sentiment Analysis From CRM and Support Data: Building a Score You Can Actually Trust</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Amazon Textract Data Extraction: What Breaks on Real Contracts and Reports</title>
		<link>https://john-nessime.com/blog/devops/amazon-textract-data-extraction/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 20 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon SNS]]></category>
		<category><![CDATA[Amazon SQS]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Batch Processing]]></category>
		<category><![CDATA[Boto3]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Legal Tech]]></category>
		<category><![CDATA[OCR]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Serverless]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=270</guid>

					<description><![CDATA[<p>Textract rarely fails loudly. It returns a plausible result that is quietly incomplete: a truncated result set, a tick box read as an empty string, a clause split across a page break. A practitioner's guide to the failure modes that actually bite when you point Amazon Textract at contracts, technical reports and correspondence, plus how to choose between sync and async, which feature types are worth paying for, and where Textract stops being the right tool.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/amazon-textract-data-extraction/">Amazon Textract Data Extraction: What Breaks on Real Contracts and Reports</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The pipeline passed every test. Three sample agreements, all the right fields, clean JSON out the other side. Then it ran against the real archive and somebody in legal noticed that every contract longer than about forty pages was missing its termination clause. No errors. No failed jobs. The dashboard was green the whole time.</p>



<p class="wp-block-paragraph">The bug was a missing loop. The job had finished, the results were there, and the code had read the first chunk of them and stopped.</p>



<p class="wp-block-paragraph">That is the shape of most problems with Amazon Textract data extraction on real documents. The service rarely fails loudly. It returns a plausible-looking result that is quietly incomplete, and you find out weeks later when somebody asks a question the data cannot answer. This post covers the failures that actually bite on contracts, technical reports and correspondence: sync versus async, reading the block graph without losing fields, which feature types are worth paying for, and where Textract stops being the right tool.</p>



<h2 class="wp-block-heading">Why Amazon Textract data extraction is not just OCR</h2>



<p class="wp-block-paragraph">Plain OCR gives you words and their positions. Textract gives you a graph. Every response is a flat array of <code>Block</code> objects, and each block carries an ID plus relationships to other block IDs. A page relates to its lines, a line to its words, a key to its value, a table to its cells.</p>



<p class="wp-block-paragraph">That structure is the whole point, and it is where the pain lives. Nothing is nested for you. To read one form field you find the <code>KEY_VALUE_SET</code> block with entity type <code>KEY</code>, follow its <code>VALUE</code> relationship to another block, then follow that block&#8217;s <code>CHILD</code> relationships to the words. Three hops, and getting a hop wrong returns an empty string rather than an exception.</p>



<p class="wp-block-paragraph">Contracts and reports make this harder than invoices do. An invoice has a total. A master services agreement has a liability cap buried in a numbered sub-clause that spans a page break.</p>



<h2 class="wp-block-heading">Synchronous or asynchronous: the choice is made for you</h2>



<p class="wp-block-paragraph">People reach for <code>AnalyzeDocument</code> first because it returns results in the same call and is easy to test in a notebook. Then they hit the quotas, which AWS documents as hard limits you cannot raise. Synchronous operations cap JPEG, PNG, PDF and TIFF at 10 MB in memory, and cap PDF and TIFF at <strong>one page</strong>. Asynchronous operations keep JPEG and PNG at 10 MB but take PDF and TIFF up to 500 MB and 3,000 pages.</p>



<p class="wp-block-paragraph">One page. That single constraint decides your architecture. Any real contract goes through <code>StartDocumentAnalysis</code> and <code>GetDocumentAnalysis</code>, which means it goes through S3, which means you need somewhere to stage documents and a way to learn when the job is done.</p>



<p class="wp-block-paragraph">Three other constraints from the same quota page are worth knowing before you promise anything to a client:</p>



<ul class="wp-block-list">
<li>Text detection covers English, French, German, Italian, Portuguese and Spanish. Query detection is English only.</li>

<li>Vertical text is not supported. Rotation is fine, including odd in-plane angles, but vertically written scripts are not.</li>

<li>Password-protected PDFs are rejected and XFA-based PDFs are unsupported. Both turn up in legal archives more often than you would expect.</li>
</ul>



<p class="wp-block-paragraph">Pass a <code>ClientRequestToken</code> on submission. It is an idempotency token: the same token returns the same <code>JobId</code> instead of starting a duplicate job. If your submitter is a Lambda function behind an S3 event, and S3 events can be delivered more than once, that is the difference between paying once and paying twice.</p>



<pre class="wp-block-code"><code>import boto3

textract = boto3.client("textract")

job = textract.start_document_analysis(
    DocumentLocation={
        "S3Object": {"Bucket": "contracts-intake", "Name": "msa/acme-2.pdf"}
    },
    FeatureTypes=["FORMS", "TABLES"],
    ClientRequestToken="msa-acme-2-v1",
    NotificationChannel={
        "SNSTopicArn": "arn:aws:sns:eu-west-1:111122223333:textract-done",
        "RoleArn": "arn:aws:iam::111122223333:role/TextractSnsPublish",
    },
)</code></pre>



<p class="wp-block-paragraph">The notification channel is optional but you want it. Polling in a loop inside a Lambda function burns billed duration doing nothing, and Lambda&#8217;s execution ceiling will cut you off on long documents anyway. Publish to SNS, fan out to SQS, let a second function do the reading, and you get a natural place to hang a dead letter queue.</p>



<h2 class="wp-block-heading">The result set you never finished reading</h2>



<p class="wp-block-paragraph">This is the one that cost the termination clauses. <code>GetDocumentAnalysis</code> returns results in pages. When there are more blocks than fit in one response, the response carries a <code>NextToken</code> and you have to call again with it. If you do not, you get the beginning of the document and nothing tells you so. <code>JobStatus</code> still reports success, because the job did succeed. Your code just stopped reading.</p>



<p class="wp-block-paragraph">Short test documents fit in a single response. That is exactly why this survives testing and dies in production.</p>



<pre class="wp-block-code"><code>def fetch_all_blocks(textract, job_id):
    blocks, next_token = [], None

    while True:
        kwargs = {"JobId": job_id}
        if next_token:
            kwargs["NextToken"] = next_token

        response = textract.get_document_analysis(**kwargs)

        if response["JobStatus"] == "FAILED":
            raise RuntimeError(response.get("StatusMessage", "job failed"))

        blocks.extend(response["Blocks"])
        next_token = response.get("NextToken")

        if not next_token:
            return blocks</code></pre>



<p class="wp-block-paragraph">Then make the failure detectable. Treat the highest page number present in the blocks as a checksum against the page count of the source file. Submit a 60-page PDF, get blocks that stop at page 12, and something is wrong no matter what the job status says.</p>



<h2 class="wp-block-heading">Choosing feature types without paying for all of them</h2>



<p class="wp-block-paragraph"><code>AnalyzeDocument</code> and <code>StartDocumentAnalysis</code> take a <code>FeatureTypes</code> list: <code>FORMS</code>, <code>TABLES</code>, <code>QUERIES</code>, <code>SIGNATURES</code> and <code>LAYOUT</code>. This is not cosmetic. Textract bills per page and the per-page rate depends on which features you enabled, so turning all five on because you might need them later multiplies your bill across the whole archive. Rates change and vary by region, so check the current pricing page rather than trusting a number in a blog post. The mechanism does not change: each feature adds to the per-page cost, and you pay it on every page you submit, including blank separator sheets.</p>



<ul class="wp-block-list">
<li><strong>FORMS</strong> finds key-value pairs where the document has an explicit label. Good for cover sheets, signature blocks, report headers. Useless for prose clauses with no label.</li>

<li><strong>TABLES</strong> reconstructs rows, columns and cells, and also identifies table titles, footers and merged cells. This is what you want for rate cards and appendix tables.</li>

<li><strong>QUERIES</strong> lets you ask natural-language questions and get answers back under an alias you chose. This is the feature that makes contracts tractable.</li>

<li><strong>SIGNATURES</strong> returns the location and confidence of handwritten signatures, electronic signatures and initials. It tells you something is signed at a coordinate, not whose signature it is.</li>

<li><strong>LAYOUT</strong> groups text into titles, headers, footers, section headers, paragraphs and lists in reading order. Skippable on single-column documents. On two-column reports it is the difference between coherent text and interleaved nonsense.</li>
</ul>



<p class="wp-block-paragraph">A practical pattern: run a cheap text-detection pass to classify each document, then apply the expensive feature set only to the pages that matter. A 200-page appendix of scanned site photographs does not need FORMS and TABLES.</p>



<h2 class="wp-block-heading">Reading the block graph without losing fields</h2>



<p class="wp-block-paragraph">Here is the failure that produces silent empty strings. A key-value pair&#8217;s value is not always text. If the value is a tick box, the value block&#8217;s children include a <code>SELECTION_ELEMENT</code> block, and that block has no text at all. It has a <code>SelectionStatus</code> of <code>SELECTED</code> or <code>NOT_SELECTED</code>. Concatenate only <code>WORD</code> children and a ticked box comes back as an empty string, which looks exactly like a field nobody filled in.</p>



<pre class="wp-block-code"><code>def block_text(block, block_map):
    parts = []
    for rel in block.get("Relationships", []):
        if rel["Type"] != "CHILD":
            continue
        for child_id in rel["Ids"]:
            child = block_map[child_id]
            if child["BlockType"] == "WORD":
                parts.append(child["Text"])
            elif child["BlockType"] == "SELECTION_ELEMENT":
                parts.append(child["SelectionStatus"])
    return " ".join(parts)</code></pre>



<p class="wp-block-paragraph">Build <code>block_map</code> once as a dictionary keyed on block ID before you traverse anything. Scanning the block list linearly for each lookup turns a three-hop traversal into an accidental quadratic, which you will notice the first time a 3,000-page bundle goes through.</p>



<h2 class="wp-block-heading">Queries: asking for clauses that have no label</h2>



<p class="wp-block-paragraph">FORMS works when the document says &#8220;Effective Date:&#8221; next to the date. Contracts frequently do not. The governing law sits in a paragraph of prose halfway down page nine. Queries handles that. You pass questions in plain English, each with an <code>Alias</code> you choose, and the response pairs <code>QUERY</code> blocks with <code>QUERY_RESULT</code> blocks through an <code>ANSWER</code> relationship. The alias is what makes this usable downstream: you match on <code>governing_law</code>, not on the exact wording of the question, so you can reword questions without breaking your schema.</p>



<pre class="wp-block-code"><code>QueriesConfig = {
    "Queries": [
        {"Text": "What is the governing law?",
         "Alias": "governing_law", "Pages": ["*"]},
        {"Text": "What is the termination notice period?",
         "Alias": "termination_notice", "Pages": ["9-*"]},
    ]
}</code></pre>



<p class="wp-block-paragraph">The limits matter. AWS documents a maximum of 15 queries per page for synchronous operations and 30 per page for asynchronous ones. A fifty-field schema cannot be asked in one pass. You either split it across multiple calls, which multiplies per-page spend, or you narrow <code>Pages</code> so each query runs only where the answer plausibly lives. Signature blocks are at the end, definitions near the front. Running every query against every page is the most common way people accidentally triple their bill.</p>



<p class="wp-block-paragraph">Queries are pre-trained on a spread of business documents including paystubs, bank statements, loan applications and mortgage notes. Your niche contract template was not in that set. Expect good results on common concepts and mediocre ones on house-specific terminology.</p>



<h2 class="wp-block-heading">Documents that fight back</h2>



<h3 class="wp-block-heading">Clauses that span a page break</h3>



<p class="wp-block-paragraph">Textract analyses pages. A clause starting at the bottom of page 11 and finishing at the top of page 12 is two disconnected fragments as far as the block graph is concerned. Queries will often return the fragment on one page and miss the qualifier on the other, which is the worst outcome available because the answer looks complete. The fix is not clever: reassemble full text in reading order using LAYOUT, then run clause-level logic over that rather than over per-page answers. Use Queries to <em>locate</em> a clause and the reassembled text to <em>read</em> it.</p>



<h3 class="wp-block-heading">Two-column reports and email threads</h3>



<p class="wp-block-paragraph">Without LAYOUT, a two-column technical report reads as alternating lines from both columns. Everything downstream inherits that corruption and it passes a spot check, because individual lines look fine. LAYOUT sequences elements in reading order and returns block types for titles, headers, footers and section headers, which is also what you need to chunk a report by section.</p>



<p class="wp-block-paragraph">Printed email chains are quoted text inside quoted text, newest message at the top, the same signature block repeated five times. Textract extracts all of it faithfully, which is the problem. Splitting a thread into individual messages is a text-processing job you do afterwards, not something a feature type solves.</p>



<h3 class="wp-block-heading">PDFs that were never scanned</h3>



<p class="wp-block-paragraph">This is the cheapest win available and almost everybody misses it. A large share of contract archives are born-digital PDFs exported from a word processor, and they already contain a text layer. Running OCR over them pays a service to guess at text you could have read directly. Put a triage step in front of the pipeline; <code>pdftotext</code> from poppler-utils pulls the embedded layer if there is one:</p>



<pre class="wp-block-code"><code>pdftotext -layout contract.pdf - | head -c 2000</code></pre>



<p class="wp-block-paragraph">Substantial text back means born-digital: route it down a cheaper path and reserve Textract for genuinely scanned material. Triage across a large archive is an embarrassingly parallel batch job that sits better on a plain VPS from somewhere like Contabo or InterServer than on per-invocation serverless billing, since you are CPU-bound for minutes at a time rather than reacting to events. One caveat: born-digital does not mean clean. Some exporters produce a text layer with broken word spacing or mangled ligatures, so sample before you trust it.</p>



<h2 class="wp-block-heading">Confidence scores you can actually act on</h2>



<p class="wp-block-paragraph">Every block carries a confidence score and the instinct is to threshold on it. That is usually wrong. Confidence tells you how sure the model is about <em>which characters are on the page</em>, not whether the field is semantically right. Textract can read a date with total certainty and hand you the wrong one, because it picked up the printing date in the footer instead of the effective date in the recitals. High confidence, completely wrong answer.</p>



<p class="wp-block-paragraph">It is also per block. A key-value pair has a score on the key, another on the value and separate ones on each word, so there is no single number to gate on. Validate the extracted value instead. Does the date parse and fall in a plausible range? Does the total equal the sum of the line items? Does the party name appear in the signature block as well as the preamble? Route to human review on failed validation and treat low confidence as one input among several.</p>



<h2 class="wp-block-heading">When Custom Queries adapters are worth the effort</h2>



<p class="wp-block-paragraph">If pre-trained Queries keeps getting your house document type wrong, train an adapter. Create it with <code>CreateAdapter</code>, annotate sample documents, train a version with <code>CreateAdapterVersion</code>, then reference it at inference time.</p>



<pre class="wp-block-code"><code>AdaptersConfig = {
    "Adapters": [
        {"AdapterId": ADAPTER_ID, "Version": "1", "Pages": ["1-5"]},
        {"AdapterId": ADAPTER_ID, "Version": "1", "Pages": ["6-*"]},
    ]
}</code></pre>



<p class="wp-block-paragraph">That <code>Pages</code> string takes digits, hyphens and an asterisk with no blank spaces, a page can only have one adapter applied to it, and an asterisk meaning all pages must be the only element in the list.</p>



<p class="wp-block-paragraph">Two things to weigh. AWS sets a minimum of five samples per query level for training or testing and says plainly that more is better, so treat five as a demo floor rather than a number that survives layout variation. And successful trainings per month are capped per account, so the tight annotate-train-evaluate loop you want is not available. Adapters suit one high-volume, house-specific document type with a stable schema. They are a poor fit for a long tail of one-off templates, which is what most legal archives actually are.</p>



<h2 class="wp-block-heading">Where Textract stops and a language model starts</h2>



<p class="wp-block-paragraph">This is a real decision now, so both sides deserve a hearing. Textract&#8217;s case is strong on standardised, high-volume documents. It is deterministic in a way generative models are not, it returns bounding-box geometry so you can point at exactly where a value came from, it gives per-element confidence, and it bills predictably per page. When an auditor asks why a field has the value it does, geometry beats a model&#8217;s reasoning.</p>



<p class="wp-block-paragraph">The generative side, whether that is Amazon Bedrock Data Automation as a managed document processing service or a model called directly through Amazon Bedrock, wins where the task needs reasoning rather than pattern matching. &#8220;Does this agreement contain an assignment restriction, and if so summarise it&#8221; is a comprehension question, and Textract has no answer for it. Generative approaches also cope better with layouts that were in nobody&#8217;s training set.</p>



<p class="wp-block-paragraph">Most production pipelines use both, which is what AWS&#8217;s own reference architectures push: Textract for faithful extraction with geometry and confidence, then a model over the extracted text for classification and comprehension. Choosing from scratch, ask whether your documents are standardised and high-volume. If yes, Textract plus a thin post-processing layer is cheaper and more auditable. If they are varied and volume is modest, a managed generative service gets you working faster. Google Document AI and Azure AI Document Intelligence cover similar ground if you are not committed to AWS.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Extraction stops partway through long documents.</strong> You are not following <code>NextToken</code>. Compare the highest page number in your blocks against the source page count.</li>

<li><strong>Fields come back empty on forms you know were filled in.</strong> The value is a <code>SELECTION_ELEMENT</code> and you are only reading <code>WORD</code> children.</li>

<li><strong>The document is rejected as too large.</strong> You are on the synchronous API. Multi-page PDFs must go through <code>StartDocumentAnalysis</code>, and the 10 MB cap is on in-memory size, not file size on disk.</li>

<li><strong>Textract cannot read the document at all.</strong> Check for password protection and XFA-based forms. Both are documented as unsupported.</li>

<li><strong>The service cannot access the S3 object.</strong> Usually the execution role missing <code>s3:GetObject</code>, or a bucket in a different region from the Textract endpoint you called.</li>

<li><strong>Throttling under batch load.</strong> Transactions-per-second quotas are per account per region. Smooth spiky traffic through a queue and retry with exponential backoff and jitter rather than raising concurrency.</li>

<li><strong>Garbled text on reports that look fine to you.</strong> Multi-column layout without LAYOUT enabled, or a source scan below roughly 150 DPI. Re-scanning beats any amount of post-processing.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Testing only on short documents, which hides the pagination bug entirely.</li>

<li>Enabling every feature type on every page because it is easier than deciding.</li>

<li>Running OCR over born-digital PDFs that already carry a text layer.</li>

<li>Treating a confidence score as a correctness score.</li>

<li>Scanning the block array linearly instead of building an ID map.</li>

<li>Polling for job completion inside a Lambda function instead of using SNS.</li>

<li>Storing only extracted values and discarding the raw JSON, so reprocessing means paying again.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Persist the full Textract JSON to S3 keyed on a hash of the source document. Reprocessing then costs storage, not extraction.</li>

<li>Classify first, extract second. A cheap text-detection pass tells you which expensive feature set each document needs.</li>

<li>Validate on the value, not the score. Parse dates, check ranges, cross-reference names against other parts of the document.</li>

<li>Use aliases on every query and treat them as your schema contract.</li>

<li>Set a per-job page-count assertion and alert on mismatches. That is your canary for silent truncation.</li>

<li>Keep source documents in an encrypted bucket with a lifecycle policy. Contracts and correspondence should not accumulate indefinitely by accident.</li>

<li>Track cost per document rather than per API call. Feature stacking makes the per-call figure meaningless.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can Amazon Textract handle multi-page PDF contracts?</h3>



<p class="wp-block-paragraph">Yes, through the asynchronous API only. Synchronous operations cap PDF and TIFF at one page. Asynchronous operations accept PDF and TIFF up to 500 MB and 3,000 pages, staged through S3.</p>



<h3 class="wp-block-heading">Why is my Textract output missing the end of the document?</h3>



<p class="wp-block-paragraph">Almost certainly because you read the first response from <code>GetDocumentAnalysis</code> and stopped. Results are paginated with a <code>NextToken</code> and the job status still reports success, so nothing signals the truncation. Loop until <code>NextToken</code> is absent.</p>



<h3 class="wp-block-heading">How many Textract Queries can I ask per page?</h3>



<p class="wp-block-paragraph">AWS documents 15 per page for synchronous operations and 30 per page for asynchronous ones. Larger schemas need multiple passes, which costs more, so narrowing each query&#8217;s page range is worth the effort.</p>



<h3 class="wp-block-heading">Does Textract work on handwriting and signatures?</h3>



<p class="wp-block-paragraph">It detects handwriting, and the SIGNATURES feature returns the location and confidence of handwritten signatures, electronic signatures and initials. It does not verify identity: it tells you something was signed and where, not by whom.</p>



<h3 class="wp-block-heading">What languages does Amazon Textract support?</h3>



<p class="wp-block-paragraph">Text detection covers English, French, German, Italian, Portuguese and Spanish. Queries detection is English only, vertically written text is unsupported, and Textract does not return the detected language in its output.</p>



<h3 class="wp-block-heading">Should I use Textract or a generative model?</h3>



<p class="wp-block-paragraph">Textract when documents are standardised, volume is high, and you need auditable geometry and per-field confidence. A generative service when layouts vary widely, volume is modest, or the task needs comprehension rather than extraction. Most production pipelines use both.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Amazon Textract data extraction fails quietly far more often than it fails loudly. A truncated result set, a tick box read as an empty string, a clause split across a page break, a confidently extracted wrong date: none of these throw an exception and none show up on a dashboard.</p>



<p class="wp-block-paragraph">So build the assertions that make silence detectable. Compare expected page counts against extracted ones, validate values rather than trusting scores, and keep the raw JSON so you can prove what the service actually returned. Textract is genuinely good at reading documents. Your job is noticing when it did not read all of them.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a document extraction pipeline that does not lose fields?</h2>



<p class="wp-block-paragraph">I build and fix document processing pipelines on AWS. Typical work looks like this:</p>



<ul class="wp-block-list">
<li>Auditing an existing Textract pipeline for silent truncation, dropped selection elements and mis-scoped queries</li>

<li>Designing the async architecture end to end: S3 intake, SNS and SQS fan-out, Lambda or container workers, dead letter handling and retries</li>

<li>Cutting per-page cost by classifying first, triaging born-digital PDFs out of the OCR path and narrowing feature types per page range</li>

<li>Turning block-graph JSON into a clean schema in CSV, a relational database or a data lake, with validation rules and a review queue for exceptions</li>

<li>Building the hybrid path where Textract handles extraction and a model on Amazon Bedrock handles classification and comprehension</li>

<li>Adding the observability that makes quiet failures loud: page-count assertions, per-document cost tracking, confidence distribution monitoring</li>
</ul>



<p class="wp-block-paragraph">If you have a redacted sample document, a chunk of Textract JSON or an extraction coming back half empty, send it over and I will tell you what is going wrong with it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/amazon-textract-data-extraction/">Amazon Textract Data Extraction: What Breaks on Real Contracts and Reports</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Apache Airflow on AWS: Building SaaS and API Pipelines That Don&#8217;t Lie to You</title>
		<link>https://john-nessime.com/blog/devops/apache-airflow-aws-saas-api-pipelines/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 05 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Airflow]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[MWAA]]></category>
		<category><![CDATA[Orchestration]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=145</guid>

					<description><![CDATA[<p>Most API pipeline failures are green DAGs producing incomplete data. A practical guide to running Apache Airflow on AWS for SaaS and API extraction: choosing between MWAA provisioned, MWAA Serverless and self-managed, the pool setting that silently stops throttling when you go deferrable, retry and pagination design, secrets handling, and the four cost lines that actually move.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/apache-airflow-aws-saas-api-pipelines/">Apache Airflow on AWS: Building SaaS and API Pipelines That Don&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually lands on a Monday: &#8220;the CRM numbers look wrong again.&#8221; Not missing. Wrong. The dashboard populated, every DAG run is green, and somewhere in the middle of last week&#8217;s data there is a hole where a paginated API returned a 429 and the task treated the empty body as a legitimate final page.</p>



<p class="wp-block-paragraph">That is the shape of most API pipeline incidents. Not a crash. A success that isn&#8217;t one.</p>



<p class="wp-block-paragraph">This post covers running Apache Airflow on AWS specifically for SaaS and API workloads: pulling from HubSpot, Salesforce, Stripe, Zendesk, Shopify, an internal partner API, whatever. It is organised by failure family rather than by feature, because the Airflow documentation already explains what an operator is and does a poor job of explaining which of these things will page you at 3am. I will cover choosing a deployment model, the concurrency trap that catches almost everyone, retry design, incremental state, secrets, and where the money actually goes.</p>



<h2 class="wp-block-heading">Why SaaS and API sources break differently</h2>



<p class="wp-block-paragraph">When your source is a database you control, failure is loud: connection refused, deadlock, disk full. When it is somebody else&#8217;s SaaS API, three things change.</p>



<ul class="wp-block-list">
<li><strong>You are a guest.</strong> The vendor decides your rate limit, and they can change it without telling you. Your pipeline&#8217;s correctness now depends on a number in someone else&#8217;s config file.</li>

<li><strong>Errors arrive as valid HTTP.</strong> A 429, a 200 with a truncated page, a 200 with an error object in the body. Your HTTP client is happy. Your data is not.</li>

<li><strong>Tasks spend most of their life waiting.</strong> API extraction is I/O bound almost end to end. That sounds harmless and is the root of the most expensive mistakes.</li>
</ul>



<h2 class="wp-block-heading">Pick the deployment model before you write a DAG</h2>



<p class="wp-block-paragraph">This decision constrains everything after it and is harder to reverse than people expect. Three realistic options.</p>



<h3 class="wp-block-heading">Amazon MWAA, provisioned</h3>



<p class="wp-block-paragraph">AWS runs the scheduler, web server, workers, triggerer and metadata database on Fargate; you drop DAGs into an S3 bucket and they get picked up.</p>



<p class="wp-block-paragraph">Where it wins: real Airflow, custom providers, custom plugins, full control over environment configuration. If your DAGs need arbitrary Python libraries, this option will not fight you.</p>



<p class="wp-block-paragraph">Where it doesn&#8217;t: the environment bills by the hour whether or not anything is running. There is no scale to zero on the base environment. If you sync six APIs once a day and each run takes twenty minutes, you are paying for a mostly idle cluster around the clock. The <code>mw1.micro</code> class exists precisely for the small case, but it collapses the scheduler and worker into a single Fargate task and caps worker autoscale low, so treat it as a dev or isolation tier rather than a cheap production tier.</p>



<h3 class="wp-block-heading">Amazon MWAA Serverless</h3>



<p class="wp-block-paragraph">You submit workflow definitions and AWS runs each task in its own Fargate container, billing per task duration with a one-minute minimum rather than per environment hour.</p>



<p class="wp-block-paragraph">Where it wins: spiky or infrequent schedules. If the workload is &#8220;six syncs a day, nothing overnight,&#8221; the cost profile beats a permanently running environment by a wide margin. Each workflow also gets its own IAM execution role, which is a real security improvement over one shared role per environment.</p>



<p class="wp-block-paragraph">Where it doesn&#8217;t: it leans on declarative YAML workflow definitions based on the DAG Factory format and a curated set of AWS operators. That is a deliberate trade: because the definition is declarative, the service can schedule tasks without executing your DAG code. It also means custom operators, exotic third-party providers and clever Python at parse time are not the sweet spot. It is also available in fewer regions than provisioned MWAA, so check your region before you design around it.</p>



<h3 class="wp-block-heading">Self-managed on ECS, EKS or a VPS</h3>



<p class="wp-block-paragraph">On EKS with the Kubernetes executor you get per-task pods and tight cost control. On a single VPS from a provider like InterServer or Hetzner, a Docker Compose stack with a Postgres metadata database will run a modest set of API syncs for a fraction of any managed price.</p>



<p class="wp-block-paragraph">Where it wins: cost at both extremes, and total control. Where it doesn&#8217;t: you now own metadata database upgrades, major version migrations, log retention and the 2am scheduler restart. Managed Airflow is a bet that your time is worth more than the hourly premium. For a solo engineer with three pipelines that bet often loses; for a data team of eight it usually wins. Astronomer is the main non-AWS managed option worth pricing alongside these.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: the throttle that silently stops throttling</h2>



<p class="wp-block-paragraph">You start with a normal setup: an Airflow pool named <code>crm_api</code> with four slots, and every task that touches the vendor assigned to it. Four concurrent requests, comfortably under the vendor&#8217;s limit. This works.</p>



<p class="wp-block-paragraph">Then you notice those tasks spend nearly all their runtime waiting on HTTP, burning worker slots to sit still. So you switch them to deferrable operators. A deferrable task suspends itself while waiting, releases its worker slot, and hands the waiting to the triggerer, which polls asynchronously. Worker pressure drops. Everything looks better.</p>



<p class="wp-block-paragraph">And your rate limiting quietly stops working.</p>



<p class="wp-block-paragraph">By default, a pool does not count tasks in the <em>deferred</em> state as occupying slots. That was deliberate, and the logic is sound in the abstract: a deferred task is not consuming a worker. But if you were using the pool to protect an external API rather than your own workers, it has just stopped doing the job you gave it. Every task can defer at once, and the vendor sees the full fan-out.</p>



<p class="wp-block-paragraph">The fix is a per-pool flag, <code>include_deferred</code>, which tells the scheduler to count deferred tasks against the slot budget. It is off by default. You can set it when editing the pool in the Airflow UI, or through the API.</p>



<p class="wp-block-paragraph">The failure signature is what makes this nasty. Nothing errors. Your DAG gets faster. The vendor starts returning 429s that your retry logic absorbs, and the only symptom is that runs take a little longer and occasionally a page goes missing. Weeks can pass. Two related traps in the same family:</p>



<ul class="wp-block-list">
<li><code>max_active_tasks</code> at the DAG level has the same blind spot with deferred tasks, and there is no equivalent opt-in flag. If you need a hard external concurrency cap, use a pool with <code>include_deferred</code> enabled, not DAG-level concurrency.</li>

<li>On MWAA, the triggerer runs alongside the scheduler on the same Fargate task, so scheduler count and triggerer capacity are linked. If you go heavily deferrable and your deferred tasks start stalling, scheduler capacity is the thing to look at.</li>
</ul>



<h2 class="wp-block-heading">Failure family two: retries that make the outage worse</h2>



<p class="wp-block-paragraph">The default instinct is to set <code>retries</code> high and move on. Against a rate-limited API, a fixed retry delay across many parallel tasks is just a slower version of the same stampede.</p>



<p class="wp-block-paragraph">What you want is exponential backoff with a ceiling. The shape:</p>



<pre class="wp-block-code"><code>from datetime import timedelta

from airflow.sdk import dag, task

@dag(
    schedule="0 5 * * *",
    catchup=False,
    max_active_runs=1,          # never let two runs of this DAG overlap
    default_args={
        "retries": 5,
        "retry_delay": timedelta(seconds=30),
        "retry_exponential_backoff": True,   # 30s, 60s, 120s, 240s...
        "max_retry_delay": timedelta(minutes=15),  # stop doubling here
        "pool": "crm_api",      # shared budget across every task touching this vendor
    },
    tags=["crm", "extract"],
)
def crm_extract():

    @task(max_active_tis_per_dag=4)
    def fetch_page(page_token: str) -&gt; str:
        ...

crm_extract()</code></pre>



<p class="wp-block-paragraph">The lines that matter:</p>



<ul class="wp-block-list">
<li><code>retry_exponential_backoff</code> turns <code>retry_delay</code> into a base rather than a constant, so repeated failures spread out instead of hammering in lockstep.</li>

<li><code>max_retry_delay</code> caps the doubling. Without it, a task that fails five times can sit idle for hours and blow past the window you actually cared about.</li>

<li><code>max_active_runs=1</code> is the one people skip. If a run overruns its schedule, the next one starts anyway, and now two runs are fetching the same pages from the same vendor with the same credentials. This is a common way to trigger a rate limit you have never hit before.</li>

<li><code>max_active_tis_per_dag</code> limits how many instances of that specific task run concurrently across DAG runs, which is the right knob for dynamically mapped extraction tasks.</li>
</ul>



<p class="wp-block-paragraph">One thing Airflow will not do for you: honour a <code>Retry-After</code> header. Airflow&#8217;s retry timing is computed from your config, not from the vendor&#8217;s response. If the API tells you exactly how long to wait, you have to catch that in your own code and sleep or reschedule accordingly. Ignoring a header the vendor bothered to send is a good way to get your API key throttled harder.</p>



<h2 class="wp-block-heading">Failure family three: pagination, cursors and the empty page</h2>



<p class="wp-block-paragraph">Back to the Monday message. The specific bug behind most &#8220;the numbers are wrong but nothing failed&#8221; incidents is a loop that treats any non-error response as a terminating condition. Three rules prevent it:</p>



<ol class="wp-block-list">
<li><strong>Never infer &#8220;done&#8221; from an empty result.</strong> Terminate on the explicit signal the API gives you: a null <code>next_cursor</code>, a missing <code>Link</code> header, a page count. An empty array with a valid cursor still has more data behind it.</li>

<li><strong>Assert the response shape before you use it.</strong> Check the status code explicitly and validate that the fields you depend on exist. A 200 carrying <code>{"error": "..."}</code> should raise, not return zero rows.</li>

<li><strong>Land raw, transform later.</strong> Write the untouched API response to S3 first, then parse from S3. When the vendor changes a field type, you can replay from raw instead of re-extracting from an API that no longer serves that window.</li>
</ol>



<h3 class="wp-block-heading">Where to keep incremental state</h3>



<p class="wp-block-paragraph">The tempting pattern is to store the last-seen timestamp in an Airflow Variable and update it at the end of a run. Do not make that your source of truth. If a run dies midway, the Variable is in an undefined state, and clearing and re-running the DAG will not restore it. Airflow&#8217;s retry and backfill machinery has no idea it exists.</p>



<p class="wp-block-paragraph">Better: make each run&#8217;s window a function of the run itself, and write output to a deterministic, run-scoped location such as <code>s3://bucket/source=crm/dt=&lt;logical-date&gt;/</code>. Re-running the same interval overwrites the same prefix. That is what makes a task idempotent, and idempotency is the difference between &#8220;clear the task and let it rerun&#8221; and a two-hour manual repair.</p>



<p class="wp-block-paragraph">Then overlap your windows deliberately. Many SaaS APIs order results by <em>modified</em> time with eventual consistency, so a record edited at the boundary can appear after you have already moved on. Query a window slightly wider than your schedule interval and rely on an idempotent upsert downstream to absorb the duplicates. Late-arriving data is not an edge case with SaaS sources. It is the normal case.</p>



<p class="wp-block-paragraph">Airflow&#8217;s asset-based scheduling is the clean way to trigger downstream DAGs from this: the extract DAG produces an asset, and the transform DAG runs when the asset updates, rather than being scheduled at a time you hope is late enough.</p>



<h2 class="wp-block-heading">Failure family four: credentials</h2>



<p class="wp-block-paragraph">API tokens rotate, sometimes on the vendor&#8217;s schedule rather than yours. Storing an API key in an Airflow Connection through the UI works, and is the wrong long-term answer: the value lives in the metadata database and there is no rotation story. On AWS, point Airflow&#8217;s secrets backend at AWS Secrets Manager. On MWAA that is an environment configuration option:</p>



<pre class="wp-block-code"><code>secrets.backend
  airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend

secrets.backend_kwargs
  {"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables"}</code></pre>



<p class="wp-block-paragraph">With that in place, a connection lookup for <code>crm_default</code> resolves against the secret at <code>airflow/connections/crm_default</code>. Rotating the credential is a Secrets Manager operation with no Airflow deployment involved.</p>



<p class="wp-block-paragraph">Two things to know before you turn it on. First, every connection and variable lookup becomes a Secrets Manager API call, and lookups fall through to the backend before hitting the metadata database, so a DAG that reads a Variable at parse time will generate a call on every parse cycle. Move those reads inside tasks. Second, the environment&#8217;s execution role needs explicit read permission on the relevant secret ARNs, and if you use a customer-managed KMS key, decrypt permission on that key too.</p>



<p class="wp-block-paragraph">Worth knowing if you are on Airflow 3: task code can no longer reach the metadata database directly. All runtime interaction goes through the Task Execution API. If you inherited custom operators that open a session and query Airflow&#8217;s own tables, that is a migration blocker, not a warning.</p>



<h2 class="wp-block-heading">Failure family five: the bill</h2>



<p class="wp-block-paragraph">Nobody is surprised by the environment line item. They are surprised by the other four.</p>



<ul class="wp-block-list">
<li><strong>Idle time.</strong> A provisioned MWAA environment bills continuously. Compute the ratio of hours billed to hours doing work. If it is bad, that is the argument for MWAA Serverless or for consolidating several thin pipelines into one environment.</li>

<li><strong>NAT Gateway.</strong> This is the classic one. Private-subnet workers calling public SaaS APIs route through a NAT Gateway, which charges hourly <em>and</em> per gigabyte processed. A high-volume extraction pipeline can spend more on NAT than on Airflow. VPC endpoints remove that cost for AWS service traffic, but they do nothing for calls to a third-party API, which is exactly the traffic an API pipeline generates.</li>

<li><strong>CloudWatch Logs.</strong> Task logs go to CloudWatch, and ingestion is billed per gigabyte. Set the Airflow log level per component rather than globally at DEBUG, and set a retention policy on the log groups. The default is to keep logs forever.</li>

<li><strong>S3 requests.</strong> Landing raw API responses one small object per page generates a lot of PUTs. Batch pages into larger objects where you can.</li>
</ul>



<p class="wp-block-paragraph">Rates and dimensions change, so model your own workload against the current pricing page rather than trusting a number from a blog post. The point is knowing which four lines to look at.</p>



<h2 class="wp-block-heading">Troubleshooting Apache Airflow on AWS when API pipelines misbehave</h2>



<h3 class="wp-block-heading">Tasks sit in &#8220;queued&#8221; and never start</h3>



<p class="wp-block-paragraph">Usually a slot problem, not a broken scheduler. Check, in order: is the pool full; has DAG-level <code>max_active_tasks</code> been hit; is worker autoscaling at its configured maximum. On MWAA, the container and queue utilisation metrics published to CloudWatch tell you which of the three it is far faster than reading scheduler logs.</p>



<h3 class="wp-block-heading">DAG file changes don&#8217;t appear</h3>



<p class="wp-block-paragraph">On MWAA, DAGs sync from S3 on an interval; it is not instant. If a file has been there for several minutes and still hasn&#8217;t appeared, it almost always failed to parse. Check the DAG processing logs in CloudWatch rather than the scheduler logs, because a broken import raises there and never reaches the scheduler.</p>



<h3 class="wp-block-heading">A new provider package won&#8217;t install</h3>



<p class="wp-block-paragraph">MWAA installs from your <code>requirements.txt</code> in the DAGs bucket, and from Airflow 2.7.2 onward that file must include a constraint line. Without one, MWAA picks a constraint for you, and pip is free to resolve a provider version that conflicts with the Airflow build in the image.</p>



<pre class="wp-block-code"><code>--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-&lt;AIRFLOW_VERSION&gt;/constraints-&lt;PYTHON_VERSION&gt;.txt"

apache-airflow-providers-http
apache-airflow-providers-salesforce</code></pre>



<p class="wp-block-paragraph">Substitute the literal Airflow version your environment runs and the Python version bundled with it. MWAA does not expand shell variables in that file. Test the requirements file against a local Airflow image before you upload it, because a failed install on MWAA surfaces as a partially working environment rather than a clean error.</p>



<h3 class="wp-block-heading">A backfill is stuck and you need to clear it</h3>



<p class="wp-block-paragraph">You do not need a web login token for this. MWAA exposes the Airflow REST API through a signed AWS API call, so you can drive it from CI or a runbook with normal IAM credentials:</p>



<pre class="wp-block-code"><code>aws mwaa invoke-rest-api 
  --name MyMWAAEnvironment 
  --path "/dags/crm_extract/clearTaskInstances" 
  --method POST 
  --body '{"dry_run": true}'</code></pre>



<p class="wp-block-paragraph">Start with <code>dry_run</code> set to true so the response tells you which task instances would be cleared before you actually clear them. Note that the resource paths differ between Airflow 2 and Airflow 3 environments, so confirm against the API version your environment exposes.</p>



<h3 class="wp-block-heading">Deferred tasks stall forever</h3>



<p class="wp-block-paragraph">If deferred tasks stop resuming while the environment reports healthy, suspect the triggerer rather than your DAG. A triggerer that has lost its ability to process triggers can keep heartbeating normally, so the scheduler sees nothing wrong while every deferred task drifts toward timeout. This class of bug has been fixed and re-fixed upstream, so check your Airflow version&#8217;s release notes before assuming it is your code.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Switching to deferrable operators without enabling <code>include_deferred</code> on the pools that were protecting the API.</li>

<li>Leaving <code>max_active_runs</code> unset, so a slow run and the next scheduled run compete for the same rate limit budget.</li>

<li>Treating an empty response page as the end of pagination.</li>

<li>Storing the incremental watermark in an Airflow Variable and updating it mid-run.</li>

<li>Calling an API or reading a Variable at DAG parse time, which executes on every parse cycle rather than once per run.</li>

<li>Transforming during extraction, so a vendor schema change means re-pulling data the API may no longer serve.</li>

<li>Sizing the environment for peak concurrency when the actual constraint is the vendor&#8217;s rate limit.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>One pool per vendor, sized to their published limit with headroom, and <code>include_deferred</code> enabled on every one of them.</li>

<li>Land raw responses to S3 before parsing. Extraction and transformation are separate tasks with separate failure modes.</li>

<li>Make every task idempotent and window-scoped, so &#8220;clear and rerun&#8221; is always a safe repair.</li>

<li>Overlap extraction windows and deduplicate downstream rather than trusting a vendor&#8217;s timestamps to be exact.</li>

<li>Secrets Manager for credentials, with the execution role scoped to specific secret ARNs.</li>

<li>Alert on row counts and freshness, not just task state. A green DAG that produced 40% of yesterday&#8217;s rows is the failure you actually care about. Shipping Airflow&#8217;s StatsD metrics into Prometheus, Grafana Cloud or Datadog makes that a dashboard rather than a discovery.</li>

<li>Define the environment in Terraform or OpenTofu. Recreating an MWAA environment by hand after a bad configuration change is a bad afternoon.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is MWAA worth it compared to self-hosting Airflow on EC2?</h3>



<p class="wp-block-paragraph">It depends almost entirely on how many people share the platform. MWAA&#8217;s premium buys you managed metadata database upgrades, patched images and version migration support. If one engineer maintains three DAGs, self-hosting on a modest VPS is cheaper and the operational load is real but small. Once several teams depend on the scheduler being up, the premium is easy to justify.</p>



<h3 class="wp-block-heading">Should I use Step Functions instead of Airflow for API pipelines?</h3>



<p class="wp-block-paragraph">Step Functions is genuinely better for event-driven, AWS-service-centric orchestration with modest branching, and it scales to zero. Airflow wins when you need scheduled batch semantics, backfills over historical windows, dependencies between many pipelines, and a UI that non-platform engineers can use to see why last Tuesday failed. Backfill is usually the deciding feature.</p>



<h3 class="wp-block-heading">Do deferrable operators reduce my AWS bill?</h3>



<p class="wp-block-paragraph">On provisioned MWAA, they reduce worker <em>pressure</em>, which reduces autoscaling into additional worker instances. The base environment cost is unchanged. On a Kubernetes executor setup where each task is a pod, the saving is more direct. Either way, do not adopt them purely for cost without revisiting your pool configuration first.</p>



<h3 class="wp-block-heading">How do I handle a vendor with no documented rate limit?</h3>



<p class="wp-block-paragraph">Start conservative, one or two concurrent requests, and instrument the response status codes. Raise the pool size gradually and watch for 429s or rising latency. Latency creeping up under load is often the earlier signal, because some vendors throttle by slowing you down before they start rejecting.</p>



<h3 class="wp-block-heading">Can Airflow read a Retry-After header automatically?</h3>



<p class="wp-block-paragraph">No. Airflow computes retry timing from <code>retry_delay</code> and the backoff settings on the task. If a vendor sends <code>Retry-After</code>, you need to handle it in your own request code or in a custom operator.</p>



<h3 class="wp-block-heading">What breaks when upgrading to Airflow 3?</h3>



<p class="wp-block-paragraph">The big one for API pipelines is that task code can no longer access the metadata database directly; everything goes through the Task Execution API. Imports also move to the <code>airflow.sdk</code> namespace, and several core operators now live in the standard provider package. Audit custom operators first, since that is where direct database access hides. MWAA requires you to be on the latest Airflow 2 minor version before a major upgrade, so plan two steps.</p>



<h3 class="wp-block-heading">How many DAGs can one MWAA environment handle?</h3>



<p class="wp-block-paragraph">The binding constraint is usually the metadata database and scheduler CPU, not DAG count. Watch metadata database memory and scheduler CPU utilisation; when either saturates, you either move up an environment class or split into multiple environments. Splitting also gives you blast-radius isolation, which matters more than people expect.</p>



<h2 class="wp-block-heading">Wrapping up</h2>



<p class="wp-block-paragraph">Running Apache Airflow on AWS for SaaS and API pipelines is mostly not an Airflow problem. The scheduler works. The operators work. What bites is the gap between &#8220;the task succeeded&#8221; and &#8220;the data is correct,&#8221; and that gap lives in concurrency settings, pagination logic and retry design rather than anywhere Airflow will warn you about.</p>



<p class="wp-block-paragraph">If you take one thing away: <strong>a green DAG is not a signal that your data is complete.</strong> Enable <code>include_deferred</code> on the pools protecting your vendors, terminate pagination on an explicit signal instead of an empty page, make every task idempotent, and alert on row counts. Those four things prevent most of the incidents that never show up as a failed task.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with your Airflow pipelines on AWS?</h2>



<p class="wp-block-paragraph">I work with teams running data and API pipelines on AWS, usually somewhere between &#8220;it works but nobody trusts it&#8221; and &#8220;we need to move off cron.&#8221; Things I can help with:</p>



<ul class="wp-block-list">
<li>Reviewing existing DAGs for silent data loss: pagination logic, retry behaviour, pool and concurrency configuration.</li>

<li>Choosing between MWAA provisioned, MWAA Serverless and self-managed Airflow, with a cost model for your actual schedule rather than a generic comparison.</li>

<li>Building SaaS extraction pipelines that are idempotent and safely re-runnable, landing raw to S3 with incremental windows that survive failure.</li>

<li>Cutting MWAA cost: environment right-sizing, NAT Gateway traffic, CloudWatch log volume and dependency install time.</li>

<li>Airflow 2 to 3 migration audits, focused on custom operators and direct metadata database access.</li>

<li>Data freshness and volume alerting in Grafana or CloudWatch, so you learn about a partial sync before the business does.</li>
</ul>



<p class="wp-block-paragraph">If something specific is broken, send me the DAG file, the task log, or the CloudWatch metrics for the run that went wrong. It is usually faster to look at the real thing than to describe it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/apache-airflow-aws-saas-api-pipelines/">Apache Airflow on AWS: Building SaaS and API Pipelines That Don&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a Jira Analytics Pipeline with AWS Lambda and Athena (Without Double-Counting Everything)</title>
		<link>https://john-nessime.com/blog/devops/jira-analytics-pipeline-aws-lambda-athena/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 04 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Business Intelligence]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Engineering Metrics]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Jira]]></category>
		<category><![CDATA[Partition Projection]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[SQL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=125</guid>

					<description><![CDATA[<p>Jira's built-in reports stop at the board boundary. This guide walks through a Jira analytics pipeline built on AWS Lambda, S3 and Athena, organised around the four failure families that actually bite: the removed search endpoint, silently truncated changelogs, incremental loads that duplicate rows, and an S3 layout that quietly inflates your query bill.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/jira-analytics-pipeline-aws-lambda-athena/">Building a Jira Analytics Pipeline with AWS Lambda and Athena (Without Double-Counting Everything)</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone in a delivery review asks why cycle time went up last quarter. You open the Jira dashboard, and it can tell you what is in progress right now, roughly. It cannot tell you what &#8220;in progress&#8221; meant three months ago, how long each ticket sat in code review, or whether the increase came from one team or all six. The control chart resets when the board configuration changes, the sprint report only knows about sprints, and the CSV export tops out somewhere unhelpful.</p>



<p class="wp-block-paragraph">That is usually the moment someone says &#8220;let&#8217;s just pull it into a warehouse.&#8221; A <strong>Jira analytics pipeline</strong> built on AWS Lambda, S3 and Athena is a reasonable answer to that, and it is genuinely cheap to run. It is also easy to build a version that looks correct for two weeks and then quietly reports numbers that are thirty percent wrong.</p>



<p class="wp-block-paragraph">This post covers the extraction and modelling problems that actually cost you time: the search endpoint Atlassian removed, the change history that truncates without erroring, the incremental load pattern that duplicates rows across partitions, and the S3 layout decisions that decide whether Athena costs you pennies or hundreds. Code where it clarifies something, and honest notes on what I would skip.</p>



<h2 class="wp-block-heading">Where Jira&#8217;s own reporting genuinely stops</h2>



<p class="wp-block-paragraph">Give the built-in tooling its due first. Jira&#8217;s velocity, burndown and control charts are fine for a single team inspecting its own recent work, they need no infrastructure, and they update instantly. Marketplace apps like eazyBI and Custom Charts cover a lot of ground without you writing a line of Python. If your question is &#8220;how did this sprint go,&#8221; you do not need a pipeline.</p>



<p class="wp-block-paragraph">Where it stops is anything that crosses a boundary. Comparing lead time across projects that use different workflows. Joining ticket data to deploy events from your CI system or incident data from PagerDuty. Retaining a consistent view of history after someone renames a status or archives a board. Answering a question nobody anticipated when the board was configured. Those need the raw data somewhere you control, in a shape you decide.</p>



<h2 class="wp-block-heading">The shape of the pipeline</h2>



<p class="wp-block-paragraph">The architecture is unremarkable, which is the point:</p>



<ol class="wp-block-list">
<li>EventBridge Scheduler triggers a Lambda function on a schedule.</li>

<li>Lambda reads a Jira API token from Secrets Manager and pages through the Jira Cloud REST API.</li>

<li>It writes Parquet files to S3, partitioned by load date.</li>

<li>The Glue Data Catalog holds the table definitions, with partition projection so nothing has to crawl.</li>

<li>Athena queries S3 directly. Grafana, Power BI, Metabase or QuickSight sit on top of Athena.</li>
</ol>



<p class="wp-block-paragraph">No cluster, no always-on database, nothing to patch. The whole thing costs about as much as a small EC2 instance for a mid-sized Jira site, and most of that is S3 storage. Terraform or CloudFormation to define it, GitHub Actions to deploy it.</p>



<p class="wp-block-paragraph">The complexity is not in the wiring. It is in four places, and they are worth taking in order.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family 1: the search endpoint you were probably going to use is gone</h2>



<p class="wp-block-paragraph">Almost every Jira extraction tutorial and a good number of client libraries still reach for <code>GET /rest/api/3/search</code>. Atlassian removed it from Jira Cloud. It returns 410 Gone. If you are copying a script from a blog post that predates the change, this is the first thing that breaks, and at least it breaks loudly.</p>



<p class="wp-block-paragraph">The replacement is <code>/rest/api/3/search/jql</code>, available as both GET and POST. Use POST for anything real, because JQL strings get long and you avoid URL encoding entirely. Three behavioural changes matter more than the URL:</p>



<h3 class="wp-block-heading">Pagination is cursor-based, and there is no total</h3>



<p class="wp-block-paragraph"><code>startAt</code> is gone. You get a <code>nextPageToken</code> back and hand it to the next request. There is no <code>total</code> in the response, which means any progress bar, any &#8220;expected N rows&#8221; sanity check, and any loop that terminated on <code>startAt &gt;= total</code> needs rewriting.</p>



<p class="wp-block-paragraph">If you only need a count, there is a separate operation, <code>POST /rest/api/3/search/approximate-count</code>, which takes a JQL body and returns an approximate figure without paging through results. It is genuinely useful as a reconciliation check: run it before extraction, compare against the row count you actually wrote, and alert on a large gap.</p>



<p class="wp-block-paragraph">There have been persistent community reports of <code>isLast</code> behaving unreliably on this endpoint, including tokens that chain without ever terminating. Do not trust <code>isLast</code> as your loop condition. Terminate on the absence of <code>nextPageToken</code>, and put a hard page cap in as a circuit breaker so a bad token cannot burn your entire Lambda budget in one invocation.</p>



<pre class="wp-block-code"><code>def search_issues(session, base_url, jql, fields, max_pages=2000):
    """Page through /search/jql. Terminates on missing nextPageToken,
    not on isLast, which has been reported as unreliable."""
    token = None
    for _ in range(max_pages):
        body = {"jql": jql, "fields": fields, "maxResults": 100}
        if token:
            body["nextPageToken"] = token
        r = session.post(f"{base_url}/rest/api/3/search/jql",
                         json=body, timeout=60)
        r.raise_for_status()
        page = r.json()
        for issue in page.get("issues", []):
            yield issue
        token = page.get("nextPageToken")
        if not token:
            return
    raise RuntimeError("page cap hit, refusing to loop further")</code></pre>



<h3 class="wp-block-heading">You have to ask for fields explicitly</h3>



<p class="wp-block-paragraph">The new endpoint does not hand you every field by default. Omit <code>fields</code> and you get essentially nothing back. This is the failure that looks like success: the pipeline runs, files land in S3, row counts look plausible, and every analytical column is null.</p>



<p class="wp-block-paragraph">Be explicit and be narrow. Every field you request costs response size, and Jira sites accumulate hundreds of custom fields nobody uses. Name what you need:</p>



<pre class="wp-block-code"><code>FIELDS = [
    "summary", "status", "issuetype", "project", "priority",
    "assignee", "reporter", "created", "updated", "resolutiondate",
    "labels", "components", "parent",
    "customfield_10016",   # story points on this site, verify yours
]</code></pre>



<p class="wp-block-paragraph">Custom field IDs are per-site. Do not hardcode one you read in someone else&#8217;s blog post. Pull <code>/rest/api/3/field</code> once, find the field by name, and either store the mapping in config or resolve it at runtime and log what it resolved to. When a Jira admin rebuilds a field, an ID-based pipeline goes null and a name-resolving pipeline keeps working.</p>



<h3 class="wp-block-heading">Rate limits and the fifteen-minute wall</h3>



<p class="wp-block-paragraph">Jira Cloud applies cost-based rate limiting and returns HTTP 429 when you exceed it, typically with a <code>Retry-After</code> header. Respect that header rather than inventing your own backoff. A naive retry loop that ignores it turns a brief throttle into a sustained one.</p>



<p class="wp-block-paragraph">Lambda&#8217;s hard ceiling is fifteen minutes. A full historical backfill of a large Jira site will not finish in one invocation, and the ugly failure mode is a function that times out at minute fifteen having written half its data with no record of where it stopped. Two ways out:</p>



<ul class="wp-block-list">
<li><strong>Shard the work.</strong> Fan out one Lambda invocation per project key, or per month of created date. Each one is small, independently retryable, and finishes well inside the limit.</li>

<li><strong>Checkpoint and continue.</strong> Persist the current <code>nextPageToken</code> to DynamoDB or S3 after each page. When the function is close to its deadline, stop cleanly and let Step Functions re-invoke it from the checkpoint.</li>
</ul>



<p class="wp-block-paragraph">Sharding is simpler and I reach for it first. Checkpointing is what you need when a single project is itself too large. Either way, watch Lambda&#8217;s ephemeral storage: the default <code>/tmp</code> allocation is 512 MB, and buffering a large Parquet write there will fail before your API calls do. Raise it or stream to S3 in chunks.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family 2: the change history is the whole point, and it truncates silently</h2>



<p class="wp-block-paragraph">Current issue state answers almost none of the interesting questions. How long a ticket spent waiting for review, how many times it bounced back from QA, when it actually entered development rather than when someone remembered to drag the card, all of that lives in the changelog. Without it you have a list of tickets. With it you have a process.</p>



<p class="wp-block-paragraph">Here is the trap. Requesting an issue with <code>expand=changelog</code> returns a capped number of history entries, commonly the first hundred, and it does not tell you it truncated. Well-worn tickets with lots of field edits blow past that easily. Your data does not error, it just quietly loses the later transitions, which are usually the ones near completion. Cycle time comes out looking better than reality.</p>



<p class="wp-block-paragraph">Two correct approaches:</p>



<ul class="wp-block-list">
<li><code>GET /rest/api/3/issue/{issueIdOrKey}/changelog</code> and page it properly. Correct, but it is one request per issue, which is brutal against rate limits on a large site.</li>

<li><code>POST /rest/api/3/changelog/bulkfetch</code>, which accepts <code>issueIdsOrKeys</code> and an optional <code>fieldIds</code> filter, with the same <code>nextPageToken</code> pagination. Far fewer round trips.</li>
</ul>



<p class="wp-block-paragraph">Bulk fetch is the one I would use, with a caveat: it has carried an experimental designation, so pin your expectations and keep the per-issue path available as a fallback. Filter <code>fieldIds</code> to <code>status</code> if status history is all you model. That cuts the response size enormously, because most changelog volume is description edits and label churn nobody analyses.</p>



<p class="wp-block-paragraph">Store changelog as its own narrow table, one row per field change, not nested inside the issue record. Athena can handle nested structures, but flat is dramatically easier to reason about in SQL and much cheaper to scan.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family 3: the incremental load that duplicates everything</h2>



<p class="wp-block-paragraph">This is the one that bites hardest, because nothing fails. Everything runs green for weeks and the numbers are wrong the entire time.</p>



<p class="wp-block-paragraph">The obvious incremental design is a watermark: track the last successful run, then pull <code>updated &gt;= watermark</code> on each run and append the results to S3. It is the standard pattern and it works for immutable event data.</p>



<p class="wp-block-paragraph">Jira issues are not immutable. A ticket created in January and touched every week appears in every weekly extract. If you append each run into its own partition, that one issue now exists as a row in twenty partitions. Athena is doing exactly what you asked: <code>SELECT count(*) FROM jira_issues WHERE status = 'Open'</code> counts twenty things. Your open bug count is inflated, your throughput is inflated, and the inflation is proportional to how much a ticket gets edited, which correlates with how contentious it was. The busiest work is the most over-counted.</p>



<p class="wp-block-paragraph">Nobody catches this from the dashboard, because the numbers move in the right direction and look approximately sane. It surfaces months later when someone reconciles against a JQL query in Jira and the two disagree.</p>



<h3 class="wp-block-heading">Pick one of three fixes, deliberately</h3>



<p class="wp-block-paragraph"><strong>Full snapshot per load, dedupe at read time.</strong> Keep every version, partition by <code>load_date</code>, and always read through a view that takes the latest row per issue key. Storage is cheap, history is free, and you can answer &#8220;what did the board look like in March&#8221; without any extra machinery. The cost is that every query pays for the deduplication.</p>



<pre class="wp-block-code"><code>CREATE OR REPLACE VIEW jira_issues_current AS
SELECT * FROM (
  SELECT
    i.*,
    ROW_NUMBER() OVER (
      PARTITION BY issue_key
      ORDER BY load_date DESC, updated DESC
    ) AS rn
  FROM jira_issue_snapshot i
  WHERE load_date &gt;= date_format(current_date - interval '7' day, '%Y-%m-%d')
) WHERE rn = 1;</code></pre>



<p class="wp-block-paragraph">The <code>load_date</code> filter inside the view matters. Without it the deduplication window scans the entire table on every query, which is the single most common way a cheap Athena setup becomes an expensive one.</p>



<p class="wp-block-paragraph"><strong>Overwrite the affected partitions.</strong> Partition by something stable, usually issue created month, and rewrite whole partitions when any issue in them changes. Clean reads, no dedupe cost, but you now own read-modify-write logic in Lambda and a concurrency problem if two runs overlap.</p>



<p class="wp-block-paragraph"><strong>Use an ACID table format.</strong> Apache Iceberg gives you real <code>MERGE INTO</code> semantics on S3, and Athena supports it natively. This is the right answer if you are already running Iceberg elsewhere or if the pipeline will grow to a dozen sources. It is not worth adopting solely to load one Jira site, because you inherit compaction and snapshot expiry as ongoing maintenance.</p>



<p class="wp-block-paragraph">For a single Jira site feeding a handful of dashboards, snapshot plus a dedupe view is what I would build. It has the fewest moving parts and it gives you point-in-time history as a side effect, which you will want the first time someone asks a retrospective question.</p>



<h3 class="wp-block-heading">Deletions and moves</h3>



<p class="wp-block-paragraph">A JQL watermark query never returns deleted issues, so they persist in your data forever. Same for issues moved out of scope or into an archived project. Periodically reconcile: pull the full set of issue keys with a minimal <code>fields</code> list, compare against what you hold, and mark the difference. Monthly is usually enough. Skip this and your historical counts drift upward permanently.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family 4: the S3 layout that decides your Athena bill</h2>



<p class="wp-block-paragraph">Athena&#8217;s standard pricing model bills on bytes scanned, rounded up, with a small per-query minimum. DDL statements are free. That means your storage layout, not your SQL, is what determines cost.</p>



<p class="wp-block-paragraph">Three things do almost all the work:</p>



<ul class="wp-block-list">
<li><strong>Write Parquet, not JSON.</strong> Columnar storage lets Athena read only the columns your query touches. A dashboard selecting five columns from a forty-column table scans a small fraction of what the equivalent JSON would. Snappy or Zstd compression on top of that.</li>

<li><strong>Avoid tiny files.</strong> A Lambda that runs hourly and writes one small file per invocation produces thousands of objects. Athena spends more time opening files than reading them. Batch writes so files land in the low hundreds of megabytes, or run a periodic compaction job.</li>

<li><strong>Use partition projection.</strong> This is the one people skip and then wonder why queries have a fixed few-second overhead.</li>
</ul>



<p class="wp-block-paragraph">Partition projection lets Athena calculate partition locations from table properties instead of calling <code>GetPartitions</code> against the Glue Data Catalog. On a table with a couple of years of daily partitions, that lookup is real latency on every single query, and it grows as the table does. Projection removes it, and removes your need for a Glue crawler entirely, which is one less scheduled job and one less IAM role.</p>



<pre class="wp-block-code"><code>CREATE EXTERNAL TABLE jira_issue_snapshot (
  issue_id        string,
  issue_key       string,
  project_key     string,
  issue_type      string,
  status          string,
  status_category string,
  assignee_id     string,
  created         timestamp,
  updated         timestamp,
  resolutiondate  timestamp,
  story_points    double
)
PARTITIONED BY (load_date string)
STORED AS PARQUET
LOCATION 's3://example-jira-lake/issue_snapshot/'
TBLPROPERTIES (
  'projection.enabled'                = 'true',
  'projection.load_date.type'         = 'date',
  'projection.load_date.format'       = 'yyyy-MM-dd',
  'projection.load_date.range'        = 'NOW-3YEARS,NOW',
  'projection.load_date.interval'     = '1',
  'projection.load_date.interval.unit'= 'DAYS',
  'storage.location.template'         =
    's3://example-jira-lake/issue_snapshot/load_date=${load_date}'
);</code></pre>



<p class="wp-block-paragraph">One caution that surprises people: projection describes partitions Athena will look for, not partitions that exist. Set a range wider than your data and queries without a <code>load_date</code> filter will probe empty prefixes. Keep the range tight to what you actually hold.</p>



<p class="wp-block-paragraph">Finally, set <code>BytesScannedCutoffPerQuery</code> on the Athena workgroup. It kills any query that exceeds a scan threshold. One analyst running <code>SELECT *</code> against three years of data in a BI tool&#8217;s preview pane is the classic surprise line item, and this stops it at the source.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Modelling time in status</h2>



<p class="wp-block-paragraph">Once status changelog rows are landed flat, the core metric is a window function. Each transition&#8217;s duration is the gap to the next transition on the same issue:</p>



<pre class="wp-block-code"><code>WITH transitions AS (
  SELECT
    issue_key,
    to_status,
    changed_at,
    LEAD(changed_at) OVER (
      PARTITION BY issue_key ORDER BY changed_at
    ) AS next_changed_at
  FROM jira_changelog
  WHERE field_id = 'status'
    AND load_date &gt;= date_format(current_date - interval '90' day, '%Y-%m-%d')
)
SELECT
  issue_key,
  to_status,
  SUM(date_diff('second', changed_at,
                COALESCE(next_changed_at, current_timestamp))) / 3600.0
    AS hours_in_status
FROM transitions
GROUP BY issue_key, to_status;</code></pre>



<p class="wp-block-paragraph">The <code>COALESCE</code> handles the current status, which has no successor transition. Two modelling decisions to make consciously: whether to subtract non-working hours, and how to treat a ticket that moves backwards through the workflow. Both are business questions, not technical ones, and both should be settled in a documented view rather than reimplemented in each dashboard.</p>



<p class="wp-block-paragraph">Map raw status names to a stable category early. Teams rename statuses constantly, and a metric keyed on the literal string breaks the moment someone changes &#8220;In Review&#8221; to &#8220;Peer Review&#8221;. A small mapping table joined at query time keeps history comparable.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Every analytical column is null.</strong> You did not pass <code>fields</code>, or you passed a custom field ID from another site. Log the resolved field list on every run.</li>

<li><strong>Counts higher than the same JQL in Jira.</strong> Duplicate rows across partitions. Check whether your query goes through the dedupe view or straight at the base table.</li>

<li><strong>Extraction loops forever.</strong> Do not terminate on <code>isLast</code>. Terminate on missing <code>nextPageToken</code> and keep a page cap.</li>

<li><strong>HTTP 410 from the API.</strong> You are still calling the removed <code>/rest/api/3/search</code>. Check your client library version too, not just your own code.</li>

<li><strong>Cycle times suspiciously low.</strong> Truncated changelog. Move to the dedicated changelog endpoint or bulk fetch.</li>

<li><strong>Athena returns zero rows but the files are there.</strong> Partition projection range does not cover the partition, or the S3 prefix does not match <code>storage.location.template</code>. Compare a real object key against the template character by character.</li>

<li><strong>Queries slow down as the table grows.</strong> Either you have no projection and Glue lookups dominate, or you have accumulated small files.</li>

<li><strong>Authentication failures overnight with no deploy.</strong> API token expired or was revoked. Alert on the specific status code rather than on &#8220;run failed&#8221;.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Appending incremental extracts without a deduplication strategy.</li>

<li>Building on current issue state and adding changelog later. Retrofitting history is far more work than including it from the start.</li>

<li>Hardcoding custom field IDs copied from documentation.</li>

<li>Storing the API token in a Lambda environment variable instead of Secrets Manager.</li>

<li>Running an hourly schedule when the dashboard is read once a day. You pay in small files and API quota for freshness nobody uses.</li>

<li>Letting BI tools query base tables directly instead of curated views.</li>

<li>Never reconciling against Jira. A scheduled check comparing approximate count to your row count catches drift within a day.</li>
</ul>



<h2 class="wp-block-heading">Best practices for a Jira analytics pipeline that survives</h2>



<ul class="wp-block-list">
<li>Land raw API responses to S3 before transforming. When your parsing is wrong, and it will be, you replay from raw rather than re-hammering the API.</li>

<li>Make the load idempotent. Re-running for the same date should produce the same result, not a second copy.</li>

<li>Define the pipeline in Terraform or CloudFormation and deploy it from CI. The IAM policy for Lambda and Athena is fiddly and you do not want to rebuild it from memory.</li>

<li>Alert on a run that succeeds with zero rows, not just on runs that error. Silent empty loads are the more common failure.</li>

<li>Scope the Jira token to a service account with read access to exactly the projects you need.</li>

<li>Put an S3 lifecycle policy on the raw zone. It grows faster than you expect and nobody queries last year&#8217;s raw JSON.</li>

<li>Expose curated views, not tables, to Grafana, Metabase, Power BI or QuickSight, and enable Athena&#8217;s query result reuse for repeated dashboard loads.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Should I use Lambda or Glue for Jira extraction?</h3>



<p class="wp-block-paragraph">Lambda, for an API-driven pull. The work is mostly waiting on HTTP responses, which Spark&#8217;s distributed compute does nothing for, and you would be paying for a Glue job&#8217;s minimum billing on something that is idle. Glue earns its place downstream, if you have heavy joins across several sources. Athena CTAS often covers that too.</p>



<h3 class="wp-block-heading">How often should the pipeline run?</h3>



<p class="wp-block-paragraph">Match the decision cadence, not the data cadence. Delivery metrics are reviewed weekly or in sprint ceremonies, so daily is almost always enough and produces far better file sizes. Reserve hourly for something with a real-time consumer, and know that you are buying that freshness with small-file overhead and API quota.</p>



<h3 class="wp-block-heading">Can I use webhooks instead of polling?</h3>



<p class="wp-block-paragraph">You can, and for near-real-time reaction it is the right tool. For analytics it is a poor primary source, because a missed delivery leaves a permanent hole you have no way to detect. The pattern that works is webhooks for freshness plus a scheduled reconciliation pull as the source of truth. If you only build one, build the scheduled pull.</p>



<h3 class="wp-block-heading">Does this work with Jira Data Center or Server?</h3>



<p class="wp-block-paragraph">The AWS half is identical. The extraction half is not. Cursor pagination and the <code>/search/jql</code> endpoint are Cloud-only changes, so self-hosted instances still use the older offset-based <code>/rest/api/2/search</code>. Write the extractor behind an interface if you need to support both, and expect the auth model to differ as well.</p>



<h3 class="wp-block-heading">How much does an Athena-based Jira pipeline cost to run?</h3>



<p class="wp-block-paragraph">The mechanism matters more than any figure I could quote. You pay for S3 storage, Lambda invocation time, and Athena per byte scanned with a small per-query minimum. Jira issue data compresses extremely well as Parquet, so storage stays small. The variable is dashboard query volume, which is why partitioning and workgroup scan limits matter. Model it against current AWS rates and your own expected query count.</p>



<h3 class="wp-block-heading">Why not just use eazyBI or a Jira reporting app?</h3>



<p class="wp-block-paragraph">Often you should. If your questions stay inside Jira, a Marketplace app gets you there in an afternoon with no infrastructure. The case for a pipeline is joining Jira to data that lives elsewhere, retaining history the app does not, or needing your data in a warehouse you already run. Build the pipeline when the app has actually failed you, not in anticipation.</p>



<h3 class="wp-block-heading">Should I load into Redshift instead of querying S3?</h3>



<p class="wp-block-paragraph">Only if you have concurrent BI users hitting the same tables constantly and Athena&#8217;s per-query latency is a real complaint. Jira data volumes are small, the query pattern is bursty, and Athena&#8217;s zero-idle-cost model fits that far better. Redshift makes sense as a consolidation layer across many sources, not for one issue tracker.</p>



<h2 class="wp-block-heading">The one thing to carry away</h2>



<p class="wp-block-paragraph">The hard part of a Jira analytics pipeline is not the AWS wiring. Lambda, S3, Glue and Athena will be working within a day. The hard part is that Jira issues are mutable, so an incremental load that appends is an incremental load that duplicates, and it does so without a single error in your logs.</p>



<p class="wp-block-paragraph">Decide your deduplication strategy before you write the first extract, not after someone reconciles a dashboard against JQL and finds a gap. Snapshot with a read-time dedupe view, partition projection so queries stay fast as history accumulates, and a scheduled reconciliation check that alerts on drift. Get those three right and the rest is plumbing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need this built or fixed?</h2>



<p class="wp-block-paragraph">I design and run serverless data pipelines on AWS, and Jira extraction has more sharp edges than most sources. Things I can help with directly:</p>



<ul class="wp-block-list">
<li>Migrating an extractor off the removed <code>/rest/api/3/search</code> endpoint to cursor-based <code>/search/jql</code> without losing rows in the cutover</li>

<li>Auditing an existing Jira pipeline for duplicate rows and reconciling your numbers back against JQL</li>

<li>Building changelog-based cycle time and time-in-status models that survive workflow renames</li>

<li>Restructuring an S3 layer with Parquet, partition projection and file compaction to cut Athena scan costs</li>

<li>Packaging the whole thing as Terraform or CloudFormation with CI deployment and least-privilege IAM</li>

<li>Connecting Athena to Grafana, Power BI, Metabase or QuickSight with curated views instead of raw tables</li>
</ul>



<p class="wp-block-paragraph">Send me your extractor code, a Glue table definition, or an Athena query that is scanning more than it should, and I will tell you what I would change.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/jira-analytics-pipeline-aws-lambda-athena/">Building a Jira Analytics Pipeline with AWS Lambda and Athena (Without Double-Counting Everything)</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Run It Twice, Get Two Answers: Building an ETL Pipeline From Zoho CRM to Amazon S3</title>
		<link>https://john-nessime.com/blog/devops/zoho-crm-to-amazon-s3/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 02 Aug 2026 18:16:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[Zoho]]></category>
		<category><![CDATA[Zoho CRM]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=86</guid>

					<description><![CDATA[<p>You re-run the same extract for the same window and get a different set of rows. Nothing errored. You were paginating a result set that kept changing while you read it. Here's how to build a Zoho CRM to S3 pipeline whose runs are repeatable, from closed read windows to Bulk Read and deletions.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/zoho-crm-to-amazon-s3/">Run It Twice, Get Two Answers: Building an ETL Pipeline From Zoho CRM to Amazon S3</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The monthly deal count in your warehouse is short by about thirty rows. You re-run the extract for exactly the same window, expecting to confirm the bug, and this time you get a different thirty. Run it a third time and the number moves again.</p>



<p class="wp-block-paragraph">Nothing is broken in the way you are looking for. The records exist in Zoho. Your job did not error. What happened is that you asked for every deal modified since your last watermark, sorted, and then walked through it page by page while sales reps kept working. Records got modified during the read, changed their position in the sort order, and shifted from page 4 to page 2 after you had already read page 2. So you never saw them.</p>



<p class="wp-block-paragraph">That is the failure that defines building an <strong>ETL pipeline from Zoho CRM to Amazon S3</strong>, and it does not look like a bug. It looks like flaky data. It only shows up on busy modules, it never produces an error, and re-running the job produces a different wrong answer, which is the single most confusing symptom in data engineering.</p>



<p class="wp-block-paragraph">The fix is one rule, and everything else in this post follows from it. This covers that rule, which extraction API to use and when, authentication and the data centre trap that wastes an afternoon, handling deletions (Zoho is genuinely good here), shaping nested CRM JSON for S3, and staying inside your API credits.</p>



<h2 class="wp-block-heading">The rule: bound both ends of the window</h2>



<p class="wp-block-paragraph">Almost every incremental extract is written like this:</p>



<pre class="wp-block-code"><code>Modified_Time &gt; {last_run}</code></pre>



<p class="wp-block-paragraph">That query has no upper bound, which means the result set keeps growing while you read it. You are paginating a moving target. Add an upper bound and put it slightly in the past:</p>



<pre class="wp-block-code"><code>Modified_Time &gt; {last_run} AND Modified_Time &lt; {now_minus_lag}</code></pre>



<p class="wp-block-paragraph">Now the set is frozen. Records modified during your run land after the upper bound and get picked up next time. The extract becomes repeatable: run it twice, get the same rows twice. That property is worth more than any amount of retry logic, because it means a failed run costs you nothing and a suspicious number can be checked by re-running.</p>



<p class="wp-block-paragraph">Two details that matter. The lag needs to comfortably exceed how long your extract takes plus any clock skew between you and Zoho; ten minutes is a sensible starting point and costs you ten minutes of freshness. And <strong>only advance the watermark after the entire window has landed in S3</strong>, never after the API call succeeds. Those are different moments, and the gap between them is where data goes missing.</p>



<pre class="wp-block-code"><code>#!/usr/bin/env bash
set -euo pipefail

LAG_MINUTES=10
T1=$(cat state/deals.watermark)
T2=$(date -u -d "-${LAG_MINUTES} minutes" +%Y-%m-%dT%H:%M:%S+00:00)

extract_window "$T1" "$T2"

# Watermark advances only once the data is durably in S3.
echo "$T2" &gt; state/deals.watermark</code></pre>



<p class="wp-block-paragraph">Keep that state somewhere durable and versioned, not on the box running the job. DynamoDB, Parameter Store, or a small object in S3 all work. A watermark file on an ephemeral runner is a watermark you will lose.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Authentication, and the trap that costs an afternoon</h2>



<p class="wp-block-paragraph">Zoho uses OAuth 2.0. You register a client in the Zoho API Console, get a client ID and secret, exchange a grant token for a refresh token once, and then trade the refresh token for short-lived access tokens from then on. Access tokens last an hour, so cache them rather than requesting one per call.</p>



<p class="wp-block-paragraph">Here is the part that catches everyone: <strong>Zoho runs multiple data centres and they are separate worlds.</strong> An account in the EU data centre authenticates against a different accounts domain and calls a different API domain than a US account. A token issued in one region is meaningless in another, and the error you get back does not say &#8220;wrong data centre&#8221;, it says the token is invalid.</p>



<p class="wp-block-paragraph">So make the domain a configuration value from the first line of code, never a hardcoded string. The examples below use the US domain. If your org lives elsewhere, both the accounts domain and the API domain change together.</p>



<p class="wp-block-paragraph">Two more things worth getting right at the start. Request the narrowest OAuth scopes that work, read-only on the modules you actually extract, because scopes are easy to widen later and awkward to explain in an audit. And store the refresh token in Secrets Manager rather than an environment variable, since it does not expire and is effectively a permanent key to your CRM.</p>



<h2 class="wp-block-heading">Which API to extract with</h2>



<p class="wp-block-paragraph">Three options, and the choice is mostly about volume.</p>



<h3 class="wp-block-heading">Bulk Read, for anything large</h3>



<p class="wp-block-paragraph">This is the right default for a lake. You POST a job description, Zoho runs the export server-side, and you poll for status or supply a callback URL. When it finishes you get a download URL for a ZIP of CSV. A single job handles up to 200,000 records, with paging beyond that, and it does not consume your standard API limits the way record-by-record calls do.</p>



<p class="wp-block-paragraph">The important property for our purposes: the pagination problem disappears. Zoho assembles the export on its side, so there is no page-by-page read for records to slip between.</p>



<pre class="wp-block-code"><code>curl "https://www.zohoapis.com/crm/bulk/v8/read" 
  -X POST 
  -H "Authorization: Zoho-oauthtoken $ACCESS_TOKEN" 
  -H "Content-Type: application/json" 
  -d @job.json</code></pre>



<pre class="wp-block-code"><code>{
  "query": {
    "module": { "api_name": "Deals" },
    "fields": ["id", "Deal_Name", "Amount", "Stage", "Modified_Time"],
    "criteria": {
      "group_operator": "and",
      "group": [
        {
          "field": { "api_name": "Modified_Time" },
          "comparator": "greater_than",
          "value": "{T1}"
        },
        {
          "field": { "api_name": "Modified_Time" },
          "comparator": "less_than",
          "value": "{T2}"
        }
      ]
    },
    "page": 1
  }
}</code></pre>



<p class="wp-block-paragraph">Two limits to design around. The download URL is only valid for about a day, so fetch and land the file promptly rather than queueing it for a later step. And downloads are rate limited per minute, so if you fan out across many modules at once you will start collecting 429s.</p>



<h3 class="wp-block-heading">COQL, for moderate volumes and real filtering</h3>



<p class="wp-block-paragraph">Zoho&#8217;s SQL-like query API. Up to 2,000 records per call and up to 100,000 records total per unique criteria through pagination. Past that, Zoho&#8217;s own documentation tells you to use Bulk Read, which is good advice to take rather than work around.</p>



<p class="wp-block-paragraph">If you do paginate COQL, paginate by key rather than by offset. Sort by <code>id</code> and carry the last ID you saw into the next call, so shifting records cannot cause a skip:</p>



<pre class="wp-block-code"><code>{
  "select_query": "select id, Deal_Name, Amount, Stage, Modified_Time from Deals where (Modified_Time &gt; '{T1}' and Modified_Time &lt; '{T2}') and id &gt; {last_id} order by id asc limit 2000"
}</code></pre>



<p class="wp-block-paragraph">Note the shape: a closed time window that freezes the set, plus a keyset cursor that walks it deterministically. Offset pagination gives you neither. Zoho&#8217;s own documentation uses this pattern in its examples, which is a fair hint about what they expect.</p>



<h3 class="wp-block-heading">Get Records, for small modules and metadata</h3>



<p class="wp-block-paragraph">The plain module endpoint returns up to 200 records a page and supports an <code>If-Modified-Since</code> header for incremental reads. Fine for lookup tables and small custom modules. Not what you want pointed at a Deals module with six figures of rows.</p>



<p class="wp-block-paragraph">Worth saying plainly: if all of this sounds like a fortnight of work you would rather not own, a managed connector from Fivetran or Airbyte solves the extraction half and you spend your time on the modelling instead. Price it against your own hours honestly, because the build-it-yourself option is habitually costed at zero.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Deletions, which Zoho actually handles well</h2>



<p class="wp-block-paragraph">An incremental extract can never see a deletion, because a record that no longer exists cannot appear in a query for modified records. Most CRMs make you infer this. Zoho gives you a direct endpoint, and it is one of the nicer parts of the API.</p>



<pre class="wp-block-code"><code># type: all | recycle | permanent
curl "https://www.zohoapis.com/crm/v8/Deals/deleted?type=all&amp;per_page=200&amp;page=1" 
  -X GET 
  -H "Authorization: Zoho-oauthtoken $ACCESS_TOKEN" 
  -H "If-Modified-Since: {T1}"</code></pre>



<p class="wp-block-paragraph">The retention windows are generous: records in the recycle bin are retrievable for up to 60 days from deletion, and permanently deleted records for up to 120 days. Compared with warehouses where the detection window is a couple of weeks and not guaranteed, that is a lot of slack. It also means there is no excuse for missing a deletion, since any sane sync interval sits comfortably inside it.</p>



<p class="wp-block-paragraph">Run the deleted-records call as part of every extract cycle, using the same closed window, and mark the matching rows as deleted in your curated layer rather than removing them from raw. Soft-delete preserves the audit trail and lets you answer &#8220;when did this disappear&#8221; later.</p>



<p class="wp-block-paragraph">Even with a good endpoint, run a periodic full ID reconciliation as a backstop, monthly is plenty. Pull just the ID column for the whole module, diff it against the lake, and flag anything you still believe in that Zoho does not. It catches the cases the deletion endpoint does not describe cleanly, such as records merged or moved between modules.</p>



<h2 class="wp-block-heading">Shaping CRM data for S3</h2>



<p class="wp-block-paragraph">Land raw first, exactly as returned, in its own prefix. Then convert to Parquet in a second step. Skipping the raw layer feels efficient right up to the first time you need to reprocess a month with corrected logic.</p>



<pre class="wp-block-code"><code># Partition by the window's upper bound, not by a business date.
# One bad run is then one partition to replace.
s3://acme-lake/raw/zoho_crm/deals/window_end=YYYY-MM-DDTHH/
s3://acme-lake/curated/zoho_crm/deals/window_end=YYYY-MM-DDTHH/</code></pre>



<p class="wp-block-paragraph">Zoho returns nested structures that do not map onto a flat table. Lookup fields come back as objects with an ID and a display name. Multi-select fields and tags come back as arrays. Subforms come back as arrays of objects. You have three choices per field and should make each one deliberately:</p>



<ul class="wp-block-list">
<li><strong>Flatten into columns.</strong> A lookup becomes <code>account_id</code> and <code>account_name</code>. Correct for anything you filter or join on.</li>
<li><strong>Split into a child table.</strong> Subforms and line items become their own Parquet dataset keyed by parent ID. Correct when the nested rows are things people count.</li>
<li><strong>Keep as a JSON string column.</strong> Fine for rarely-queried arrays, and Athena can parse it on demand. Not fine for anything in a regular report.</li>
</ul>



<p class="wp-block-paragraph">Pin your types explicitly during conversion rather than letting the writer infer them from a batch. Inference is the reason a column is a string in January&#8217;s files and a double in February&#8217;s, and Athena will happily read both and quietly fail to reconcile them.</p>



<p class="wp-block-paragraph">Register the result in the Glue Data Catalog and query with Athena. If you expect to apply updates and deletes in place rather than rebuilding partitions, Apache Iceberg earns its extra setup: row-level operations and schema evolution are exactly what a CRM feed generates.</p>



<p class="wp-block-paragraph">On the AWS side, the orchestration is unremarkable and should stay that way: EventBridge on a schedule, Lambda or a small container for the extract, Step Functions if you need to poll a Bulk Read job and branch on the result. Resist Glue jobs for the extraction itself unless you are already deep in Glue; a Python container you can run locally is easier to debug at seven in the morning.</p>



<h2 class="wp-block-heading">API credits</h2>



<p class="wp-block-paragraph">Zoho meters API usage as credits, allocated by edition and user count, over a rolling 24-hour window, and shared with every other integration on the org. There are also concurrency limits, so hammering the API in parallel fails differently from exceeding your daily allowance.</p>



<ul class="wp-block-list">
<li><strong>Bulk Read for volume.</strong> It sidesteps the standard limits and is the whole reason the API exists.</li>
<li><strong>Ask for fewer fields.</strong> Selecting only the columns you actually land reduces payload and processing on both sides.</li>
<li><strong>Mind the COQL limit value.</strong> Credits scale with the page size you request, so grabbing 2,000 rows costs more than grabbing 200. Larger pages are still usually the better trade, but know you are making it.</li>
<li><strong>Give the pipeline its own connected app and user.</strong> Then consumption is attributable when someone asks who exhausted the credits.</li>
<li><strong>Back off properly on 429.</strong> Exponential backoff with jitter, not a fixed sleep, and treat rate limiting as an expected condition rather than an error.</li>
</ul>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Two runs of the same window return different rows</h3>



<p class="wp-block-paragraph">Open-ended window, offset pagination, or both. Add the upper bound, switch to keyset pagination or Bulk Read, and the symptom disappears. This is the one to check before anything else, because it makes every other investigation unreliable.</p>



<h3 class="wp-block-heading">Invalid token, but the credentials are definitely right</h3>



<p class="wp-block-paragraph">Data centre mismatch. Confirm which region the org lives in and that your accounts domain and API domain both match it. This is far more common than an actually bad token.</p>



<h3 class="wp-block-heading">Row counts drift upward over time</h3>



<p class="wp-block-paragraph">Deletions are not being applied. Run the deleted-records endpoint for the last 60 days and see how much comes back, then run a full ID reconciliation to catch the rest.</p>



<h3 class="wp-block-heading">Bulk Read job finished but the file is gone</h3>



<p class="wp-block-paragraph">The download URL expires after roughly a day. Land the file as soon as the job reports complete, rather than deferring it to a downstream step that might not run until tomorrow.</p>



<h3 class="wp-block-heading">Athena fails with a schema mismatch across partitions</h3>



<p class="wp-block-paragraph">Type inference changed between runs, usually because a nullable field was all-null in one batch and populated in the next. Define the schema explicitly at write time and reprocess the affected partitions from raw.</p>



<h3 class="wp-block-heading">A new custom field never appears</h3>



<p class="wp-block-paragraph">You are selecting fields explicitly, which is correct, and nobody told you a field was added. Poll the module&#8217;s field metadata on a schedule and alert on changes, so schema drift is a notification rather than a discovery six weeks later.</p>



<h3 class="wp-block-heading">Frequent 429s</h3>



<p class="wp-block-paragraph">Either concurrency or the download rate limit, depending on which call is failing. Serialise the module extracts rather than fanning them all out at once, and add jittered backoff.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>An open-ended <code>Modified_Time</code> filter with no upper bound.</li>
<li>Offset pagination over a dataset that is being modified while you read it.</li>
<li>Advancing the watermark when the API call succeeds rather than when the data lands.</li>
<li>Keeping watermark state on an ephemeral runner.</li>
<li>Hardcoding the Zoho API domain and discovering data centres the hard way.</li>
<li>Never calling the deleted-records endpoint, so counts only ever grow.</li>
<li>Deferring the Bulk Read download until after the URL has expired.</li>
<li>Landing only Parquet with no raw layer, so reprocessing means re-extracting.</li>
<li>Letting the Parquet writer infer types per batch.</li>
<li>Flattening subforms into a wide table and double-counting parent rows.</li>
<li>Fanning out every module in parallel and collecting rate limits.</li>
<li>Sharing the pipeline&#8217;s connected app with other integrations, so nobody can attribute credit usage.</li>
<li>Storing the refresh token in an environment variable and forgetting it never expires.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Closed read windows with a deliberate lag, always.</li>
<li>Keyset pagination when you paginate at all, Bulk Read when you can avoid it.</li>
<li>Watermarks in durable storage, advanced only on confirmed landing.</li>
<li>Idempotent runs, so re-running a window is always safe.</li>
<li>Deleted-records endpoint every cycle, full ID reconciliation monthly.</li>
<li>Raw layer untouched, curated layer derived, partitions keyed by window end.</li>
<li>Explicit schemas on write, explicit field lists on read.</li>
<li>Its own connected app, narrow scopes, refresh token in Secrets Manager.</li>
<li>Jittered exponential backoff and serialised module extracts.</li>
<li>Alerting on schema drift and on row-count delta against Zoho.</li>
<li>Encryption and a retention policy on CRM data in S3 from day one, since it is personal data.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">What is the best way to export Zoho CRM data to S3?</h3>



<p class="wp-block-paragraph">Bulk Read for anything of size, on a schedule, with closed time windows. COQL for moderate volumes where you want real filtering. The plain records endpoint only for small modules. Land raw output first, convert to Parquet second.</p>



<h3 class="wp-block-heading">Why does my extract return different results each run?</h3>



<p class="wp-block-paragraph">Your query has no upper time bound, so the result set changes while you paginate through it. Bound both ends of the window and put the upper bound a few minutes in the past. That single change makes runs repeatable.</p>



<h3 class="wp-block-heading">How do I capture deleted records?</h3>



<p class="wp-block-paragraph">Call the module&#8217;s deleted-records endpoint each cycle. Recycle-bin deletions stay retrievable for 60 days and permanent deletions for 120, so a daily or hourly sync has plenty of margin. Add a monthly full ID reconciliation as a backstop.</p>



<h3 class="wp-block-heading">Will this exhaust my Zoho API credits?</h3>



<p class="wp-block-paragraph">Not if you use Bulk Read for the heavy lifting, since it does not draw on standard API limits the way per-record calls do. The credit pool is shared across the org, so give the pipeline its own connected app and monitor consumption rather than finding out when someone else&#8217;s integration breaks.</p>



<h3 class="wp-block-heading">How do I handle subforms and multi-select fields?</h3>



<p class="wp-block-paragraph">Decide per field. Flatten lookups into ID and name columns, split subforms and line items into their own child datasets keyed by parent ID, and keep genuinely peripheral arrays as JSON strings. Flattening a subform into the parent row is how you end up double-counting deals.</p>



<h3 class="wp-block-heading">Build it or buy a connector?</h3>



<p class="wp-block-paragraph">Buy it if Zoho CRM is one source among several and you have no unusual requirements; managed connectors handle pagination, deletes and schema drift, which is most of the work described here. Build it when you need control over the shape of the output, want to avoid per-row pricing, or already run the surrounding infrastructure.</p>



<h3 class="wp-block-heading">How fresh can the data realistically be?</h3>



<p class="wp-block-paragraph">Hourly is comfortable and suits nearly all reporting. Every fifteen minutes is achievable with COQL on modest modules. Anything closer to real time means webhooks or notification subscriptions rather than polling, which is a considerably larger commitment for a benefit most dashboards do not use.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">An extract that returns a different answer each time it runs is not a data quality problem, it is a read consistency problem, and no amount of downstream validation will fix it. Bound both ends of the window, put the upper bound in the past, paginate by key rather than offset, and advance the watermark only when the data is safely in S3.</p>



<p class="wp-block-paragraph">Get that right and the rest of the pipeline becomes ordinary engineering: land it, convert it, catalogue it, query it. Get it wrong and you will spend months chasing numbers that move every time you look at them.</p>



<h2 class="wp-block-heading">Want this built or reviewed?</h2>



<p class="wp-block-paragraph">Most CRM pipelines I get handed work fine on a quiet module and lose rows on the busy one, which is the hardest version to notice. Work I take on:</p>



<ul class="wp-block-list">
<li>Building a Zoho CRM to S3 pipeline end to end: OAuth, Bulk Read extraction, Parquet conversion, Glue catalog, Athena query layer.</li>
<li>Auditing an existing pipeline for read consistency and telling you whether it is silently dropping records.</li>
<li>Reworking incremental logic into closed windows, keyset pagination and durable watermarks.</li>
<li>Deletion handling and reconciliation jobs so row counts stay equal to the CRM.</li>
<li>Modelling nested CRM data into a schema that does not double-count.</li>
<li>Orchestration and alerting on AWS: EventBridge, Lambda or containers, Step Functions, and monitoring that catches drift early.</li>
</ul>



<p class="wp-block-paragraph">Send me your extract query and how you paginate, and I will tell you whether it can lose rows.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/zoho-crm-to-amazon-s3/">Run It Twice, Get Two Answers: Building an ETL Pipeline From Zoho CRM to Amazon S3</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
