<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Idempotency | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/idempotency/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/idempotency/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Thu, 12 Nov 2026 07:09:08 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Idempotency | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/idempotency/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>GPS Data Gets Ugly Fast: A Fleet GPS Data Pipeline That Stays Queryable</title>
		<link>https://john-nessime.com/blog/data-engineering/fleet-gps-data-pipeline/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 12 Nov 2026 07:09:08 +0000</pubDate>
				<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Databases]]></category>
		<category><![CDATA[Logistics Technology]]></category>
		<category><![CDATA[Continuous Aggregates]]></category>
		<category><![CDATA[Data Partitioning]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Fleet Telematics]]></category>
		<category><![CDATA[Geofencing]]></category>
		<category><![CDATA[GNSS]]></category>
		<category><![CDATA[Hypertables]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Late-Arriving Data]]></category>
		<category><![CDATA[Map Matching]]></category>
		<category><![CDATA[MobilityDB]]></category>
		<category><![CDATA[PostGIS]]></category>
		<category><![CDATA[PostgreSQL]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Time-Series Data]]></category>
		<category><![CDATA[TimescaleDB]]></category>
		<category><![CDATA[Track and Trace]]></category>
		<category><![CDATA[Trajectory Data]]></category>
		<category><![CDATA[Valhalla]]></category>
		<category><![CDATA[Vehicle Data]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=714</guid>

					<description><![CDATA[<p>Telematics devices lie in specific, documented ways: stale fixes reported as new, zero coordinates after reboot, and buffered records replayed days late. Here is how to build a fleet GPS data pipeline on PostgreSQL, PostGIS and TimescaleDB that survives all three and still answers trip and geofence queries fast.</p>
<p>The post <a href="https://john-nessime.com/blog/data-engineering/fleet-gps-data-pipeline/">GPS Data Gets Ugly Fast: A Fleet GPS Data Pipeline That Stays Queryable</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">It usually starts with a screenshot. Someone in operations sends the live map with a van sitting in the middle of a field, or off the coast entirely, and asks whether the tracker is broken. It isn&#8217;t. The tracker is doing exactly what its firmware says it should. Nobody decided what a bad point looks like before the data was written down, so now there are eleven million rows and no way to tell the real ones from the noise.</p>



<p class="wp-block-paragraph">By the time it reaches you it looks like a query problem. The trip report takes ninety seconds. The mileage total disagrees with the odometer. A geofence alert fires for a vehicle that never moved. Those are symptoms. The cause sits upstream, in the shape of the table and in the decision, made by default, to store every record the device sent as though it were true.</p>



<p class="wp-block-paragraph">This post covers how I&#8217;d build a fleet GPS data pipeline that stays queryable as it grows: what the dirty data looks like at the source, how to split storage so cleaning stays reversible, how to keep the raw table from drowning in its own volume, and why nearly every question your users ask is a question about trips rather than points.</p>



<h2 class="wp-block-heading">The four ways GPS data goes ugly before you query it</h2>



<p class="wp-block-paragraph">Organise your defences by failure family, not by field. Each of these produces rows that are individually well-formed and collectively wrong.</p>



<h3 class="wp-block-heading">The device sends its last known position and calls it new</h3>



<p class="wp-block-paragraph">This is the one that bites hardest, because it is invisible. When a unit loses its fix, plenty of firmware does not send a null. Teltonika&#8217;s FMB documentation is explicit: when a record has no valid coordinates because there was no fix at acquisition time, the longitude, latitude and altitude fields carry the <em>last valid fix</em>, while angle, satellites and speed are set to zero. The row looks normal. It has a timestamp, it has coordinates, it parses.</p>



<p class="wp-block-paragraph">That is worse than a gap. A van parked underground for six hours emits hundreds of records claiming the same coordinates, and your &#8220;last seen&#8221; logic reports it sitting at the entrance the whole time. Then it drives out, gets a fix elsewhere, and the trip builder draws a straight line through everything in between.</p>



<p class="wp-block-paragraph">The signal you want is the satellite count and the dilution of precision, not the coordinates. Store both. Teltonika exposes a GPS Accuracy setting expressed as a minimum HDOP, PDOP or VDOP before the device refreshes its GPS data, plus a minimum satellite count, so the hardware can be told to be pickier before it transmits. Configure that, and still keep the fields, because you will want to re-derive the threshold later without re-collecting the fleet.</p>



<h3 class="wp-block-heading">Null Island, and the reboot that gets you there</h3>



<p class="wp-block-paragraph">Zero latitude and zero longitude is a real place off the coast of Ghana, and it is where a surprising share of any fleet appears to live. The same Teltonika documentation notes that after a reboot the device sends zero coordinates. A naive pipeline stores that as a point, and every map, bounding box and distance calculation now carries an outlier several thousand kilometres from the fleet. Trivially filterable; it survives into production because nobody looks until the map is built, and by then the filter has to be retrofitted into six queries.</p>



<h3 class="wp-block-heading">Records that arrive days late, in the wrong order</h3>



<p class="wp-block-paragraph">Telematics units buffer. A vehicle drives through a tunnel or a rural dead zone, keeps recording, and dumps the backlog on reconnect. Devices in this class also let you choose whether records are saved after a position fix, after a time sync, or always, which means you can receive records whose device clock was never synchronised at all.</p>



<p class="wp-block-paragraph">So the timestamp on the record and the moment you received it are two different facts, and you need both. Storing one is the most common design mistake I see in fleet ingest. Keep only the device timestamp and you cannot answer &#8220;what did the dashboard show at 4pm yesterday&#8221;, which is what every dispute becomes. Keep only the receive time and your trips are wrong.</p>



<h3 class="wp-block-heading">Parked-vehicle drift</h3>



<p class="wp-block-paragraph">A stationary receiver with a marginal sky view reports a slow random walk of a few metres at a time, all night, in a depot. Sum the distance between consecutive points and you have invented a three-kilometre trip. Multiply by a fleet and the mileage report is one nobody trusts.</p>



<p class="wp-block-paragraph">Drift is not caught by the same rule as teleporting. A jump filter catches physically impossible movement. Drift is physically possible, it just isn&#8217;t real. You need ignition state, reported speed or an accelerometer movement flag to tell &#8220;not moving&#8221; from &#8220;moving very slowly in traffic&#8221;.</p>



<h2 class="wp-block-heading">Split the fleet GPS data pipeline into three tables</h2>



<p class="wp-block-paragraph">The instinct is one positions table with cleaning applied on write. Resist it. Once you discard a record you cannot revisit the decision, and you will change your mind about the thresholds at least twice.</p>



<ol class="wp-block-list">
<li><strong>Raw.</strong> Everything the device sent, as sent, plus a receive timestamp. Append-only, never cleaned in place.</li>



<li><strong>Clean.</strong> Points that passed your quality rules, with a geometry column. Derived from raw and fully rebuildable from it.</li>



<li><strong>Trips.</strong> One row per journey, with the whole path as a single geometry. This is what nearly every user-facing query reads.</li>
</ol>



<p class="wp-block-paragraph">Raw is what makes the rest safe. When a mileage figure is disputed eight months from now, you can re-run the cleaning under the old rules and the new ones and show the difference. Two timestamps, quality fields as first-class columns, and anything device-specific in JSONB so new hardware doesn&#8217;t force a migration:</p>



<pre class="wp-block-code"><code>CREATE TABLE gps_raw (
    device_id   text        NOT NULL,
    recorded_at timestamptz NOT NULL,   -- the device's clock
    received_at timestamptz NOT NULL DEFAULT now(),
    lon         double precision,
    lat         double precision,
    speed_kph   double precision,
    heading     double precision,
    satellites  smallint,
    hdop        double precision,
    ignition    boolean,
    payload     jsonb       NOT NULL DEFAULT '{}'::jsonb
);</code></pre>



<p class="wp-block-paragraph">No primary key on device and time, deliberately. Devices resend. A unique constraint here turns a harmless duplicate into an ingest failure at three in the morning. Deduplicate on the way into the clean table, where a conflict is a no-op rather than an incident.</p>



<h2 class="wp-block-heading">Making the raw table survive its own growth</h2>



<p class="wp-block-paragraph">A few hundred vehicles reporting every few seconds produces a table that ordinary PostgreSQL handles for a while and then doesn&#8217;t, usually right after someone adds a date filter to a dashboard. Time partitioning is the fix, either through native declarative partitioning or the TimescaleDB extension.</p>



<pre class="wp-block-code"><code>SELECT create_hypertable('gps_raw', 'recorded_at',
                         chunk_time_interval =&gt; INTERVAL '1 day');</code></pre>



<p class="wp-block-paragraph">Pick the chunk interval from your query pattern, not your ingest rate. If dashboards ask for &#8220;today&#8221; and &#8220;this week&#8221;, daily chunks let the planner exclude almost everything. If reporting is monthly, daily chunks leave a lot of small chunks to plan across.</p>



<p class="wp-block-paragraph">Compression is where the storage argument is won, and the segment key is what people get wrong. Segmenting by vehicle groups each vehicle&#8217;s rows together inside a compressed batch, which is what keeps &#8220;one vehicle, one day&#8221; fast afterwards:</p>



<pre class="wp-block-code"><code>ALTER TABLE gps_raw SET (
    timescaledb.compress,
    timescaledb.compress_orderby   = 'recorded_at',
    timescaledb.compress_segmentby = 'device_id'
);</code></pre>



<h3 class="wp-block-heading">The licensing detail that catches you at runtime</h3>



<p class="wp-block-paragraph">TimescaleDB ships in two editions. The Apache 2.0 build gives you hypertables. Compression, continuous aggregates and retention policies live in the Community edition under the Timescale License, which is source-available rather than OSI-open-source. The vendor states the practical boundary plainly: as long as you are not offering TimescaleDB itself as a hosted database service, you can run the Community features free and self-hosted in production.</p>



<p class="wp-block-paragraph">The trap is that this fails at runtime, not at install. If your distribution or provider ships the Apache-licensed module, <code>CREATE EXTENSION</code> succeeds, the hypertable works, and then the <code>ALTER TABLE</code> above returns an error saying the functionality is not supported under the current licence. Zabbix users have filed bugs about exactly this. Check the edition before you design around compression, especially on managed PostgreSQL where you don&#8217;t choose the build.</p>



<p class="wp-block-paragraph">If that blocks you, native partitioning plus a scheduled job rolling old partitions into archive storage is a fine answer. More code, less magic, no licence question. The related fork is whether to self-host at all: a telemetry database is steady, disk-heavy and predictable, which is where a large VPS from Hetzner, Contabo or InterServer compares well against a managed hourly bill, and also where you inherit backups, upgrades and the extension build. My <a href="https://john-nessime.com/tools/vps-vs-managed-aws/">VPS versus managed AWS calculator</a> prices your own admin hours against the managed bill and returns a break-even rather than an opinion.</p>



<h3 class="wp-block-heading">Rollups, and the late records that quietly miss them</h3>



<p class="wp-block-paragraph">Continuous aggregates pre-compute the hourly numbers so dashboards stop scanning points:</p>



<pre class="wp-block-code"><code>CREATE MATERIALIZED VIEW vehicle_hourly
WITH (timescaledb.continuous) AS
SELECT device_id,
       time_bucket(INTERVAL '1 hour', recorded_at) AS bucket,
       count(*)        AS fixes,
       max(speed_kph)  AS max_speed_kph
FROM gps_clean
GROUP BY device_id, bucket;</code></pre>



<p class="wp-block-paragraph">Now connect that to buffering. The refresh policy has a window, and anything older than its start is never revisited:</p>



<pre class="wp-block-code"><code>SELECT add_continuous_aggregate_policy('vehicle_hourly',
    start_offset      =&gt; INTERVAL '7 days',
    end_offset        =&gt; INTERVAL '1 hour',
    schedule_interval =&gt; INTERVAL '1 hour');</code></pre>



<p class="wp-block-paragraph">A device out of coverage for ten days reconnects, replays its backlog, the rows land in the clean table, and the hourly rollup for those ten days stays exactly as wrong as it was. Raw is right, the dashboard is wrong, nothing errors. Set the start offset wider than your worst realistic backlog, and graph the gap between <code>recorded_at</code> and <code>received_at</code> per device model so you know what that worst case actually is. If Grafana already points at this database, it costs you one panel.</p>



<h2 class="wp-block-heading">Cleaning: two filters that do most of the work</h2>



<p class="wp-block-paragraph">Promotion from raw to clean is where the quality rules live. Make it an idempotent batch over a time window so any period can be re-run without duplicating anything.</p>



<pre class="wp-block-code"><code>INSERT INTO gps_clean (device_id, recorded_at, geom, speed_kph)
SELECT device_id,
       recorded_at,
       ST_SetSRID(ST_MakePoint(lon, lat), 4326),
       speed_kph
FROM gps_raw
WHERE recorded_at &gt;= $1
  AND recorded_at &lt;  $2
  AND lat IS NOT NULL
  AND lon IS NOT NULL
  AND NOT (lat = 0 AND lon = 0)
  AND satellites &gt;= 4
  AND hdop &lt;= 5
ON CONFLICT (device_id, recorded_at) DO NOTHING;</code></pre>



<p class="wp-block-paragraph">Treat those numbers as starting points to tune against your own hardware, not constants. Four satellites is the minimum for a three-dimensional fix; the dilution threshold depends on your receivers and your operating environment. Write the chosen values somewhere a human will find them, because in a year someone will ask why a specific point was dropped.</p>



<p class="wp-block-paragraph">The second filter is the jump test, which needs the previous row. Cast to <code>geography</code> so distance comes back in metres rather than degrees:</p>



<pre class="wp-block-code"><code>WITH stepped AS (
  SELECT device_id, recorded_at, geom,
         LAG(geom)        OVER w AS prev_geom,
         LAG(recorded_at) OVER w AS prev_at
  FROM gps_clean
  WHERE device_id = $1
    AND recorded_at &gt;= $2
  WINDOW w AS (PARTITION BY device_id ORDER BY recorded_at)
)
SELECT device_id,
       recorded_at,
       ST_Distance(geom::geography, prev_geom::geography) AS metres,
       EXTRACT(epoch FROM recorded_at - prev_at)          AS seconds
FROM stepped
WHERE prev_geom IS NOT NULL
  AND ST_Distance(geom::geography, prev_geom::geography)
      / NULLIF(EXTRACT(epoch FROM recorded_at - prev_at), 0) &gt; 60;</code></pre>



<p class="wp-block-paragraph">Sixty metres per second is a little over 200 km/h. No delivery van does that, so anything above it is a bad fix, a clock problem, or two vehicles sharing a device identifier. Flag them rather than deleting them; the pattern tells you which of the three you have.</p>



<h2 class="wp-block-heading">Trips are the abstraction that makes queries fast</h2>



<p class="wp-block-paragraph">Users don&#8217;t ask point questions. They ask trip questions: how far did this vehicle go, when did it enter the site, how long was it stopped, which vehicles passed through this area. Answering those from a point table means aggregating thousands of rows per answer, every time.</p>



<p class="wp-block-paragraph">PostGIS has a native representation for this. A trajectory is a LINESTRING whose M dimension carries time, with M increasing from each vertex to the next, and <code>ST_IsValidTrajectory</code> checks exactly that. Once your paths are valid trajectories the spatio-temporal functions come free: <code>ST_ClosestPointOfApproach</code> returns the measure at which two moving objects were nearest, <code>ST_DistanceCPA</code> returns how near, and <code>ST_CPAWithin</code> gives the boolean. That turns &#8220;were these two vehicles ever within fifty metres of each other&#8221; into one function call instead of a self-join across a point table.</p>



<pre class="wp-block-code"><code>CREATE TABLE trips (
    trip_id     bigserial PRIMARY KEY,
    device_id   text        NOT NULL,
    started_at  timestamptz NOT NULL,
    ended_at    timestamptz NOT NULL,
    path        geometry(LineStringM, 4326) NOT NULL,
    distance_m  double precision NOT NULL,
    point_count integer     NOT NULL
);

CREATE INDEX trips_path_gix ON trips USING GIST (path);
CREATE INDEX trips_device_time_idx ON trips (device_id, started_at DESC);</code></pre>



<p class="wp-block-paragraph">Building the path is an aggregate over the clean points, with epoch seconds as the M value:</p>



<pre class="wp-block-code"><code>SELECT ST_MakeLine(
         ST_MakePointM(ST_X(geom), ST_Y(geom),
                       EXTRACT(epoch FROM recorded_at))
         ORDER BY recorded_at
       ) AS path
FROM gps_clean
WHERE device_id = $1
  AND recorded_at &gt;= $2
  AND recorded_at &lt;  $3;</code></pre>



<p class="wp-block-paragraph">Segmenting points into trips is a business decision dressed as a technical one. Ignition on to ignition off is cleanest where the wiring supports it; otherwise speed below a floor for longer than a gap threshold ends the trip. Whichever you pick, store the rule as a column on the trip, because someone will change it and you&#8217;ll need to know which rows were built under which version.</p>



<p class="wp-block-paragraph">For rendering, keep a simplified two-dimensional copy. <code>ST_SimplifyPreserveTopology</code> runs Douglas-Peucker without collapsing the line into nonsense, and a route drawn at city zoom doesn&#8217;t need a vertex every two seconds. Simplify for display only, never for the geometry you measure distance from.</p>



<h2 class="wp-block-heading">Map matching: when to snap, and when to skip it</h2>



<p class="wp-block-paragraph">Snapping traces to the road network fixes a cosmetic problem, paths wandering across the pavement, and a real one, distance summed from raw points overstating the journey. Valhalla&#8217;s Meili implements the well-known hidden Markov approach to matching noisy, sparse traces. Its API exposes two actions: <code>trace_route</code> returns the shape snapped to the road network with directions, and <code>trace_attributes</code> returns per-segment attribution along the matched path. Requests should be POSTed, because an encoded trace gets large. OSRM&#8217;s match service solves the same problem with a similar model.</p>



<pre class="wp-block-code"><code>curl -s -X POST https://valhalla.internal.example/trace_attributes 
  -H 'Content-Type: application/json' 
  -d '{
        "shape": [
          {"lat": 52.0907, "lon": 5.1214},
          {"lat": 52.0912, "lon": 5.1229}
        ],
        "costing": "auto",
        "shape_match": "map_snap"
      }'</code></pre>



<p class="wp-block-paragraph">Two things decide whether it&#8217;s worth running. First, trace density: Valhalla&#8217;s own guidance is to keep points roughly between one per second and one per ten seconds, and warns the match degrades as that spacing widens. If your devices report every two minutes to save data, matching will confidently snap you to plausible roads you never used, which is worse than an obviously wobbly line because it looks correct.</p>



<p class="wp-block-paragraph">Second, what the number is for. Billing, driver disputes and compliance want the matched path. An internal ops map does not. Matching is either a per-trip network call or a self-hosted routing service with a tile build to maintain, and it&#8217;s the most operationally expensive piece in this design. If no money rides on the number, skip it and say so.</p>



<p class="wp-block-paragraph">Self-hosting, Valhalla and OSRM both run fine in containers against an OpenStreetMap extract, on a separate box from the one serving queries because the tile build is memory-hungry. Worth pasting that stack into my <a href="https://john-nessime.com/tools/docker-compose-production-checker/">Docker Compose production checker</a> first, since routing containers are exactly the kind of thing that quietly exposes a port or writes tiles to a volume that dies on restart.</p>



<h2 class="wp-block-heading">Geofences that actually use the index</h2>



<p class="wp-block-paragraph">Geofence queries are where a working platform turns slow, and the cause is nearly always a function that can&#8217;t use the spatial index. <code>ST_Intersects</code>, <code>ST_Within</code> and <code>ST_DWithin</code> are index-assisted. <code>ST_Distance</code> is not. Writing <code>WHERE ST_Distance(a, b) &lt; 500</code> gets the right answer by computing distance for every candidate row.</p>



<pre class="wp-block-code"><code>-- Polygon fences: which trips crossed a depot this week
SELECT g.name, t.trip_id, t.device_id
FROM trips t
JOIN geofences g ON ST_Intersects(t.path, g.area)
WHERE t.started_at &gt;= now() - INTERVAL '7 days';

-- Radius fences: within 500 m of a point, using the index
SELECT t.trip_id
FROM trips t
WHERE ST_DWithin(t.path::geography, $1::geography, 500);</code></pre>



<p class="wp-block-paragraph">Running fences against trips rather than points is the other half of the win. A hundred fences against ten million points is a hundred million comparisons. A hundred fences against fifty thousand trip geometries, each with a bounding box the index rejects in one step, is a different query entirely.</p>



<p class="wp-block-paragraph">The catch is latency. Trips only exist once they end, so live entry and exit alerts still run against incoming points. Keep that path separate and deliberately small: active fences only, evaluated as records arrive. Don&#8217;t make one query serve both the live alert and the monthly report.</p>



<h2 class="wp-block-heading">Troubleshooting a fleet data platform that has gone wrong</h2>



<p class="wp-block-paragraph"><strong>Mileage disagrees with the odometer.</strong> Compare summed point-to-point distance against the matched-path distance for the same trip. Consistently higher means drift or noise, so tighten the quality filter and check whether stationary periods are included. Lower means missing points, with gaps bridged by straight lines.</p>



<p class="wp-block-paragraph"><strong>A dashboard number is wrong but the raw data is right.</strong> Suspect the rollup window first. Query the raw table and the continuous aggregate for the same bucket and compare. A difference means late data outside the refresh offset.</p>



<p class="wp-block-paragraph"><strong>A vehicle sits still for hours, then teleports.</strong> Check satellites and HDOP across the stationary block. Zero satellites throughout means repeated stale fixes, not a parked vehicle.</p>



<p class="wp-block-paragraph"><strong>Queries slowed suddenly with no traffic change.</strong> Check whether the planner is excluding chunks. A predicate wrapped in a function on the time column, or a parameter type that doesn&#8217;t match, prevents partition pruning and silently turns a one-day scan into a full-table scan. Run <code>EXPLAIN</code> and count the chunks in the plan.</p>



<p class="wp-block-paragraph"><strong>Two vehicles occupy the same position.</strong> Almost always a device moved between vehicles without the assignment history being updated. Model the device-to-vehicle link as a table with validity intervals, not a column on the vehicle.</p>



<p class="wp-block-paragraph">One failure worth rehearsing before it happens is the ingest endpoint going down. Devices buffer, so a short outage survives and a long one becomes a thundering replay on recovery. If you&#8217;ve never asked what breaks when the database node dies mid-shift, my <a href="https://john-nessime.com/tools/cloud-architecture-assessment/">cloud architecture assessment</a> scores where the single points of failure sit.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Cleaning on ingest and keeping nothing else. The rules will change and you&#8217;ll have no way to apply them retroactively.</li>



<li>Storing latitude and longitude as plain numeric columns. Fine until the first spatial query, then you&#8217;re building indexes on a table you can&#8217;t afford to lock.</li>



<li>Treating the device timestamp as authoritative. Unsynchronised clocks and buffered replays make it a claim, not a fact.</li>



<li>Using <code>ST_Distance</code> in a WHERE clause instead of <code>ST_DWithin</code>. Same answer, no index.</li>



<li>Designing around compression or continuous aggregates without checking which TimescaleDB edition is installed.</li>



<li>Map matching a sparse trace and trusting the output. Confident wrong answers are harder to catch than obvious ones.</li>



<li>Putting live alerting and historical reporting through the same queries.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Keep raw append-only and every downstream table rebuildable from it. Test that rebuild before you need it.</li>



<li>Store both timestamps on every record, and alert on the lag between them per device model.</li>



<li>Keep the quality fields after filtering on them. Satellites and HDOP are how you re-derive a threshold later.</li>



<li>Match the chunk interval to how people query, and revisit it when the reporting cadence changes.</li>



<li>Answer user questions from trips. Reach for points only when you genuinely need sub-trip resolution.</li>



<li>Record the segmentation rule and quality thresholds as data, not as code comments.</li>



<li>Retire raw data on a written retention schedule rather than when the disk fills.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need TimescaleDB, or is plain PostgreSQL enough?</h3>



<p class="wp-block-paragraph">Plain PostgreSQL with native partitioning and PostGIS carries a small fleet a long way, with no licensing question attached. TimescaleDB earns its place when you want compression and continuous aggregates without maintaining that machinery yourself. It saves you code, not capability.</p>



<h3 class="wp-block-heading">Should I use PostGIS geometry or geography for GPS points?</h3>



<p class="wp-block-paragraph">Store as <code>geometry</code> in SRID 4326 and cast to <code>geography</code> for distance and radius work. Geometry has broader function coverage and cheaper operations; geography gives metres on a sphere instead of degrees. Casting at the point of use is the pragmatic middle.</p>



<h3 class="wp-block-heading">How long should I keep raw GPS points?</h3>



<p class="wp-block-paragraph">Long enough to cover the dispute window your business actually faces, which is driven by employment law, insurance or customer contracts rather than by engineering. Decide it with whoever owns that risk, write it down, enforce it with a retention policy. Trips are far smaller and usually worth keeping much longer.</p>



<h3 class="wp-block-heading">What about ClickHouse or InfluxDB instead?</h3>



<p class="wp-block-paragraph">Both are genuinely strong at high-volume ingest and time-range scans, and for a purely time-series workload I&#8217;d take either seriously. The reason I reach for PostgreSQL here is that fleet data isn&#8217;t purely time-series: it joins constantly to vehicles, drivers, contracts and sites, and it needs spatial predicates in the same query as the time filter. One database beats two and a reconciliation job. If your spatial needs are trivial and your volume is enormous, that flips.</p>



<h3 class="wp-block-heading">Is MobilityDB worth adding?</h3>



<p class="wp-block-paragraph">MobilityDB extends PostgreSQL and PostGIS with proper moving-object types, <code>tgeompoint</code> and <code>tgeogpoint</code>, that interpolate between observations so you can query a position at any instant. It&#8217;s an OSGeo project and the modelling is better than hand-rolled trajectories. The cost is another extension to install and upgrade, including <code>shared_preload_libraries</code> and locking configuration on the server. I&#8217;d add it when trajectory analysis is the product, not when it&#8217;s a reporting feature.</p>



<h3 class="wp-block-heading">How do I stop parked vehicles generating fake mileage?</h3>



<p class="wp-block-paragraph">Don&#8217;t rely on distance alone, because drift is physically plausible. Use ignition state or a movement flag to mark stationary periods, exclude those points from distance summation, and keep them in raw so the stop duration stays auditable.</p>



<h3 class="wp-block-heading">Where does a fleet GPS data pipeline break first at scale?</h3>



<p class="wp-block-paragraph">The dashboard queries, and nearly always for the same reason: something in the predicate stopped the planner pruning partitions. Before adding hardware, run <code>EXPLAIN</code> and confirm how many chunks the query touches. Adding vehicles rarely breaks a partitioned table. Bad predicates always do.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A fleet GPS data pipeline stays queryable because of two decisions made early, not because of the database you picked. Keep the raw record untouched so every cleaning rule stays reversible, and answer questions from trips rather than points so the expensive aggregation happens once instead of on every page load.</p>



<p class="wp-block-paragraph">Everything else is tuning. Chunk intervals, compression settings and index choices all matter, and all of them are recoverable. Throwing away the record of what the device actually sent is the only mistake here you can&#8217;t undo later.</p>



<h2 class="wp-block-heading">Where I can help</h2>



<p class="wp-block-paragraph">Most fleet data work arrives as a number somebody disputes, and the fix lives three layers upstream of the report. Things I&#8217;d take on here:</p>



<ul class="wp-block-list">
<li>Reviewing an existing positions schema and working out what it takes to make it partition-friendly without a full rewrite.</li>



<li>Designing the raw, clean and trip split, including the idempotent promotion job so any window can be safely re-run.</li>



<li>Tracking down why a rollup and the raw data disagree, which is usually late records falling outside a refresh window.</li>



<li>Building quality filters against your actual hardware, so thresholds fit your receivers rather than someone else&#8217;s blog post.</li>



<li>Getting geofence and trip queries onto the spatial index, with the EXPLAIN output to show what changed.</li>



<li>Standing up self-hosted map matching, and deciding honestly if you need it at all.</li>
</ul>



<p class="wp-block-paragraph">If any of that is live for you, send the concrete thing: a table definition, the <code>EXPLAIN</code> output from the slow query, or a day of records from one device. That&#8217;s usually enough to say something useful straight away.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://john-nessime.com/projects/data-pipeline-review/">Data pipeline rerun review</a></div>



<div class="wp-block-button is-style-outline"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/data-engineering/fleet-gps-data-pipeline/">GPS Data Gets Ugly Fast: A Fleet GPS Data Pipeline That Stays Queryable</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Your CRM Is Not a Database: Where GoHighLevel Should Stop</title>
		<link>https://john-nessime.com/blog/data-engineering/gohighlevel-as-a-database/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 26 Sep 2026 06:00:23 +0000</pubDate>
				<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Enterprise Integration]]></category>
		<category><![CDATA[SaaS Engineering]]></category>
		<category><![CDATA[Architecture Debt]]></category>
		<category><![CDATA[CRM Audit]]></category>
		<category><![CDATA[Custom Objects]]></category>
		<category><![CDATA[Data Modeling]]></category>
		<category><![CDATA[GoHighLevel]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Marketing Automation]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Reverse ETL]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Snapshots]]></category>
		<category><![CDATA[Sub-Account Management]]></category>
		<category><![CDATA[System of Record]]></category>
		<category><![CDATA[Vendor Lock-In]]></category>
		<category><![CDATA[Webhooks]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=649</guid>

					<description><![CDATA[<p>A CRM never pushes back when you add one more field, which is exactly why the schema drifts until someone asks a question that needs a join. Four boundaries for deciding what belongs in GoHighLevel, what Custom Objects actually change, and the CRM-in-front, system-of-record-behind pattern I default to.</p>
<p>The post <a href="https://john-nessime.com/blog/data-engineering/gohighlevel-as-a-database/">Your CRM Is Not a Database: Where GoHighLevel Should Stop</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">&#8220;Can you pull every job we did last quarter, by technician, with the margin on each one?&#8221;</p>



<p class="wp-block-paragraph">The answer was no. Not &#8220;no, that&#8217;s expensive.&#8221; Just no. The jobs weren&#8217;t records. Each one was a couple of tags on a contact, three custom fields added at different times by different people, an opportunity someone had dragged into Won, and a note cut off mid-sentence. There was no job. There was a person that a job had happened to.</p>



<p class="wp-block-paragraph">Nobody made a bad decision to get there. &#8220;Can we just add a field for that?&#8221; is a five-second request and the answer is always yes, because adding a field is easy. That&#8217;s the trap. The CRM never pushes back, so the schema drifts for eighteen months and nobody notices until someone asks a question that needs a join.</p>



<p class="wp-block-paragraph">This post is about where that line sits: not whether to use the platform, but where <strong>GoHighLevel as a database</strong> stops being a shortcut and starts being architecture debt. I&#8217;ll cover the failure that stays invisible, the four boundaries that decide it, what Custom Objects actually change, and the arrangement I default to when the CRM isn&#8217;t the right home.</p>



<h2 class="wp-block-heading">The failure mode nobody sees coming</h2>



<p class="wp-block-paragraph">Databases push back. Declare a column as a date and it refuses to hold &#8220;next Tuesday.&#8221; Declare a foreign key and it refuses to orphan the row. Half the value of a schema isn&#8217;t storage, it&#8217;s the set of things it won&#8217;t let you do.</p>



<p class="wp-block-paragraph">A custom field holds whatever you type. That&#8217;s a feature when a rep jots down a competitor&#8217;s name. It&#8217;s a slow disaster when the field is called <code>contract_value</code> and it contains &#8220;1,250.00&#8221;, &#8220;$1250&#8221;, &#8220;1250 (pending)&#8221; and &#8220;ask Dave&#8221; across four hundred records.</p>



<p class="wp-block-paragraph">Here&#8217;s why it stays invisible: nothing breaks. Workflows keep firing. Dashboards keep rendering, because they count records rather than summing values. The failure surfaces the day someone needs an aggregate, and by then you have four hundred rows of ambiguity and no way to tell which reading was right. You can&#8217;t retroactively decide whether &#8220;1250 (pending)&#8221; was invoiced.</p>



<p class="wp-block-paragraph">History is the other half. A CRM field is a current value. Overwrite it and the old one is gone. Anything touching money, entitlements or consent eventually needs to prove what the state was and when, and the platform discarded that silently, months earlier.</p>



<h2 class="wp-block-heading">What the platform is genuinely good at</h2>



<p class="wp-block-paragraph">An argument against a tool isn&#8217;t credible unless it starts with the tool&#8217;s real case. GoHighLevel is very good at a specific set of things, most of them expensive to build yourself.</p>



<ul class="wp-block-list">
<li><strong>Conversation state across channels.</strong> One thread per person spanning SMS, email and calls, with delivery status attached. Rebuilding that is months of work and a compliance surface you don&#8217;t want.</li>

<li><strong>Sequenced outreach with human interrupts.</strong> Wait steps, conditions, and a reply pulling someone out of a sequence. Easy to describe, tedious to get right.</li>

<li><strong>Tenant isolation by default.</strong> Each sub-account is its own location, and nothing crosses unless you deliberately copy it.</li>

<li><strong>Non-engineers can change it.</strong> A workflow a client can edit on a Tuesday afternoon beats an elegant pipeline only you can touch.</li>
</ul>



<p class="wp-block-paragraph">Every one of those is about people and communication. That&#8217;s the pattern: when the thing you&#8217;re storing is a person, or something that happened to a person, the CRM is the right home. When it has a life independent of any person, you&#8217;re outside the design.</p>



<h2 class="wp-block-heading">Four boundaries that decide it for you</h2>



<p class="wp-block-paragraph">&#8220;Is this a database?&#8221; isn&#8217;t a useful question. These four are decidable, and any one of them crossing is usually enough.</p>



<h3 class="wp-block-heading">1. The question you need to ask requires a join</h3>



<p class="wp-block-paragraph">Not &#8220;show me contacts tagged X.&#8221; I mean revenue by technician by quarter, or every account with more than three open items where the last payment was over sixty days ago: grouping, filtering across related records, and arithmetic on typed values, all at once. Smart lists answer a narrow, predefined set of questions well, but they aren&#8217;t a query engine, and the gap between the two is where most people discover they&#8217;ve built a database by accident.</p>



<h3 class="wp-block-heading">2. More than one system writes the same field</h3>



<p class="wp-block-paragraph">This is the one that quietly costs the most. A stock level or invoice status lives in a CRM field, and both an automation and an external sync can write it. There&#8217;s no transaction and no compare-and-set. Last write wins, and &#8220;last&#8221; is whichever request happened to land second. You see it as an occasional wrong value you can&#8217;t reproduce, which is the worst kind of bug to be handed.</p>



<p class="wp-block-paragraph">The fix isn&#8217;t better retry logic. The field has the wrong owner. Exactly one system should write any given piece of state, and if that system isn&#8217;t the CRM, the CRM holds a copy for display rather than the value itself.</p>



<h3 class="wp-block-heading">3. Someone will need the history</h3>



<p class="wp-block-paragraph">Anything where you may later have to prove a prior state needs append-only storage. The workaround people reach for is a note per change. It doesn&#8217;t hold: notes aren&#8217;t structured, aren&#8217;t queryable, and the export path treats them as an afterthought, which leads straight to the next boundary.</p>



<h3 class="wp-block-heading">4. The exit shape doesn&#8217;t match what you put in</h3>



<p class="wp-block-paragraph">People check this last and should check it first. Contact export lands as CSV, and only the most recent note travels with it, truncated to 255 characters. A year of notes exports as one, clipped. Custom object data isn&#8217;t part of a direct export at all: the documented route is a snapshot from one sub-account to another, which is a copy inside the platform, not a file you keep. Export permissions are role-gated, so a client who believes they own their data may not be able to pull it themselves.</p>



<p class="wp-block-paragraph">None of that makes it a bad choice. It makes it a bad <em>sole</em> home for anything you&#8217;d be in trouble without.</p>



<h2 class="wp-block-heading">What Custom Objects change, and what they don&#8217;t</h2>



<p class="wp-block-paragraph">Custom Objects are a real improvement and worth using. Properties, policies, vehicles, courses: things that aren&#8217;t people, modelled as first-class records with their own fields and associations back to contacts, usable in workflows. For years the only option was squashing everything into a contact, and that was worse. They don&#8217;t turn the CRM into a database, though, and the reasons are specific rather than philosophical.</p>



<ul class="wp-block-list">
<li><strong>Ten objects per sub-account, documented as a hard cap.</strong> That&#8217;s your entire entity budget. Fifteen nouns worth modelling means five go back into custom fields.</li>

<li><strong>Uniqueness is limited and one-way.</strong> Up to ten unique fields per object, and only single line text, multi line text, number and phone types can be unique. It&#8217;s enforced across the sub-account and every entry point including the API, which is genuinely useful. But downgrade a unique field to non-unique and you cannot make it unique again.</li>

<li><strong>Write access is admin-only.</strong> Creating, updating and deleting object records is restricted to location admins; regular users are read-only. Check this before assuming the front desk will maintain records.</li>

<li><strong>Records count against storage quota.</strong> Object records and attached files consume the same account storage as everything else, so high-volume machine-generated data is the wrong shape.</li>

<li><strong>The exit is a snapshot, not an export.</strong> Objects, fields and associations move between sub-accounts via snapshots, subject to the same cap in the target account.</li>
</ul>



<p class="wp-block-paragraph">Read those as design constraints rather than complaints. They tell you the intended scale: dozens of entity types, no. Millions of rows, no. A handful of business nouns with clean identity and moderate volume, yes, and it works well there.</p>



<h2 class="wp-block-heading">The API budget is a design constraint too</h2>



<p class="wp-block-paragraph">If you&#8217;re treating the platform as a datastore behind an integration, the rate limits shape what&#8217;s possible. On the v2 API a marketplace app gets a burst allowance of 100 requests per 10 seconds per resource, and 200,000 requests per day per resource, where a resource means a location or company. Version 1 is end of support, so anything new belongs on v2.</p>



<p class="wp-block-paragraph">Every v2 request carries a version header alongside the bearer token. It&#8217;s easy to forget and produces a confusing failure when missing:</p>



<pre class="wp-block-code"><code>curl -H "Authorization: Bearer $TOKEN" 
     -H "Version: 2021-04-15" 
     -H "Accept: application/json" 
     "$BASE_URL/&lt;resource&gt;?locationId=$LOCATION_ID"</code></pre>



<p class="wp-block-paragraph">More useful than the numbers is that responses carry their own accounting, so you never have to guess where you are in the window: <code>X-RateLimit-Remaining</code> and <code>X-RateLimit-Max</code> cover the burst interval, <code>X-RateLimit-Daily-Remaining</code> and <code>X-RateLimit-Limit-Daily</code> cover the day.</p>



<p class="wp-block-paragraph">Pace against the remaining count rather than reacting to failures, and watch the daily counter separately: burst limits recover in seconds, a daily allowance doesn&#8217;t, and a backfill that exhausts it takes the rest of the day&#8217;s normal traffic with it. The consequence is that polling is expensive and webhooks are cheap. Push where you can; poll only to reconcile.</p>



<h2 class="wp-block-heading">The shape I reach for: CRM in front, system of record behind</h2>



<p class="wp-block-paragraph">When something crosses one of those boundaries, this is the arrangement I default to. It&#8217;s deliberately boring.</p>



<ol class="wp-block-list">
<li><strong>Pick one owner per piece of state and write it down.</strong> Contact details and conversation history belong to the CRM. Money, inventory and anything with an audit requirement belong to the external store. No field gets two owners.</li>

<li><strong>Give every record a stable external ID on both sides.</strong> Do this on day one. Retrofitting identity onto records matched by email is genuinely painful.</li>

<li><strong>Push changes out with webhooks, not polling.</strong> Webhook to a queue, queue to a worker, and make the worker idempotent, because you will receive duplicates and you&#8217;ll need to replay after an outage.</li>

<li><strong>Reconcile on a schedule anyway.</strong> Webhooks get dropped. A nightly sweep comparing counts and updated timestamps, re-pulling only the drift, catches what the push path missed without burning the daily allowance.</li>

<li><strong>Push a flattened summary back for display.</strong> The CRM doesn&#8217;t need the ledger, just the two or three values a human wants on the contact record. Treat those fields as read-only and never let a workflow write them.</li>
</ol>



<p class="wp-block-paragraph">The external side doesn&#8217;t need to be impressive. For a small engagement it&#8217;s usually managed Postgres, or plain Postgres on a modest VPS from somewhere like Contabo or InterServer, sat behind Cloudflare so nothing is directly exposed. The glue is often n8n or Make rather than anything custom. What matters is that constraints exist:</p>



<pre class="wp-block-code"><code>CREATE TABLE job (
  id             bigserial PRIMARY KEY,
  crm_contact_id text        NOT NULL,
  technician_id  bigint      NOT NULL REFERENCES technician(id),
  completed_on   date        NOT NULL,
  value_cents    bigint      NOT NULL CHECK (value_cents &gt;= 0),
  created_at     timestamptz NOT NULL DEFAULT now()
);</code></pre>



<p class="wp-block-paragraph">Nothing clever there. But <code>value_cents</code> can&#8217;t hold &#8220;ask Dave&#8221;, <code>technician_id</code> can&#8217;t point at a technician who doesn&#8217;t exist, and the quarterly margin question becomes one query instead of a conversation about why it can&#8217;t be answered.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<ul class="wp-block-list">
<li><strong>&#8220;We&#8217;re too small for a second system.&#8221;</strong> Sometimes right, and worth taking seriously. But the cost of adding a store later isn&#8217;t the store, it&#8217;s the reconstruction, and some of that history is unrecoverable.</li>

<li><strong>&#8220;We&#8217;ll clean the data up later.&#8221;</strong> Cleanup needs to know what the value should have been. Free-text fields with no history don&#8217;t retain that. This is the one I&#8217;d push back on hardest.</li>

<li><strong>&#8220;Custom Objects solved this.&#8221;</strong> They solved modelling non-person entities, which was the biggest gap. They didn&#8217;t add joins, transactions, versioning or an export path.</li>

<li><strong>&#8220;The client wants everything in one place.&#8221;</strong> They want one place to <em>look</em>, which is a UI requirement, not a storage requirement. A summary pushed back to the contact record satisfies it.</li>

<li><strong>&#8220;We can export whenever we want.&#8221;</strong> Test it before you rely on it. Export the notes. Export a custom object. Do it as the client&#8217;s own user, not as an admin.</li>
</ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<ol class="wp-block-list">
<li><strong>Name the thing.</strong> If it&#8217;s a noun that isn&#8217;t a person, it wants to be a record, not a field on a contact.</li>

<li><strong>Write out the hardest question anyone will ask of it.</strong> The actual sentence. If it needs grouping, arithmetic, or two entity types at once, the CRM isn&#8217;t the home.</li>

<li><strong>Count the writers.</strong> More than one system writing the same field means the CRM holds a copy, not the value.</li>

<li><strong>Decide whether history matters.</strong> Assume it does for anything involving money or consent.</li>

<li><strong>Do the export before you commit.</strong> Not the docs, the actual file. Whatever doesn&#8217;t come out cleanly is data you don&#8217;t really own.</li>

<li><strong>Check the entity budget.</strong> Ten objects per sub-account. If you&#8217;re modelling toward that ceiling, decide now which nouns get a slot.</li>
</ol>



<p class="wp-block-paragraph">Steps two and five take about twenty minutes together and prevent most of what I&#8217;ve described.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I use GoHighLevel as a database for my business data?</h3>



<p class="wp-block-paragraph">For people, conversations, bookings and pipeline state, yes, and it&#8217;s a good one. For anything needing joins, transactions, versioned history or guaranteed export, no. Using <strong>GoHighLevel as a database</strong> for that second category works right up until someone asks a question the platform can&#8217;t express, and by then the data is already ambiguous.</p>



<h3 class="wp-block-heading">How many Custom Objects can a sub-account have?</h3>



<p class="wp-block-paragraph">Ten per location, documented as a hard cap. Each object supports up to ten unique fields, and only single line text, multi line text, number and phone types can be marked unique. Treat those slots as a fixed budget when you plan the model.</p>



<h3 class="wp-block-heading">Can I export Custom Object data out of the platform?</h3>



<p class="wp-block-paragraph">Not through a direct export the way contacts go to CSV. The documented path is a snapshot between sub-accounts, which carries fields and associations but stays inside the platform. The API does support CRUD on objects and records, so a scripted pull is the practical route to a copy you keep.</p>



<h3 class="wp-block-heading">Why do my exported notes look truncated?</h3>



<p class="wp-block-paragraph">Contact export includes only the most recent note, cut to 255 characters. That&#8217;s expected behaviour, not a bug. If notes hold anything you&#8217;d need later, pull them through the API and store them somewhere that keeps all of them.</p>



<h3 class="wp-block-heading">What usually trips the API rate limits?</h3>



<p class="wp-block-paragraph">Backfills, and polling loops that scale with contact count. The limits are 100 requests per 10 seconds and 200,000 per day, both per resource. Read the rate limit headers on each response and pace against them rather than retrying into a wall.</p>



<h3 class="wp-block-heading">Does a sub-account count as a real tenancy boundary?</h3>



<p class="wp-block-paragraph">For operational separation, yes: each is its own location and nothing crosses unless you copy it. The consequence is that cross-client reporting has to happen outside the platform, because no query spans locations. If portfolio-level reporting is a requirement, plan for an external store from the start.</p>



<h3 class="wp-block-heading">Should I migrate off the CRM entirely?</h3>



<p class="wp-block-paragraph">Usually not. A full migration means rebuilding the conversation and outreach layer, which is the part it does best. Moving one category of data out, with a clear owner and a summary pushed back for display, is smaller and reversible.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The CRM will never tell you that you&#8217;ve gone too far. There&#8217;s no constraint to violate, no error to raise, no migration to fail. That&#8217;s exactly what makes the boundary worth drawing on purpose, before the field gets added rather than after four hundred records have been through it.</p>



<p class="wp-block-paragraph">The test I&#8217;d keep: write down the hardest question anyone will ever ask of this data, then check whether the platform can express it. If it can&#8217;t, that&#8217;s not a reporting gap a better dashboard will close. It&#8217;s the boundary telling you where <strong>GoHighLevel as a database</strong> should stop and a real system of record should start. The identity mapping, the webhooks, the reconciliation sweep: all of that is plumbing once the decision is made.</p>



<h2 class="wp-block-heading">Working on this with me</h2>



<p class="wp-block-paragraph">Most of what I do here is unglamorous and ends with the reporting question getting answered:</p>



<ul class="wp-block-list">
<li>Auditing an existing setup and mapping which custom fields have quietly become load-bearing, with an owner assigned to each.</li>

<li>Drawing the boundary: what stays in the CRM, what moves to Custom Objects, what needs an external system of record.</li>

<li>Building the sync layer: webhooks into a queue, idempotent workers, stable ID mapping both ways, and a reconciliation job that catches dropped events without burning the API allowance.</li>

<li>Standing up the store: schema design with constraints that hold, on managed Postgres or a small VPS, with backups you&#8217;ve tested restoring rather than assumed.</li>

<li>Rescuing free-text fields into typed columns, including the judgement calls about what&#8217;s recoverable.</li>

<li>Handover packaging: a real export, documented, so a client can leave with their data whether or not they keep working with you.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;re weighing this up, send me something concrete: a list of your custom field names, a sample export, or the query you can&#8217;t currently answer. That&#8217;s usually enough to tell you where the line sits.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/data-engineering/gohighlevel-as-a-database/">Your CRM Is Not a Database: Where GoHighLevel Should Stop</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Turning External Documents Into Structured Data With Amazon Textract</title>
		<link>https://john-nessime.com/blog/technical-guides/amazon-textract-structured-data/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 22 Sep 2026 06:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon SNS]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Bedrock Data Automation]]></category>
		<category><![CDATA[Boto3]]></category>
		<category><![CDATA[Confidence Scoring]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Human In The Loop]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[OCR]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[Serverless]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=390</guid>

					<description><![CDATA[<p>Amazon Textract almost never fails loudly. A working guide to turning external PDFs and scans into structured data: picking the right operation per document family, parsing the block graph, routing on per-field confidence, and catching the limits that silently truncate your records.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/amazon-textract-structured-data/">Turning External Documents Into Structured Data With Amazon Textract</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A vendor sends over a year of delivery notes. It arrives as one PDF, several hundred pages, scanned on a machine that was clearly overdue for service. Someone asks how long it will take to get the reference numbers into the warehouse table, and the honest answer is that the OCR call is the easy part.</p>



<p class="wp-block-paragraph">Document pipelines are not hard because optical character recognition is inaccurate. Amazon Textract reads text well. They are hard because everything after the API call belongs to you, and because Textract almost never fails loudly. You get a 200 response, a list of blocks, a confidence score, and a record with an empty field where the purchase order number should have been. Nothing alerts. The row lands in the warehouse with a null, and someone in finance notices three weeks later while reconciling.</p>



<p class="wp-block-paragraph">That silent gap is the whole problem. Getting reliable Amazon Textract structured data out of files you did not design has less to do with OCR accuracy than with everything around it: which operation to call for which document family, why the response is a graph rather than a record, how to route on confidence instead of trusting it, and the limits that quietly truncate your data. All of it assumes documents you cannot ask anyone to change.</p>



<h2 class="wp-block-heading">Why external documents break pipelines that passed testing</h2>



<p class="wp-block-paragraph">Internal documents are generated by a template you control. External documents vary along three axes at once, and each one has its own failure mode:</p>



<ul class="wp-block-list">
<li><strong>Container.</strong> One logical document per file, or forty stapled into one PDF, or a single page split across three JPEGs.</li>

<li><strong>Text origin.</strong> Born-digital PDFs with a real text layer, flatbed scans, and photographs taken at an angle in bad light. Same file extension, wildly different accuracy.</li>

<li><strong>Layout stability.</strong> The vendor redesigns their invoice template and moves the PO number from the header to a footer block. Nobody tells you.</li>
</ul>



<p class="wp-block-paragraph">You cannot fix any of that upstream. You can only detect it at ingest and route accordingly. The sections below are organized by the failure family each one produces.</p>



<h2 class="wp-block-heading">Failure family one: the file is out of spec before Textract sees it</h2>



<p class="wp-block-paragraph">Textract accepts JPEG, PNG, PDF, and TIFF. It does not support XFA-based PDFs, and it will not open a password-protected PDF. Images are accepted up to 10,000 pixels on any side. For PDFs, the maximum page dimensions are 40 inches and 9,000 points.</p>



<p class="wp-block-paragraph">Two more constraints catch people out. Text detection covers English, French, German, Italian, Portuguese, and Spanish, and several capabilities are narrower still: handwriting, invoices and receipts, identity documents, and Queries are English only. Vertical text, common in Japanese and Chinese documents, is not supported at all. Rotation is fine, including heavy in-plane rotation. Vertical writing direction is not.</p>



<p class="wp-block-paragraph">Build an ingest gate that checks encryption, page count, byte size, and dimensions before it spends an API call, and give each rejection a reason code a human can act on. Then catch exceptions properly instead of wrapping the call in a bare retry. <code>BadDocumentException</code> means Textract could not read the file, and retrying fails identically. <code>DocumentTooLargeException</code> means route to the asynchronous path or split the file. <code>InvalidS3ObjectException</code> is usually IAM or a bucket-region mismatch, not a document problem. Only <code>InternalServerErrorException</code> deserves a retry with backoff.</p>



<h2 class="wp-block-heading">Failure family two: synchronous versus asynchronous is a correctness gate</h2>



<p class="wp-block-paragraph">This is the one that silently eats data, so it goes early.</p>



<p class="wp-block-paragraph">The synchronous operations, <code>DetectDocumentText</code> and <code>AnalyzeDocument</code>, cap at 10 MB in memory. For PDF and TIFF input they also cap at <strong>one page</strong>. Not one page per call with automatic paging. One page, full stop. Hand a synchronous call a 40-page PDF and you get page one, and a pipeline that reports success.</p>



<p class="wp-block-paragraph">The asynchronous operations are a different shape. PDF and TIFF go up to 500 MB and 3,000 pages, while JPEG and PNG stay at 10 MB. You start a job, Textract works in the background, and you collect results by job ID.</p>



<p class="wp-block-paragraph">A synchronous call looks like this. The document lives in S3, and you name the features you want:</p>



<pre class="wp-block-code"><code>aws textract analyze-document 
  --document '{"S3Object":{"Bucket":"my-doc-bucket","Name":"inbound/note-0042.png"}}' 
  --feature-types '["FORMS","TABLES"]' 
  --region us-east-1</code></pre>



<p class="wp-block-paragraph">The asynchronous version returns a job identifier immediately and publishes completion to an SNS topic. That is what you want in production, because polling a job inside a Lambda function means paying for a function that sleeps:</p>



<pre class="wp-block-code"><code>import boto3

textract = boto3.client("textract")

start = textract.start_document_analysis(
    DocumentLocation={
        "S3Object": {"Bucket": "my-doc-bucket", "Name": "inbound/batch-0042.pdf"}
    },
    FeatureTypes=["FORMS", "TABLES"],
    NotificationChannel={
        "SNSTopicArn": "arn:aws:sns:us-east-1:111122223333:textract-jobs",
        "RoleArn": "arn:aws:iam::111122223333:role/TextractSnsPublishRole",
    },
)

job_id = start["JobId"]</code></pre>



<p class="wp-block-paragraph">Results come back paginated, and forgetting the continuation token is a second, quieter way to lose most of a file:</p>



<pre class="wp-block-code"><code>blocks = []
next_token = None

while True:
    kwargs = {"JobId": job_id}
    if next_token:
        kwargs["NextToken"] = next_token

    page = textract.get_document_analysis(**kwargs)
    blocks.extend(page["Blocks"])

    next_token = page.get("NextToken")
    if not next_token:
        break</code></pre>



<p class="wp-block-paragraph">Check <code>JobStatus</code> before you trust <code>Blocks</code>. A job can come back partially successful, which means some pages processed and some did not. If you treat that as success you will publish a document that is missing its middle.</p>



<h2 class="wp-block-heading">Failure family three: the response is a graph, not a record</h2>



<p class="wp-block-paragraph">Textract does not return fields. It returns a flat list of <code>Block</code> objects, each with an ID and relationships pointing at other IDs. Reassembling those into a row is your code, and it is where most of the real work lives.</p>



<p class="wp-block-paragraph">For form data, the shape is specific. A key-value pair is two blocks of type <code>KEY_VALUE_SET</code>, distinguished by their <code>EntityTypes</code> array containing either <code>KEY</code> or <code>VALUE</code>. The key block carries a relationship of type <code>VALUE</code> holding the ID of its value block, and a relationship of type <code>CHILD</code> holding the IDs of the <code>WORD</code> blocks that spell out the label. The value block carries only <code>CHILD</code> relationships pointing at its own words. So reading one field means three hops.</p>



<pre class="wp-block-code"><code>blocks_by_id = {b["Id"]: b for b in blocks}

def child_text(block):
    parts = []
    for rel in block.get("Relationships", []):
        if rel["Type"] != "CHILD":
            continue
        for cid in rel["Ids"]:
            child = blocks_by_id[cid]
            if child["BlockType"] == "WORD":
                parts.append(child["Text"])
            elif child["BlockType"] == "SELECTION_ELEMENT":
                parts.append(child["SelectionStatus"])
    return " ".join(parts)

fields = {}
for b in blocks:
    if b["BlockType"] != "KEY_VALUE_SET":
        continue
    if "KEY" not in b.get("EntityTypes", []):
        continue

    value_text = ""
    for rel in b.get("Relationships", []):
        if rel["Type"] == "VALUE":
            for vid in rel["Ids"]:
                value_text = child_text(blocks_by_id[vid])

    fields[child_text(b)] = (value_text, b["Confidence"])</code></pre>



<p class="wp-block-paragraph">Two things about that snippet. The <code>SELECTION_ELEMENT</code> branch matters because checkboxes and radio buttons are values too, and their content is a selection status rather than text. And the confidence on the key block is not a separate key confidence: Textract returns the same value for both halves of a pair. If you planned to compare label confidence against value confidence to catch mismatches, that signal does not exist.</p>



<p class="wp-block-paragraph">Tables follow the same pattern with different block types, and cells carry row and column indices plus span values for merged cells. Layout adds another set, including title, header, footer, section header, page number, list, and figure blocks, and it sequences text into human reading order. That last property is why Layout is worth requesting when the output is headed for a language model rather than a database: chunking on reading order beats chunking on raw line order.</p>



<p class="wp-block-paragraph">Before hand-rolling any of this, look at the AWS-maintained parsers. The <code>amazon-textract-response-parser</code> package and the higher-level <code>amazon-textract-textractor</code> library both turn the block graph into objects with sensible properties. Writing your own traversal is a good way to understand the model and a poor way to spend two sprints.</p>



<h2 class="wp-block-heading">Getting Amazon Textract structured data out of the right API</h2>



<p class="wp-block-paragraph">Textract is a family of operations, not one endpoint, and the biggest single accuracy win available is usually picking a narrower one. Features are billed per page and they stack, so requesting <code>TABLES</code> on a corpus with no tables costs money and returns noise.</p>



<ul class="wp-block-list">
<li><strong>DetectDocumentText.</strong> Raw text and geometry. Use it when you only need a search index or a text layer.</li>

<li><strong>AnalyzeDocument with FORMS.</strong> Key-value pairs discovered from labels on the page. Works when the label is physically present and stable.</li>

<li><strong>AnalyzeDocument with TABLES.</strong> Grid reconstruction with row and column indices. The right call for line-item grids and statements.</li>

<li><strong>AnalyzeDocument with QUERIES.</strong> You ask natural language questions and get answers back. This is the fix for layout drift.</li>

<li><strong>AnalyzeDocument with LAYOUT and SIGNATURES.</strong> Reading order and structural elements; signature locations for compliance checks.</li>

<li><strong>AnalyzeExpense.</strong> Purpose-built for invoices and receipts, with normalized field types.</li>

<li><strong>AnalyzeID.</strong> Purpose-built for identity documents.</li>
</ul>



<h3 class="wp-block-heading">Why Queries beats FORMS on documents you don&#8217;t control</h3>



<p class="wp-block-paragraph">FORMS gives you whatever labels the document happens to contain. That means your downstream schema is hostage to a vendor&#8217;s wording. When they change &#8220;P.O. No.&#8221; to &#8220;Order Reference&#8221;, your parser looks for a key that no longer exists and writes a null.</p>



<p class="wp-block-paragraph">Queries inverts this. You supply the question and an alias, and the alias comes back attached to the answer, so your output key is fixed by you rather than by the document:</p>



<pre class="wp-block-code"><code>aws textract analyze-document 
  --document '{"S3Object":{"Bucket":"my-doc-bucket","Name":"inbound/note-0042.png"}}' 
  --feature-types '["QUERIES"]' 
  --queries-config '{"Queries":[
      {"Text":"What is the purchase order number?","Alias":"PO_NUMBER"},
      {"Text":"What is the delivery date?","Alias":"DELIVERY_DATE"}
  ]}' 
  --region us-east-1</code></pre>



<p class="wp-block-paragraph">The alias is the entire point. It is a contract between your schema and the extraction layer that survives a vendor redesign. There are ceilings: 15 queries per page synchronously, 30 per page asynchronously, English only. Phrase questions using words that actually appear on the page, and add positional hints when you know the layout, because the model uses them. If Queries underperforms on your corpus, adapters let you tune the feature against your own examples and pass an adapter ID and version at call time.</p>



<h3 class="wp-block-heading">Invoices are their own problem</h3>



<p class="wp-block-paragraph">For invoices and receipts, reach for AnalyzeExpense first. Its response is shaped around the domain rather than around the page: each <code>ExpenseDocument</code> holds <code>SummaryFields</code> for header-level data and <code>LineItemGroups</code> containing the line items. Fields carry a normalized <code>Type</code> such as <code>VENDOR_NAME</code>, plus a <code>ValueDetection</code> and an optional <code>LabelDetection</code>.</p>



<p class="wp-block-paragraph">That optional label is the thing to notice. AnalyzeExpense can identify a vendor name printed only inside a logo, and pull line items out of a grid with no column headers. FORMS structurally cannot do either, because FORMS needs a key to exist. On receipts that is not a small accuracy delta, it is the difference between a usable field and an empty one.</p>



<h2 class="wp-block-heading">Failure family four: confidence is per field and it is not your risk model</h2>



<p class="wp-block-paragraph">Every extracted element carries a confidence score. Two mistakes follow from that.</p>



<p class="wp-block-paragraph">The first is averaging into a document-level score. A document where every field is strong except the invoice total is not a mostly-good document. It is a document with one field that must not be trusted, surrounded by noise. Score fields, not documents.</p>



<p class="wp-block-paragraph">The second is treating one threshold as universal. A misread address on a delivery note is an annoyance. A misread bank account number is an incident. Set thresholds per field from the cost of being wrong, then route into three lanes:</p>



<ol class="wp-block-list">
<li><strong>Auto-accept.</strong> Above threshold and passing a format check. Straight into the target table.</li>

<li><strong>Review.</strong> Below threshold, or above it but failing a validation rule. A human sees the value and the cropped region it came from.</li>

<li><strong>Reject.</strong> Field absent entirely, or the document failed the ingest gate. Back to the sender with a reason.</li>
</ol>



<p class="wp-block-paragraph">Validation rules do more work than thresholds here. A total that does not equal the sum of its line items is wrong at any confidence. Check-digit rules on account and tax identifiers catch character-level errors the model was perfectly confident about. Confidence tells you how sure the model is. Validation tells you whether the answer is even possible.</p>



<p class="wp-block-paragraph">Textract can hand low-confidence results to Amazon Augmented AI through a human loop configuration on the API call, which saves building a review queue. Whether that beats a small internal tool depends on how much domain context your reviewers need on screen. Either way, capture the corrections. Reviewer edits are the only labeled data you get for free, and they are how you calibrate thresholds against your own corpus instead of guessing.</p>



<h2 class="wp-block-heading">Failure family five: reprocessing you didn&#8217;t budget for</h2>



<p class="wp-block-paragraph">Textract bills per page, per feature. Your parser, meanwhile, will change repeatedly in the first months of a pipeline&#8217;s life, because that is what parsers do.</p>



<p class="wp-block-paragraph">So separate extraction from parsing, permanently. Write the raw response JSON to S3 keyed by a hash of the document bytes plus the operation and feature set requested, then parse from that stored artifact. Reprocessing becomes free, deterministic, and diffable: you can replay a parser change across the whole back catalogue and compare outputs before promoting it. Without that cache, every parser bug fix is a bill.</p>



<pre class="wp-block-code"><code>import hashlib

digest = hashlib.sha256(doc_bytes).hexdigest()
features = "-".join(sorted(["FORMS", "TABLES"]))
key = f"textract-raw/{digest}/analyze-document/{features}.json"</code></pre>



<p class="wp-block-paragraph">Hashing content rather than filename also deduplicates the same scan arriving twice under different names, which happens constantly with email-driven intake. For the asynchronous operations, a client request token gives idempotency at the job level; reusing a token with different parameters raises a mismatch error rather than quietly starting a second job.</p>



<p class="wp-block-paragraph">One caveat: source documents often contain personal data, so a cache is a retention decision. Set S3 lifecycle rules on both the originals and the raw JSON, and if working copies land on a workstation or a build box, delete them properly. Tools like O&amp;O SafeErase exist for exactly that gap between &#8220;moved to trash&#8221; and &#8220;actually gone&#8221;.</p>



<h2 class="wp-block-heading">Where Textract stops being the right tool</h2>



<p class="wp-block-paragraph">Being fair about this matters, because AWS now offers overlapping paths to the same outcome. Amazon Bedrock Data Automation is a managed document processing service that handles classification and extraction behind one API, priced per document rather than per page and per feature. AWS positions it as the default starting point for new builds, and for mixed document types with no dominant format that recommendation is sound. You write far less orchestration.</p>



<p class="wp-block-paragraph">Textract keeps the advantage where the corpus is high volume and standardized, where geometry and per-field confidence are needed for auditability, where regional availability constrains you, or where you want extraction to be deterministic and separately testable. Bounding boxes and confidence scores are not a nice-to-have in a regulated workflow, they are the evidence trail.</p>



<p class="wp-block-paragraph">The hybrid is common and sensible: Textract for extraction, a Bedrock model for reasoning over the extracted text. Just do not send a multimodal model a full-page image because OCR felt inconvenient. It costs considerably more per page and gives you no geometry.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Only the first page came through.</strong> A synchronous call on a multi-page PDF or TIFF. Move to the asynchronous operations, and assert the page count Textract reports against the count you expected.</li>

<li><strong>Output truncates partway through a long document.</strong> Missing continuation token handling on the get operation. Loop until the token is absent.</li>

<li><strong>Fields visible on the page never appear in output.</strong> The label is implied rather than printed, or sits inside a logo. Switch that field to Queries, or to AnalyzeExpense if it is an invoice.</li>

<li><strong>Quality collapsed for one sender only.</strong> Template change. Diff the current layout against a stored sample, which is the argument for keeping raw responses.</li>

<li><strong>Throttling during a backfill.</strong> Transactions per second and concurrent job counts are per-account, per-region quotas. Queue submissions rather than fanning out, and request an increase before the migration.</li>

<li><strong>Table cells land in the wrong columns.</strong> Usually a skewed scan. Deskew and raise contrast before submitting, because image preparation is cheaper than post-processing repairs.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Testing on three clean sample documents supplied by the vendor rather than a random sample of real intake.</li>

<li>Requesting every feature type on every page because it is simpler than routing, then being surprised by the bill.</li>

<li>Using FORMS for invoices when AnalyzeExpense exists and understands unlabeled fields.</li>

<li>Treating a confidence score as an accuracy guarantee instead of pairing it with validation rules.</li>

<li>Discarding the raw JSON after parsing, which turns every parser fix into a re-extraction charge.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Classify before you extract. Route each document to the narrowest API that covers it.</li>

<li>Use Queries with explicit aliases for fields your schema depends on, so vendor wording changes cannot break your column names.</li>

<li>Store raw responses immutably, keyed by content hash, and parse from storage.</li>

<li>Set per-field confidence thresholds from the cost of being wrong, and pair every threshold with a format or arithmetic validation.</li>

<li>Instrument it like any other pipeline: extraction latency, straight-through rate, review queue depth, per-field null rate. A slow rise in nulls on one field is the earliest signal of a template change. Prometheus with Grafana, or Grafana Cloud if you would rather not run it yourself, covers this well.</li>

<li>Keep documents and pipeline in the same region, use VPC endpoints for sensitive data, and apply lifecycle rules to originals and extracted output alike.</li>

<li>Keep a golden set of real documents with known correct answers, and run it on every parser change.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can Amazon Textract process multi-page PDFs?</h3>



<p class="wp-block-paragraph">Yes, but only through the asynchronous operations. Synchronous calls accept a single page for PDF and TIFF input. Asynchronous processing handles PDF and TIFF up to 500 MB and 3,000 pages.</p>



<h3 class="wp-block-heading">What is the difference between FORMS and Queries?</h3>



<p class="wp-block-paragraph">FORMS returns whatever key-value pairs it finds from labels printed on the page, so your output keys follow the document. Queries lets you ask for values by natural language question and attach your own alias to each answer, so output keys stay fixed when the layout changes. For a schema you must keep stable, Queries is the safer contract.</p>



<h3 class="wp-block-heading">How much does Amazon Textract cost?</h3>



<p class="wp-block-paragraph">Billing is per page and varies by operation and feature, and requesting several features on one page adds their costs together. Check the current rate card for your region and multiply by pages, not documents. The practical lever is routing: send each document to the narrowest operation that answers your question, and cache raw responses so parser iteration is free.</p>



<h3 class="wp-block-heading">Does Textract handle handwriting?</h3>



<p class="wp-block-paragraph">It detects handwritten text, in English only. Treat handwritten fields as a separate confidence tier with lower auto-accept thresholds, because the variance across writers is much wider than for printed text.</p>



<h3 class="wp-block-heading">Which languages does Amazon Textract support?</h3>



<p class="wp-block-paragraph">Text detection covers English, French, German, Italian, Portuguese, and Spanish. Handwriting, invoices and receipts, identity documents, and Queries are English only. Vertically written text is not supported.</p>



<h3 class="wp-block-heading">Should I use Textract or Bedrock Data Automation?</h3>



<p class="wp-block-paragraph">Bedrock Data Automation is the lower-effort default for mixed document types, with classification and extraction behind a single managed API. Textract fits better at high volume on standardized documents, where per-field confidence and geometry matter for audit, or where extraction needs to be deterministic and independently testable. Combining them is normal.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">If one thing survives this post, make it this: Amazon Textract structured data extraction fails quietly, not loudly. The API returns success while your record is missing its most important field, because a synchronous call read one page of forty, or because a vendor renamed a label, or because a value came back at low confidence and nothing was watching.</p>



<p class="wp-block-paragraph">So build for the silence. Gate at ingest, use asynchronous operations by default, pick the narrowest API per document family, pin your schema with query aliases, store the raw response, and route on per-field confidence backed by validation rules that know what a possible answer looks like. Do that and the OCR really does become the easy part.</p>



<h2 class="wp-block-heading">Need help turning documents into a pipeline you can trust?</h2>



<p class="wp-block-paragraph">Most of the difficulty in document extraction shows up after the proof of concept, once real intake replaces the sample files. That is the part I work on:</p>



<ul class="wp-block-list">
<li>Designing the ingest gate and classification step so each document family reaches the right Textract operation instead of one call doing everything.</li>

<li>Building the asynchronous path properly: S3 events, SNS completion, paginated retrieval, dead-letter handling, idempotent reprocessing.</li>

<li>Replacing brittle FORMS parsing with query aliases or AnalyzeExpense, and validating the change against a golden set of your own documents.</li>

<li>Setting per-field confidence thresholds and validation rules, then wiring a review queue whose corrections feed back into tuning.</li>

<li>Adding observability that catches template drift early: per-field null rates, straight-through rate, and review queue depth on a dashboard.</li>

<li>Getting the cost curve under control through routing, raw-response caching, and lifecycle rules on stored documents and JSON.</li>
</ul>



<p class="wp-block-paragraph">If you have a pipeline misbehaving, send me a sample document with the fields you need and the raw Textract JSON it produced. That is usually enough to see where it is going wrong.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/amazon-textract-structured-data/">Turning External Documents Into Structured Data With Amazon Textract</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Six Signs GoHighLevel Needs a Developer, Not Another Workflow</title>
		<link>https://john-nessime.com/blog/workflow-automation/gohighlevel-custom-development-vs-workflows/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 17 Sep 2026 06:00:00 +0000</pubDate>
				<category><![CDATA[Enterprise Integration]]></category>
		<category><![CDATA[SaaS Engineering]]></category>
		<category><![CDATA[Workflow Automation]]></category>
		<category><![CDATA[Architecture Debt]]></category>
		<category><![CDATA[CRM Audit]]></category>
		<category><![CDATA[Error Handling]]></category>
		<category><![CDATA[Execution Logs]]></category>
		<category><![CDATA[Exponential Backoff]]></category>
		<category><![CDATA[GoHighLevel]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[iPaaS]]></category>
		<category><![CDATA[Marketing Automation]]></category>
		<category><![CDATA[n8n]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Over-Engineering]]></category>
		<category><![CDATA[Private Integration]]></category>
		<category><![CDATA[Project Scoping]]></category>
		<category><![CDATA[Race Conditions]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[Reconciliation]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Sub-Account Management]]></category>
		<category><![CDATA[Webhooks]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=647</guid>

					<description><![CDATA[<p>Adding another workflow is usually the cheap answer, until it isn't. Six signals that a GoHighLevel account has outgrown the builder, the honest case for staying in it, and the three levels of code to try before committing to a marketplace app.</p>
<p>The post <a href="https://john-nessime.com/blog/workflow-automation/gohighlevel-custom-development-vs-workflows/">Six Signs GoHighLevel Needs a Developer, Not Another Workflow</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The account had forty-one workflows. Two were named &#8220;Lead Follow-Up v3&#8221; and &#8220;Lead Follow-Up v3 (USE THIS ONE)&#8221;. When someone asked what happens to a contact who submits the form twice in one afternoon, nobody could answer. Not the agency owner, not the person who built it.</p>



<p class="wp-block-paragraph">That is the moment. Not a crash, not an error banner. A plain question about the system that nobody can answer from the canvas.</p>



<p class="wp-block-paragraph">GoHighLevel is good at what it was built for and most accounts never need anything else. But there is a point where the next workflow makes the account worse instead of better, and past it, <strong>GoHighLevel custom development</strong> costs less than another six months of patching. Below: the failure mode hiding under all the others, six signals you have crossed that line, the honest case for staying in the builder, and three levels of code to try before committing to a full app.</p>



<h2 class="wp-block-heading">The failure hiding under all the others</h2>



<p class="wp-block-paragraph">The workflow builder has no failure semantics. Everything below is a consequence of that.</p>



<p class="wp-block-paragraph">When a step works, the contact moves on. When a step fails, the contact usually also moves on. No transaction, no rollback, no retry budget you control, no way to say &#8220;this contact is in a bad state, stop and tell someone&#8221;. A Custom Webhook that gets a 500 back does not park the contact for review. It becomes a line in an execution log nobody reads, and the contact continues as though the other system had answered.</p>



<p class="wp-block-paragraph">Fine when the step sends a text message. Not fine when the step is the only thing keeping two systems in agreement. So the useful test is not &#8220;is this hard to build in the builder&#8221;, because plenty of hard things build fine. The test is: <em>what happens when this step fails, and would I find out?</em></p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Signal one: you are storing state in tags</h2>



<p class="wp-block-paragraph">Tags start as labels. Then someone needs to know whether a lead was called, so there is a <code>called</code> tag. Then whether they were called twice, so <code>called-2</code>. Then a workflow that removes <code>called</code> when a new form arrives, and another that puts it back if an appointment was booked first.</p>



<p class="wp-block-paragraph">Those are no longer labels. That is a state machine with no diagram, no validation, and no guarantee that only one workflow writes to it at a time. Two workflows fired by the same event will both read the old value and both write, and whichever finishes last wins. That is a race condition, and it hides during testing because you test one contact at a time. The tell is a tag name with a number in it, or a tag whose meaning needs the word &#8220;unless&#8221; to explain.</p>



<h2 class="wp-block-heading">Signal two: the data has to be right, not eventually right</h2>



<p class="wp-block-paragraph">Marketing automation tolerates drift. One extra nurture email hurts nobody. The moment GoHighLevel holds something that must reconcile with another system, that tolerance is gone. Order status from a store, invoice state from an accounting package, availability shared with a clinic system.</p>



<p class="wp-block-paragraph">Those need idempotency, meaning a repeated message produces the same result as a single one, and reconciliation, meaning something periodically compares both sides and reports the difference. The builder gives you neither: no natural home for an idempotency key, no scheduled job that reads both systems and complains. You can fake reconciliation with a workflow on a timer, but that is a batch job written in a UI designed for follow-up sequences, and the first time it half-runs you will not hear about it.</p>



<h2 class="wp-block-heading">Signal three: per-execution billing became a per-lead tax</h2>



<p class="wp-block-paragraph">The actions that make workflows powerful are premium actions, billed per execution against the agency wallet. Custom Webhook, Inbound Webhook, Custom Code and the workflow AI actions all sit in that category. HighLevel gives each sub-account a small free allowance when premium actions are first switched on, publishes volume tiers above that, and lets you rebill sub-accounts individually instead of absorbing the charge. Take the current rate and tiers from HighLevel&#8217;s billing documentation rather than any blog post, including this one.</p>



<p class="wp-block-paragraph">The mechanism is the point. You pay per execution, not per useful outcome. A workflow calling out three times to enrich, score and push a lead is three executions per lead. Multiply by volume and sub-accounts, and the plumbing line can pass the cost of a small service doing all three in one request.</p>



<p class="wp-block-paragraph">Two caveats. Consolidating three calls into one endpoint is usually cheaper than moving the whole thing to code. And whatever you move still costs something to run, whether that is a VPS from a provider like Contabo or InterServer, a hosted platform such as Make or n8n, or a serverless function. Compare both sides.</p>



<h2 class="wp-block-heading">Signal four: the same logic lives in twenty sub-accounts</h2>



<p class="wp-block-paragraph">Snapshots are the platform&#8217;s answer and better than people expect. You can refresh a snapshot and push selected assets to sub-accounts that previously imported it, choosing both the assets and the recipients, and the push is selective rather than destructive.</p>



<p class="wp-block-paragraph">Two limits. Push Update only reaches sub-accounts inside your own agency, so anything shared externally has to be re-imported through a new link. And HighLevel documents that when a snapshot refresh deletes steps, contacts waiting on those steps in workflows created from that snapshot are removed. Fine for a copy change, destructive for a structural one.</p>



<p class="wp-block-paragraph">Snapshots distribute structure. They do not give you versioning, a rollback path, a diff between intended and running state, or a test that fails before the change reaches a client. Once a change has to land correctly in twenty accounts on one day, you want those, and they come from putting the logic behind an API you control and calling it from a thin step in each account.</p>



<h2 class="wp-block-heading">Signal five: the integration must survive the other side being down</h2>



<p class="wp-block-paragraph">A Custom Webhook fires the moment the contact reaches the step. If your endpoint is slow, restarting or rate limited, that single firing is what you get. No queue in front, no backoff behind.</p>



<p class="wp-block-paragraph">The API side has numbers worth knowing. HighLevel documents a burst limit of 100 requests per 10 seconds and a daily limit of 200,000 requests, counted per marketplace app per resource, where a resource is a location or the company. Every response carries headers telling you where you stand, so backoff can follow what the server says instead of a guessed sleep.</p>



<pre class="wp-block-code"><code>curl -sS -D - -o /dev/null 
  -H "Authorization: Bearer $GHL_TOKEN" 
  -H "Version: $GHL_API_VERSION" 
  "$GHL_API_BASE/contacts/" 
  | grep -i '^x-ratelimit'</code></pre>



<p class="wp-block-paragraph"><code>-D -</code> writes response headers to stdout while <code>-o /dev/null</code> discards the body, so you read the rate limit state without dumping contact data into your shell history. Set the base URL and the required <code>Version</code> value from the current API reference. Watch <code>X-RateLimit-Max</code> and <code>X-RateLimit-Remaining</code> for the burst window, <code>X-RateLimit-Interval-Milliseconds</code> for its length, and <code>X-RateLimit-Limit-Daily</code> with <code>X-RateLimit-Daily-Remaining</code> for the day.</p>



<p class="wp-block-paragraph">The daily budget sounds enormous until a migration. Reading a contact, upserting it, then writing custom fields is three calls each, so forty thousand contacts is 120,000 calls before live traffic touches the account. Batch the reads, skip writes where nothing changed, run backfills overnight. Watch for loops too: writes made through the API can themselves fire workflow triggers, so a receiver that writes back has built a cycle.</p>



<h2 class="wp-block-heading">Signal six: someone needs a screen, not a form</h2>



<p class="wp-block-paragraph">Forms collect. They do not display. Once a client asks to see a filtered view of their own data, approve something in two clicks, or work a queue that is not a pipeline, you are looking at an interface.</p>



<p class="wp-block-paragraph">The platform supports this. Marketplace apps can add Custom Pages, which render inside HighLevel in an embedded iframe loading a URL you host, appearing on the app details page or in the left navigation. HighLevel publishes an app template to start from, plus an SSO mechanism so your page can identify the user instead of asking them to log in again. That is still a real application with hosting, auth and deployment: the biggest step on this list, the one I would defer longest, and the only honest answer when the requirement is a screen.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Where the workflow builder genuinely wins</h2>



<p class="wp-block-paragraph">A one-sided argument is not worth reading, so here is the case against writing code.</p>



<ul class="wp-block-list">
<li><strong>The person who owns the outcome can change it.</strong> An account manager adjusts send timing on a Friday with no deploy and no developer awake.</li>



<li><strong>Messaging and scheduling are solved.</strong> SMS, email, calendars, reminders and the delivery infrastructure behind them are the platform&#8217;s core. Rebuilding any of it is a bad trade.</li>



<li><strong>Timezone handling is built in.</strong> Time-based steps run in the account timezone or the contact&#8217;s own, falling back to the account when a contact has none.</li>



<li><strong>No pipeline to maintain.</strong> No repository, no secrets rotation, no dependency upgrades, no server needing patched at an awkward hour.</li>



<li><strong>Execution history is right there.</strong> For a linear sequence, reading what happened to a contact beats correlating logs across two systems.</li>
</ul>



<p class="wp-block-paragraph">If the automation is a follow-up sequence with branching, the builder is correct and code is the expensive mistake.</p>



<h2 class="wp-block-heading">Three levels before you build an app</h2>



<p class="wp-block-paragraph">&#8220;Needs a developer&#8221; is not one decision. It is three, and most accounts only need the first.</p>



<h3 class="wp-block-heading">Level one: the Custom Code action</h3>



<p class="wp-block-paragraph">The builder has a Custom Code premium action that runs JavaScript inside the workflow. Values from earlier steps are mapped in as named input properties and arrive in an <code>InputData</code> object. The action returns a JavaScript object or an array of objects for later steps to reference, and <code>console.log</code> output is captured so you can see what ran.</p>



<pre class="wp-block-code"><code>// Normalise a phone number before it reaches a branch step.
// rawPhone and country are input properties mapped from earlier steps.
const digits = String(InputData.rawPhone || "").replace(/D/g, "");
const cc = InputData.country === "US" ? "1" : "";

if (!digits) {
  console.log("no phone on contact, routing to manual branch");
  return { e164: "", usable: false };
}

return {
  e164: "+" + (digits.startsWith(cc) ? digits : cc + digits),
  usable: digits.length &gt;= 10
};</code></pre>



<p class="wp-block-paragraph">This kills a lot of branch sprawl. Ten If/Else steps existing only to reshape a value collapse into one action returning a clean value and a boolean. Still inside the platform, still billed per execution, still no retries. But logic smeared across a canvas now reads in one place.</p>



<h3 class="wp-block-heading">Level two: a private integration and a script you own</h3>



<p class="wp-block-paragraph">For one account you do not need a marketplace app. Create a Private Integration inside the sub-account, select only the scopes the job needs, and use the token as a bearer credential. No OAuth handshake, no refresh cycle, no listing.</p>



<p class="wp-block-paragraph">This is where reconciliation jobs, nightly audits, migrations and bulk cleanups belong. Run it as a cron job on a small VPS, behind a Cloudflare Tunnel if it needs an inbound endpoint, with the token in a secrets store rather than a config file. Treat that token as a credential with a rotation plan, not a string you paste once and forget.</p>



<h3 class="wp-block-heading">Level three: a marketplace app</h3>



<p class="wp-block-paragraph">You need this when the thing spans many accounts, has a user interface, or is a product you intend to sell. It means OAuth 2.0 on the authorization code grant, token storage and refresh per installed location, webhook handling, and somewhere to run it. Note the version boundary: the v1 API is end of life and unmaintained, and new builds target the OAuth-based v2. If a tutorial hands you a permanent location API key, it is describing the old world.</p>



<p class="wp-block-paragraph">Marketplace apps can also register their own workflow triggers and actions, which appear in the builder for sub-accounts that installed the app. That is the pattern I reach for first: the hard logic lives in your service, and the person who owns the campaign still drags a step onto a canvas. It requires the workflows read scope, and those custom actions are themselves premium actions billed per execution.</p>



<h2 class="wp-block-heading">Arguments that do not survive contact</h2>



<ul class="wp-block-list">
<li><strong>&#8220;We will just add one more workflow.&#8221;</strong> The cost is not building it. It is that every future workflow now has to be reasoned about alongside it. Ten workflows listening for one tag is not ten problems, it is the interactions between them.</li>



<li><strong>&#8220;No-code means no developer.&#8221;</strong> It means no deployment. Someone still holds the whole model in their head, and when they leave, an undocumented canvas is harder to inherit than a repository with a README.</li>



<li><strong>&#8220;Zapier or Make will handle it.&#8221;</strong> Often true and worth trying first. What they do not give you is one place where business logic lives. Split rules across three tools and debugging means three log formats.</li>



<li><strong>&#8220;It works, so it is fine.&#8221;</strong> Every silent failure worked right up until someone counted. Absence of errors in a system with no error reporting is not evidence.</li>



<li><strong>&#8220;Custom code is lock-in.&#8221;</strong> Backwards. Logic on a canvas is bound to one vendor&#8217;s UI and exports only as a snapshot. Logic in a repository moves.</li>
</ul>



<h2 class="wp-block-heading">How I would decide</h2>



<ol class="wp-block-list">
<li><strong>Write down what happens when each external call fails.</strong> If the answer anywhere is &#8220;the contact continues as if it succeeded&#8221; and that matters, you have found your requirement.</li>



<li><strong>Count the workflows writing to the same tag or field.</strong> More than two means you have a state machine. Draw it before touching anything.</li>



<li><strong>Count premium executions per lead, then multiply by monthly volume.</strong> Compare against a few hours a month maintaining a small service. Sometimes the builder still wins. Do the arithmetic instead of assuming.</li>



<li><strong>Ask how a change reaches every account.</strong> If it involves opening sub-accounts one at a time, that is the real cost, and it grows with every client.</li>



<li><strong>Try the smallest level that fits, and leave the trigger in the builder.</strong> Custom Code action, then private integration script, then marketplace app. Whatever moves out, keep the entry point as a workflow step so the marketing side can still change timing and audience without you.</li>
</ol>



<p class="wp-block-paragraph">Whatever you build, monitor it. An uptime check on your receiver and an alert when the reconciliation job reports a non-zero difference will catch more real problems than any amount of careful workflow naming.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need a marketplace app to use the GoHighLevel API?</h3>



<p class="wp-block-paragraph">No. For a single sub-account, create a Private Integration inside that account, grant only the scopes you need, and use the token as a bearer credential. Marketplace apps and OAuth are for anything spanning accounts or distributed to other people.</p>



<h3 class="wp-block-heading">How much automation should stay in workflows?</h3>



<p class="wp-block-paragraph">The parts a non-developer should be able to change: timing, audience, message copy, branching on obvious contact attributes. Move out the parts where being wrong is expensive and being wrong is silent.</p>



<h3 class="wp-block-heading">Why does my workflow show as successful when the integration clearly failed?</h3>



<p class="wp-block-paragraph">A step firing and a step achieving something are different events, and the builder records the first. A webhook returning an error still counts as having run. If the outcome matters, verify it explicitly rather than trusting the log.</p>



<h3 class="wp-block-heading">Should I avoid premium workflow actions on cost grounds?</h3>



<p class="wp-block-paragraph">Not on principle. They cost per execution because they are the expensive verbs. The problem is fragmentation: three separate calls where one would do triples the bill for the same outcome. Consolidate before considering a move off the platform.</p>



<h3 class="wp-block-heading">What does a GoHighLevel custom development project usually start with?</h3>



<p class="wp-block-paragraph">An audit, not a build. Map which workflows write to which fields, where external calls happen, and what the failure path is for each. Often that document alone removes the need for the project.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Ask what happens when a step fails. That question separates automation belonging in the builder from automation needing code, more reliably than counting steps, counting workflows, or judging how clever the branching has become.</p>



<p class="wp-block-paragraph">If failure is visible and cheap, stay in the builder and enjoy that your client can edit it themselves. If failure is silent and expensive, GoHighLevel custom development stops being an upgrade and becomes maintenance you already deferred. Then start small: a Custom Code action beats a marketplace app you will not want to maintain.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Working on this with me</h2>



<p class="wp-block-paragraph">I work on the engineering side of GoHighLevel accounts, usually the parts failing quietly rather than the parts that look broken.</p>



<ul class="wp-block-list">
<li>Auditing an account for silent failures: which workflows write to which tags and fields, and what happens on every external call that returns an error.</li>



<li>Replacing branch sprawl with Custom Code actions, so logic reads as one block instead of fifteen steps on a canvas.</li>



<li>Webhook receivers and sync services that retry, deduplicate and reconcile, with the rate limit headers driving backoff instead of a fixed sleep.</li>



<li>Migrations and backfills planned against the daily API budget, batched and run in the account&#8217;s timezone so live traffic keeps working.</li>



<li>Private integration setup with least-privilege scopes, token storage and a rotation plan that is written down.</li>



<li>Marketplace app work when genuinely warranted: OAuth handling, custom workflow actions, and Custom Pages hosted somewhere you control.</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion, send something concrete: a screenshot of the workflow you are least sure about, an execution log entry that makes no sense, or the response body your webhook endpoint is returning. That is a faster conversation than a scoping call.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/workflow-automation/gohighlevel-custom-development-vs-workflows/">Six Signs GoHighLevel Needs a Developer, Not Another Workflow</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>One Customer, Four Systems: Unifying Shopify, Stripe, CRM and Support Data</title>
		<link>https://john-nessime.com/blog/technical-guides/unify-shopify-stripe-crm-support-data/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 29 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Customer 360]]></category>
		<category><![CDATA[Customer Support Analytics]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Data Warehouse]]></category>
		<category><![CDATA[dbt]]></category>
		<category><![CDATA[Ecommerce Analytics]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Event-Driven Architecture]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Identity Resolution]]></category>
		<category><![CDATA[PII Redaction]]></category>
		<category><![CDATA[Reverse ETL]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[Stripe]]></category>
		<category><![CDATA[Webhooks]]></category>
		<category><![CDATA[Zendesk]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=350</guid>

					<description><![CDATA[<p>Three CRM records, one person, and nothing in the logs to explain it. A practical walkthrough of what actually breaks when you unify Shopify, Stripe, CRM and support data: guest-checkout identity, duplicate and out-of-order webhooks, revenue that never ties to payouts, deletion requests that miss half your copies, and reverse ETL write loops. Organized by failure family, with schema and handler patterns you can apply directly.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/unify-shopify-stripe-crm-support-data/">One Customer, Four Systems: Unifying Shopify, Stripe, CRM and Support Data</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The project usually starts with a screenshot. Someone in support pastes three CRM records into a channel, all carrying the same person&#8217;s name, and asks which one is real. One has the order history. One has the subscription. The third has four tickets attached and no orders at all.</p>



<p class="wp-block-paragraph">Nobody broke anything. Shopify, Stripe, the CRM and the helpdesk each created a customer record using whatever identifier they had at the moment they needed one, and none of those identifiers agreed. The systems are all correct in isolation. The person is the thing that got lost.</p>



<p class="wp-block-paragraph">This post covers what actually breaks when you unify Shopify, Stripe, CRM and support data, and how to build the join so it survives guest checkouts, duplicate webhooks, refunds, currency conversion and deletion requests. It&#8217;s organized by failure family rather than by tool, because the tools get acquired and renamed every couple of years and the failures never change.</p>



<h2 class="wp-block-heading">Failure family one: the join key is not the email address</h2>



<p class="wp-block-paragraph">Every one of these systems will happily give you an email address, which is exactly why so many pipelines join on it and quietly produce garbage.</p>



<p class="wp-block-paragraph">Start with where identity actually lives in each system:</p>



<ul class="wp-block-list">
<li><strong>Shopify</strong> has a customer ID, but an order does not have to be attached to one. Guest checkout produces an order with an email on it and no durable customer record behind it.</li>

<li><strong>Stripe</strong> has a customer ID, but only if a Customer object was created. Checkout Sessions that don&#8217;t create one are associated with guest customers instead, which Stripe groups in the Dashboard based on the same card, email or phone. That grouping is read-only and it is not an ID you can join on.</li>

<li><strong>The CRM</strong> keys on email in almost every default configuration, and cheerfully creates a second contact when the email differs by one character.</li>

<li><strong>The helpdesk</strong> keys on the requester&#8217;s email, which is whatever the person typed into a contact form or sent the ticket from. It is very often not the address on the order.</li>
</ul>



<p class="wp-block-paragraph">Here&#8217;s the part that stays invisible until someone finally checks: nothing errors. Every dashboard still returns a number. Lifetime value gets silently split across three profiles, so your LTV looks lower and your repeat-purchase rate looks worse than it is. Support metrics look <em>better</em> than they are, because a customer coming back with the same unresolved problem shows up as a brand new contact. You can run like that for a year without a single failed job.</p>



<h3 class="wp-block-heading">Build an identifier table, not a fuzzy match</h3>



<p class="wp-block-paragraph">The fix is boring and it works: a surrogate person key, and a separate table of every identifier ever observed for that person, with the source that reported it.</p>



<pre class="wp-block-code"><code>-- One row per real human, generated by you, owned by you.
CREATE TABLE person (
  person_key      uuid PRIMARY KEY,
  created_at      timestamptz NOT NULL DEFAULT now(),
  merged_into     uuid NULL REFERENCES person(person_key)
);

-- One row per identifier per source. This is the whole trick.
CREATE TABLE person_identifier (
  person_key      uuid NOT NULL REFERENCES person(person_key),
  source          text NOT NULL,   -- shopify | stripe | crm | helpdesk
  id_type         text NOT NULL,   -- email | phone | shopify_customer_id | stripe_customer_id
  id_value        text NOT NULL,   -- already normalized
  first_seen_at   timestamptz NOT NULL,
  last_seen_at    timestamptz NOT NULL,
  PRIMARY KEY (source, id_type, id_value)
);</code></pre>



<p class="wp-block-paragraph">Normalization happens before the row is written, not in the join. Lowercase and trim emails, store phone numbers in E.164, strip whitespace from external IDs. Resist the urge to get clever with Gmail&#8217;s dot and plus handling. Stripping <code>+tags</code> is a business policy decision with real consequences, not a technical detail, and the same rule applied to a domain that treats plus addresses as distinct mailboxes will merge two genuinely different people.</p>



<p class="wp-block-paragraph">Deterministic matching first, always. Exact matches on shared identifiers, transitively closed. Probabilistic matching on name plus address plus device is worth reaching for only when you have exhausted the deterministic keys, and it has an asymmetric failure mode: a household sharing one card gets collapsed into one person, and unmerging afterwards is expensive because every downstream system has already copied the merged key into its own records. Merges are cheap to make and painful to undo. Bias toward leaving two records separate.</p>



<p class="wp-block-paragraph">One thing worth doing early on the Stripe side: when you create a Customer, write your own identifier into <code>metadata</code>. Stripe allows fifty key-value pairs per object, with keys up to forty characters and values up to five hundred, and it only returns metadata for requests made with a secret key. That&#8217;s enough room for a person key and an order reference, which turns a future fuzzy join into an exact one. Never put anything sensitive there.</p>



<h2 class="wp-block-heading">Failure family two: webhooks arrive twice, out of order, or not at all</h2>



<p class="wp-block-paragraph">Both platforms are explicit about this, and both get treated as if they weren&#8217;t.</p>



<p class="wp-block-paragraph">Stripe delivers at least once, does not guarantee ordering, and retries failed deliveries with exponential backoff for up to three days. The event <code>id</code> (the one starting <code>evt_</code>) stays identical across every retry, which makes it your deduplication key. Shopify sends <code>X-Shopify-Webhook-Id</code> for the same purpose, signs the raw body into <code>X-Shopify-Hmac-Sha256</code>, and states plainly that ordering isn&#8217;t guaranteed within a topic or across topics for the same resource. Shopify&#8217;s own documentation goes further and tells you not to rely on webhooks as your only source of data.</p>



<p class="wp-block-paragraph">For a customer-data pipeline specifically, out-of-order delivery is nastier than duplication. Consider a <code>customers/update</code> that changes someone&#8217;s email arriving before the <code>orders/create</code> that was placed under the old address. Process those in arrival order and you attach the order to a person who doesn&#8217;t exist yet, or worse, to the wrong one. Order your writes by the event&#8217;s own timestamp, using <code>X-Shopify-Triggered-At</code> or the <code>updated_at</code> in the payload, and make profile updates last-writer-wins on that timestamp rather than on insertion time.</p>



<h3 class="wp-block-heading">Do almost nothing in the handler</h3>



<p class="wp-block-paragraph">Verify the signature, claim the event ID, write the raw body to durable storage, return 2xx. That&#8217;s it. Identity resolution, CRM lookups and enrichment calls all belong in a worker reading from that store. A handler that tries to resolve identity inline will eventually be slow enough to time out, which triggers a retry, which arrives while the first attempt is still running.</p>



<p class="wp-block-paragraph">Make the claim atomic rather than check-then-act, because two retries genuinely can land concurrently:</p>



<pre class="wp-block-code"><code>-- Atomic claim. If this inserts zero rows, someone already has it.
INSERT INTO webhook_event (source, event_id, topic, received_at, payload)
VALUES ($1, $2, $3, now(), $4)
ON CONFLICT (source, event_id) DO NOTHING
RETURNING event_id;</code></pre>



<p class="wp-block-paragraph">Return a 2xx for a duplicate. A duplicate is a success from the sender&#8217;s point of view, and returning an error just schedules more of them.</p>



<p class="wp-block-paragraph">Signature verification has one recurring trap on the Shopify side: it runs against the <em>raw</em> body. Any JSON body parser that runs first will change the bytes and every signature will fail. Capture the raw body before parsing, and compare in constant time.</p>



<pre class="wp-block-code"><code>import base64, hashlib, hmac

def verify_shopify(raw_body: bytes, header_value: str, client_secret: str) -&gt; bool:
    digest = hmac.new(
        client_secret.encode("utf-8"),
        raw_body,
        hashlib.sha256,
    ).digest()
    expected = base64.b64encode(digest).decode("utf-8")
    return hmac.compare_digest(expected, header_value)</code></pre>



<p class="wp-block-paragraph">Worth knowing before you spend an afternoon on it: after rotating an app&#8217;s client secret, Shopify notes it can take up to an hour before signatures are generated with the new one. If verification starts failing right after a rotation, that&#8217;s usually why.</p>



<p class="wp-block-paragraph">And because delivery isn&#8217;t guaranteed, webhooks alone are never enough. Run a reconciliation job that pulls orders and customers modified since the last watermark and fills the gaps. This is the job that saves you during an outage on your side, and it&#8217;s the one people skip. If you&#8217;d rather not build the buffering and replay layer, managed webhook infrastructure like Hookdeck exists for exactly this, and self-hosting a small queue and worker on a modest VPS from somewhere like Contabo or InterServer is perfectly reasonable too. The point is that something has to catch what the webhook missed.</p>



<h2 class="wp-block-heading">Failure family three: &#8220;revenue&#8221; means four different things</h2>



<p class="wp-block-paragraph">This is the one that turns into a meeting. Finance says the number is wrong. Analytics says the number is right. Both are correct, because they&#8217;re measuring different things and nobody wrote down which.</p>



<ul class="wp-block-list">
<li>Shopify reports gross sales at the moment the order is placed, in the store&#8217;s currency and the store&#8217;s timezone.</li>

<li>Stripe pays out in batches, net of processing fees, refunds and disputes, on its own schedule, in the settlement currency of the account.</li>

<li>A refund issued in Shopify and a refund issued in Stripe are two different events that may or may not be the same underlying money movement.</li>

<li>A dispute lands days or weeks after the order and reverses revenue in a period that has already been reported.</li>
</ul>



<p class="wp-block-paragraph">There&#8217;s a structural trap here too. If the store runs Shopify Payments, there is no Stripe account for you to query. Shopify Payments is built on Stripe&#8217;s processing infrastructure, but it&#8217;s Shopify&#8217;s gateway, and payouts, fees and disputes live in the Shopify admin rather than behind a Stripe API key. Plenty of businesses end up with Shopify Payments handling storefront orders and a separate Stripe account handling subscriptions or invoicing, which means two payment systems, two payout schedules and one customer who has no idea any of this exists.</p>



<p class="wp-block-paragraph">Do not build a single <code>revenue</code> column. Build separate fact tables at their natural grain (orders, order line items, payments, refunds, disputes, payouts) plus a bridging table that links a payout back to the transactions inside it. Then let each team define its own metric on top. Finance gets net settled cash. Marketing gets gross booked revenue. Neither has to be wrong for the other to be right.</p>



<p class="wp-block-paragraph">Store every timestamp in UTC and keep the original timezone as a separate column. Store every amount in minor units as an integer alongside its currency code, and store the FX rate that was actually applied rather than recalculating it later from a rates table. Recalculated FX is the single most common source of &#8220;the number moved and nobody changed anything.&#8221;</p>



<p class="wp-block-paragraph">On ingestion, Stripe Data Pipeline can push Stripe data directly into Snowflake, Redshift or Databricks as a data share, which removes a connector from the picture if you&#8217;re already on one of those. Otherwise a managed ELT tool like Fivetran or a self-hosted Airbyte will handle Shopify, Stripe, the CRM and the helpdesk with off-the-shelf connectors. Both routes are fine. The connector is genuinely the easy part.</p>



<h2 class="wp-block-heading">Failure family four: deletion is the requirement nobody models</h2>



<p class="wp-block-paragraph">If you&#8217;re building this as a Shopify app, three compliance webhooks are mandatory: <code>customers/data_request</code>, <code>customers/redact</code> and <code>shop/redact</code>. You acknowledge with a 200 and complete the action within thirty days. <code>shop/redact</code> fires forty-eight hours after uninstall and covers the shop&#8217;s data. These are enforced at app review, so you&#8217;ll find out about them either way.</p>



<p class="wp-block-paragraph">The interesting problem isn&#8217;t receiving the webhook. It&#8217;s that by the time it arrives, that person&#8217;s email exists in your raw landing zone, your staging tables, your identity table, your customer view, a BI extract, the CRM, the helpdesk, and a Slack message from a debugging session six months ago. A delete that only touches the golden record is not a delete.</p>



<p class="wp-block-paragraph">The design decision that makes this tractable is to hold personal data in exactly one place. Every downstream table carries <code>person_key</code> and nothing else. Facts, aggregates, models, extracts and syncs all reference the key. When a redaction request lands, you redact one row set in the identifier table and the rest of the warehouse is already anonymous. Getting this right at schema-design time costs an afternoon; retrofitting it costs a quarter.</p>



<p class="wp-block-paragraph">Practical pieces that make it real:</p>



<ul class="wp-block-list">
<li>Tag every table and column that can contain an identifier, in your catalog or in dbt metadata, and treat the tag list as the deletion scope.</li>

<li>Partition the raw landing zone so you can rewrite a partition rather than scan an entire bucket looking for one email.</li>

<li>Keep a deletion queue with a status per downstream system, including the CRM and the helpdesk, which have their own APIs and their own idea of what deletion means.</li>

<li>Write down what you retain and why. Financial records generally have to survive a deletion request for tax and dispute reasons. Say so rather than promising a hard delete you can&#8217;t perform.</li>
</ul>



<h2 class="wp-block-heading">Failure family five: pushing the unified view back out</h2>



<p class="wp-block-paragraph">A customer view nobody can act on is a report. The value shows up when a support agent opens a ticket and can see subscription status and lifetime order count without switching tabs.</p>



<p class="wp-block-paragraph">That&#8217;s reverse ETL, and the market is well served: Hightouch, Census (now part of Fivetran), Polytomic, RudderStack, and iPaaS-style tools like Workato if you need branching logic rather than field syncs. The warehouse-native argument is straightforward and mostly correct: your models already live there, and you avoid maintaining a second copy of the customer profile inside a vendor&#8217;s rigid schema. The counter-argument is also real. Reverse ETL is batch by nature, so if you need sub-minute reaction to an event, a packaged CDP or a direct event pipeline will serve you better, and you&#8217;ll be running several vendors plus warehouse compute plus the people to maintain the models.</p>



<p class="wp-block-paragraph">Three things go wrong here more or less every time.</p>



<p class="wp-block-paragraph"><strong>The write loop.</strong> You compute a churn-risk field and sync it into the CRM. Your ELT connector reads the CRM back into the warehouse. Now the computed field is an input to the model that computes it. This is genuinely hard to spot because nothing errors, the value just drifts. Namespace every synced field with a consistent prefix and explicitly exclude that namespace from ingestion.</p>



<p class="wp-block-paragraph"><strong>The resync stampede.</strong> Incremental syncs push diffs and stay small. Change a model definition and the next run wants to update every record, which will meet a helpdesk API rate limit at speed. Cap the batch size, and treat a full resync as a planned operation rather than something a merge to main can trigger.</p>



<p class="wp-block-paragraph"><strong>Syncing logic instead of keys.</strong> Push <code>person_key</code> into a custom field on the CRM contact and the helpdesk user. It makes the key searchable by agents, it gives the next pipeline a deterministic join, and it stops three teams from each rebuilding their own version of your matching rules.</p>



<h2 class="wp-block-heading">A pipeline shape that holds up</h2>



<p class="wp-block-paragraph">Put together, the order of operations matters more than the tool choice:</p>



<ol class="wp-block-list">
<li>Land raw. Every webhook body and every API page goes to immutable storage first, keyed by source and event ID, before anything interprets it.</li>

<li>Deduplicate on the provider&#8217;s event ID, atomically, with a retention window longer than the provider&#8217;s retry window.</li>

<li>Normalize identifiers into the identifier table as rows arrive, with the source recorded.</li>

<li>Resolve identity deterministically into <code>person_key</code>, as a scheduled job, never inside a request handler.</li>

<li>Build fact tables per system at their own grain, keyed on <code>person_key</code>, with no raw PII.</li>

<li>Model the customer view on top, in dbt or whatever your team already uses. The Fivetran <code>dbt_customer360</code> package is a reasonable starting point if you want to see how someone else structured the identity step.</li>

<li>Activate a small, named, namespaced set of fields back into the CRM and helpdesk.</li>

<li>Run reconciliation and freshness checks on a schedule, and alert on them like you would any other production job.</li>
</ol>



<h2 class="wp-block-heading">Troubleshooting: symptom to root cause</h2>



<p class="wp-block-paragraph">These are the ones that come up repeatedly, and the check that usually settles them.</p>



<ul class="wp-block-list">
<li><strong>Warehouse customer count is much higher than Shopify&#8217;s.</strong> Guest orders are minting a new person on every purchase. Check how many <code>person</code> rows have exactly one identifier and one order.</li>

<li><strong>Warehouse count is much lower.</strong> An over-eager merge rule. Look for person keys with an implausible number of distinct emails or shipping addresses attached.</li>

<li><strong>BI revenue doesn&#8217;t match Stripe payouts.</strong> Grain mismatch, not a data loss. Reconcile a single payout against its constituent transactions before touching anything else.</li>

<li><strong>A CRM field keeps flipping between two values.</strong> Write loop. Check whether the field&#8217;s namespace is excluded from your ELT source configuration.</li>

<li><strong>Recent orders missing, older ones fine.</strong> Webhook delivery gap. Confirm the reconciliation job ran, and check its watermark rather than trusting its exit code.</li>

<li><strong>Every Shopify signature suddenly fails.</strong> Either a body parser is running before verification, or the client secret was rotated within the last hour.</li>

<li><strong>Tickets attached to the wrong customer after a merge.</strong> The merge happened in the warehouse but the synced key in the helpdesk still points at the old person. Merges need to propagate outward, not just inward.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Joining on email in the model layer instead of resolving identity once, upstream.</li>

<li>Treating webhooks as the system of record instead of as a low-latency hint, with a reconciliation pull behind them.</li>

<li>Doing identity resolution, enrichment or CRM writes inside the webhook handler.</li>

<li>Ordering profile updates by arrival time rather than by the event&#8217;s own timestamp.</li>

<li>One <code>revenue</code> column that finance and marketing are both told to use.</li>

<li>Storing amounts as floats, or recalculating historical FX from a rates table.</li>

<li>Copying PII into every downstream table because it was convenient at the time.</li>

<li>Reaching for probabilistic matching before the deterministic keys have been exhausted.</li>

<li>Assuming a Shopify store on Shopify Payments has a Stripe account you can query.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>Own the surrogate key. Never let a vendor&#8217;s internal ID become the primary identity of a person in your systems.</li>

<li>Write your <code>person_key</code> into Stripe <code>metadata</code> and Shopify customer metafields at creation time, so future joins are exact rather than inferred.</li>

<li>Keep raw payloads immutable and replayable. Every model bug you&#8217;ll ever have is cheaper to fix if you can reprocess from raw.</li>

<li>Version your matching rules and record which version produced each merge, so you can audit and reverse a bad rule.</li>

<li>Alert on freshness and on row-count deltas per source, not just on job success. A connector returning zero rows exits cleanly.</li>

<li>Reconcile one payout end to end by hand before you trust the automated version. It&#8217;s the fastest way to find a grain error.</li>

<li>Document the metric definitions next to the models. Most &#8220;the data is wrong&#8221; tickets are definition disputes.</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Do I need a CDP to unify Shopify, Stripe, CRM and support data?</h3>



<p class="wp-block-paragraph">Not necessarily. If you already run a warehouse and have someone comfortable with dbt, ELT plus modelling plus reverse ETL covers the same ground with more control over the data model. A packaged CDP earns its cost when you need real-time behavioural segmentation, when nobody on the team can maintain models, or when marketing needs to ship changes without waiting on data engineering.</p>



<h3 class="wp-block-heading">How do I handle guest checkout customers?</h3>



<p class="wp-block-paragraph">Create a person from the order&#8217;s email anyway, and mark the record as identity-weak. When that email later appears attached to a real Shopify customer or Stripe Customer object, the deterministic match links them and the order history follows. The mistake is discarding guest orders because they have no customer ID, which is exactly the revenue you most want attributed.</p>



<h3 class="wp-block-heading">Should I sync everything into the CRM or keep it in the warehouse?</h3>



<p class="wp-block-paragraph">Keep the full picture in the warehouse and sync a deliberately small set of fields outward. The test is whether an agent or a rep will act on the field during a conversation. Lifetime value, subscription status, open ticket count and churn risk pass. A hundred-column profile does not, and it turns every model change into a rate-limit incident.</p>



<h3 class="wp-block-heading">Why don&#8217;t my Shopify sales match my Stripe payouts?</h3>



<p class="wp-block-paragraph">Because they&#8217;re measuring different things at different times. Shopify shows gross sales at order time; Stripe pays out batched amounts net of fees, refunds and disputes, and the timing differs by days. Add currency conversion and a timezone mismatch and the numbers will never tie without an explicit bridging table linking payouts to the transactions inside them.</p>



<h3 class="wp-block-heading">Can I use email as the primary key if my store requires account creation?</h3>



<p class="wp-block-paragraph">You can get away with it longer, but it&#8217;s still a bad primary key. People change email addresses, use one address for orders and another for support, and share addresses within a household. A surrogate key with an identifier table costs very little up front and means an email change is a new row rather than a broken join and a duplicate customer.</p>



<h3 class="wp-block-heading">How do I test the identity resolution before trusting it?</h3>



<p class="wp-block-paragraph">Run it in shadow mode. Compute person keys without activating them, then have support pick twenty customers they know are duplicates and twenty they know are distinct, and check what the resolver did. That exercise finds normalization bugs and over-eager merge rules faster than any aggregate metric, because the people who answer the tickets already know where the duplicates are.</p>



<h3 class="wp-block-heading">What&#8217;s the minimum viable version of this?</h3>



<p class="wp-block-paragraph">Raw landing with deduplication, an identifier table with deterministic matching, one fact table per source, and a single synced field into the helpdesk. That&#8217;s a couple of weeks of work and it removes most of the pain. Everything above it is refinement.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Conclusion: unify Shopify, Stripe, CRM and support data around a key you own</h2>



<p class="wp-block-paragraph">If you take one thing from this: the hard part of unifying Shopify, Stripe, CRM and support data is not moving the data. Connectors are commodity. The hard part is deciding what a customer <em>is</em>, writing that decision down as a key you own, and then refusing to let any downstream system quietly invent its own version of it.</p>



<p class="wp-block-paragraph">Do the identity work first, land everything raw so you can reprocess when you get it wrong, and keep personal data in one place so deletion stays a single operation. The dashboards will follow. Skip that order and you get exactly the screenshot this post opened with, eighteen months later and with a lot more rows behind it.</p>



<h2 class="wp-block-heading">Need help wiring this together?</h2>



<p class="wp-block-paragraph">I work on exactly this kind of plumbing: the unglamorous layer between an ecommerce platform, a payment processor and the tools your team actually opens all day. Things I can help with here:</p>



<ul class="wp-block-list">
<li>Designing the identity model (surrogate keys, identifier tables, deterministic matching rules) and auditing an existing one for over-merging.</li>

<li>Building idempotent, signature-verified webhook intake for Shopify and Stripe, with a dead-letter path and a reconciliation job behind it.</li>

<li>Setting up ELT into Snowflake, BigQuery, Redshift or Postgres, whether with Fivetran, self-hosted Airbyte, or Stripe Data Pipeline where it fits.</li>

<li>Modelling orders, payments, refunds, disputes and payouts so finance and marketing can each get a number they trust.</li>

<li>Reverse ETL back into the CRM and helpdesk without write loops, rate-limit incidents or a hundred junk fields.</li>

<li>Making deletion and data-subject requests actually work across the warehouse, the raw landing zone and every downstream copy.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;re stuck on a specific piece, send me the concrete thing: a webhook payload, a schema dump, the query that disagrees with your Stripe payout report. It&#8217;s much easier to say something useful about a real artifact than about a diagram.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/unify-shopify-stripe-crm-support-data/">One Customer, Four Systems: Unifying Shopify, Stripe, CRM and Support Data</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>MediaConvert Cost Per Minute: How the Bill Is Built and How Jobs Fail</title>
		<link>https://john-nessime.com/blog/technical-guides/mediaconvert-cost-per-minute/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 29 Aug 2026 09:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Adaptive Bitrate]]></category>
		<category><![CDATA[Amazon CloudFront]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Error Handling]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[MediaConvert]]></category>
		<category><![CDATA[Normalized Minutes]]></category>
		<category><![CDATA[QVBR]]></category>
		<category><![CDATA[Reserved Transcode Slots]]></category>
		<category><![CDATA[Video Transcoding]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=542</guid>

					<description><![CDATA[<p>One checkbox can reprice every output in a MediaConvert job. A working breakdown of how the per-minute charge is assembled from normalized minutes and per-output billing, why failed jobs are free but still expensive, and how to build retry and recovery that classifies before it retries.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/mediaconvert-cost-per-minute/">MediaConvert Cost Per Minute: How the Bill Is Built and How Jobs Fail</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A turnaround complaint lands: the promo team&#8217;s masters take too long to appear in the CMS. Someone opens the job template, finds a setting called accelerated transcoding, ticks it, and moves on. Turnaround improves. Nobody mentions it again.</p>



<p class="wp-block-paragraph">The invoice is where it surfaces, and not only on the big outputs. The 240p ladder rung went up. The audio-only rendition went up. Every output in every job that template touches went up, including the ones that never needed the feature.</p>



<p class="wp-block-paragraph">That is not a billing bug. It is the documented rule working as designed, and it is the most common reason a bill moves without anyone changing how much video goes through it. Understanding <strong>MediaConvert cost per minute</strong> starts with accepting that the unit you pay for is not a job, not a source file, and not a minute of compute.</p>



<p class="wp-block-paragraph">What follows is how that charge is assembled, what happens to it when a job fails, and how to build recovery around AWS Elemental MediaConvert. Rates are deliberately absent; AWS changes them and they vary by region. The mechanism is what catches people.</p>



<h2 class="wp-block-heading">How MediaConvert cost per minute is actually assembled</h2>



<p class="wp-block-paragraph">Three rules stack on top of each other. Miss one and your cost model is wrong by a multiple, not a rounding error.</p>



<h3 class="wp-block-heading">You pay per output, not per job</h3>



<p class="wp-block-paragraph">An eight-rung ABR ladder plus an audio-only rendition is nine billable items, each charged for its own duration. AWS bills the duration of the finished assets, not the transcoding time, in hundredth-of-a-minute increments. A ten-minute source producing nine outputs bills as ninety minutes of output.</p>



<p class="wp-block-paragraph">Ladder design is therefore a cost decision, not only a quality one. The bottom rungs are not cheap because they are small. They cost a full pass of the source duration at whatever multiplier their settings attract, whether anyone streams them or not.</p>



<h3 class="wp-block-heading">Normalized minutes, and why multipliers compound</h3>



<p class="wp-block-paragraph">There is no flat rate per output minute. MediaConvert converts each output into normalized minutes, a unit that scales with the work the encode demanded. Resolution, frame rate, codec and number of encoding passes each carry a multiplier, and those multipliers combine. Single-pass to multi-pass is one. 30fps to 60fps is another. Do both to a UHD rung and you have stacked three.</p>



<p class="wp-block-paragraph">Volume discounts then apply to your accumulated normalized minutes for the month, per region, with basic-tier and professional-tier usage accruing separately toward their own thresholds. Two consequences: splitting a workload across regions splits the accrual and can leave you paying a higher effective rate in both, and under consolidated billing in AWS Organizations usage accrues across accounts, so a per-account model understates the discount you actually get.</p>



<h3 class="wp-block-heading">The job-wide setting that reprices everything</h3>



<p class="wp-block-paragraph">Back to the opening. Each output is assessed for its billing tier separately, but only when the job has no job-wide professional-tier feature enabled. Turn one on, accelerated transcoding being the usual culprit, and every output in that job bills at pro-tier rates. An SD rung that would have billed at basic tier gets repriced because a different output in the same job needed to finish faster.</p>



<p class="wp-block-paragraph">If a subset of content genuinely needs acceleration, split it out. Long, visually complex assets get a template with the feature; routine web renditions get one without. Two templates is less tidy than one, and considerably cheaper than a single checkbox repricing your catalogue.</p>



<h2 class="wp-block-heading">Failed jobs are free, which changes how you should retry</h2>



<p class="wp-block-paragraph">MediaConvert charges only when a job reaches COMPLETED. Jobs ending in ERROR are not billed. That sounds like it removes the urgency from failure handling. It does the opposite.</p>



<p class="wp-block-paragraph">A job that fails on output seven of nine may still have written outputs one through six to S3. Those partial artefacts cost storage, and if a naive retry writes to the same prefix you get a mix of old and new segments that a player will serve quite happily until someone notices the ladder is inconsistent. Free at the transcode line, expensive everywhere downstream.</p>



<p class="wp-block-paragraph">&#8220;Not billed&#8221; also makes blind retries feel harmless, so people write them. A loop against a deterministic failure, a corrupt source or a settings mismatch, will never succeed. It burns concurrency paying jobs need and produces identical alerts until the team ignores the channel. The service already retries some conditions itself and exposes a retry count, so read that field before adding an attempt of your own.</p>



<h2 class="wp-block-heading">The queue model is a throughput control, not a cost control</h2>



<p class="wp-block-paragraph">On-demand queues bill per output minute. How many jobs run at once is governed by a service quota for concurrent jobs per account in the region, a quota per individual queue, and a concurrent-jobs setting on each queue. Creating a second queue does not conjure capacity out of the account; you raise it through Service Quotas. Jobs sitting in SUBMITTED rather than moving to PROGRESSING is the signal you have hit the ceiling.</p>



<p class="wp-block-paragraph">Reserved queues work on a different axis. You buy reserved transcode slots against a twelve-month commitment and pay for that capacity regardless of use. Each slot has its own dedicated compute, so one slot runs a given job just as fast as ten would. More slots means more jobs in parallel, not faster jobs. Three things about the commitment deserve a second read before anyone clicks agree:</p>



<ul class="wp-block-list">
<li>You cannot cancel it, and you cannot decrease the slot count.</li>

<li>Adding capacity does not extend the existing term. It starts a fresh twelve-month commitment for the larger count, from the day you add it.</li>

<li>When the plan expires, the queue persists and keeps accepting jobs. MediaConvert just does not run them.</li>
</ul>



<p class="wp-block-paragraph">That last one is the invisible failure in this whole topic. Your submission code gets a job ID back. The API call succeeded. Nothing errors. The jobs simply never progress, and unless you alarm on queue standby time or on the gap between jobs submitted and completed, the first person to notice is whoever asks why the library stopped updating. Set auto-renew deliberately, either way, and diary the term end date.</p>



<p class="wp-block-paragraph">Queue hopping is the pressure valve. Give a job a wait time and a destination queue, and if it is still waiting when the timer expires it moves. The usual shape is reserved to on-demand during a spike. Billing follows the queue the job actually runs in, so a hopped job becomes on-demand spend.</p>



<pre class="wp-block-code"><code>{
  "Settings": { "OutputGroups": [], "Inputs": [] },
  "HopDestinations": [
    {
      "WaitMinutes": 10,
      "Queue": "arn:aws:mediaconvert:us-west-2:111122223333:queues/ondemandqueue",
      "Priority": 25
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">WaitMinutes accepts 1 to 4320. Priority runs from -50 to 50 and defaults to 0, and the optional priority under HopDestinations lets a hopped job land higher or lower in the destination queue than it sat in the original. Hopping is a cost decision as much as a throughput one: it converts capacity you have already paid for into incremental on-demand spend.</p>



<h2 class="wp-block-heading">The failure families that actually bite</h2>



<p class="wp-block-paragraph">MediaConvert publishes numbered error codes with documented causes. Rather than reproduce the table, group them by what you have to do about them, because the correct response differs sharply between groups.</p>



<h3 class="wp-block-heading">Access failures</h3>



<p class="wp-block-paragraph">The service could not open an input or output location. Causes cluster around IAM permissions on the role the job runs under, cross-account bucket policies when the source lives elsewhere, keys that do not exist, and files whose headers cannot be read. Retry these once with a delay, because uploads still in flight are a real condition. If the second attempt fails identically, it is configuration, and no amount of retrying fixes configuration.</p>



<h3 class="wp-block-heading">Settings failures that only appear on some inputs</h3>



<p class="wp-block-paragraph">The nastiest to diagnose, because the template has been fine for the last thousand assets. An encoder initialisation failure on one rung, triggered by an unusual source aspect ratio or frame rate meeting a fixed constraint in the template, is the classic shape. Never retry these; the job fails identically forever. The fix is usually to loosen the constraint rather than pin it harder, because letting the service derive a value the source cannot satisfy removes the whole class of failure.</p>



<h3 class="wp-block-heading">Warnings, which are not errors and are easy to miss</h3>



<p class="wp-block-paragraph">MediaConvert also returns warning codes for problems that do not stop the job. It completes, you get billed, and something about the output is wrong. Video padding inserted to keep audio and video aligned, black frames carried through from the source, a QVBR score well below the rest of the catalogue: none raise an error, all are visible in metrics. Alert only on ERROR and this entire family is invisible to you.</p>



<h3 class="wp-block-heading">Pinned engine versions that quietly expire</h3>



<p class="wp-block-paragraph">MediaConvert groups its releases into job engine versions so you can hold production on one while validating another. Access is requested through AWS Support rather than open to every account, so check whether it applies to you first. The behaviour worth knowing is expiry: requesting an expired version does not fail. The job runs as though no version had been specified, and the field reporting the version used comes back empty. An invalid version, by contrast, returns a 400 and creates no job. The failure mode of pinning is silence.</p>



<h2 class="wp-block-heading">Building failure recovery that does not cost you twice</h2>



<p class="wp-block-paragraph">The pattern that holds up is event-driven, classified and idempotent. In order:</p>



<ol class="wp-block-list">
<li>Subscribe to job state changes in EventBridge rather than polling. The ERROR event carries the code and message in its detail block.</li>

<li>Route by error code, not by status: one target for retryable access conditions, another for deterministic settings failures that need a human, and a suppression path for anything already seen on that asset.</li>

<li>Write a retry&#8217;s outputs to a new prefix, or clear the old one first. Never let attempt two land beside attempt one&#8217;s partial results.</li>

<li>Send a client request token on job creation. Reusing the same token within a minute of a successful request returns the original job instead of creating a duplicate, which protects you when your orchestration retries a create call whose response it never saw.</li>

<li>Cap attempts and land the remainder in a dead letter queue with the original event attached. A failure nobody can see is a failure nobody fixes.</li>

<li>Carry your own identifiers through user metadata on the job, so the returning event joins back to the asset without a lookup table.</li>
</ol>



<pre class="wp-block-code"><code>{
  "source": ["aws.mediaconvert"],
  "detail-type": ["MediaConvert Job State Change"],
  "detail": {
    "status": ["ERROR"]
  }
}</code></pre>



<p class="wp-block-paragraph">The matching event delivers a detail object containing the queue ARN, job ID, status, an errorCode and an errorMessage. Route on errorCode. That one decision is the difference between a recovery system and a retry loop.</p>



<h2 class="wp-block-heading">Measuring what you spend before the invoice tells you</h2>



<p class="wp-block-paragraph">MediaConvert sends metrics to CloudWatch at the end of every job, several mapping directly onto the billing dimensions. The ones I reach for first:</p>



<ul class="wp-block-list">
<li><code>SDOutputDuration</code>, <code>HDOutputDuration</code>, <code>UHDOutputDuration</code>, <code>8KOutputDuration</code> and <code>AudioOutputDuration</code>, in milliseconds per queue. Summed over a month, that is billable output volume by resolution class, days ahead of Cost Explorer.</li>

<li><code>JobErrorsByCode</code>, counting jobs that failed with a particular code. Build the dashboard on this one; it shows which failure family is growing.</li>

<li><code>StandbyTime</code>, how long a job waits before processing starts. Alarm on it and you catch backlogs, and you catch an expired reserved plan.</li>

<li><code>JobsSubmittedCount</code> against <code>JobsCompletedCount</code>. A widening gap is the earliest honest signal something upstream is broken.</li>

<li><code>QVBRAvgQualityLowBitrate</code> and <code>QVBRMinQualityHighBitrate</code>, for quality regressions that never raise an error.</li>
</ul>



<p class="wp-block-paragraph">For attribution rather than volume, enable cost allocation tags and set the billing tags source on your jobs so per-output costs land against a tag you chose. Pushing that into a wider FinOps view through Vantage, CloudZero or a Grafana Cloud dashboard fed from CloudWatch gets you per-title cost without building the aggregation yourself. Either way, these metrics let you model MediaConvert cost per minute continuously instead of discovering it monthly.</p>



<p class="wp-block-paragraph">Transcoding is only part of the line. Storage for sources and renditions in S3, and egress through Amazon CloudFront or whichever CDN fronts delivery, are separate charges and often the larger ones. A cost model that stops at the encoder will be wrong.</p>



<h2 class="wp-block-heading">Troubleshooting a failed job</h2>



<ol class="wp-block-list">
<li>Get the error code first. Everything else follows from which family it belongs to.</li>

<li>Ask whether other assets in the same bucket succeeded in the same window. Yes points at the file, no points at permissions or configuration.</li>

<li>For a settings failure, bisect the job. Strip it to a single video output, confirm it runs, then add outputs and features back one at a time. Tedious, and faster than reasoning about it.</li>

<li>Check warnings on jobs that completed nearby. Warnings often precede errors from the same root cause.</li>

<li>If nothing ran at all, look at the queue before the job: plan status on a reserved queue, standby time on an on-demand one.</li>
</ol>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Modelling cost from source duration instead of total output duration across the ladder.</li>

<li>Enabling a job-wide professional-tier feature in a shared template and repricing every output in it.</li>

<li>Creating extra on-demand queues for throughput when the account quota is the real ceiling.</li>

<li>Buying reserved slots off a peak-week estimate and paying for idle capacity the other fifty-one.</li>

<li>Retrying every failure identically, including the deterministic ones that cannot succeed.</li>

<li>Alerting on ERROR only, and never looking at warnings or quality scores.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>Separate templates by cost profile rather than content type. Anything with a job-wide pro-tier feature gets its own.</li>

<li>Audit the ladder against playback data at least once. Rungs nobody streams are pure cost.</li>

<li>Keep a workload in one region so volume discounts accrue against a single pool.</li>

<li>Run a representative job on an on-demand queue with the reserved simulation setting enabled before committing to slots.</li>

<li>Use job priority instead of extra queues when the goal is for urgent work to jump ahead.</li>

<li>Derive client request tokens from the asset, not from a timestamp, so retries are genuinely idempotent.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I pay for MediaConvert jobs that fail?</h3>



<p class="wp-block-paragraph">No. Billing applies only to jobs reaching COMPLETED status, so a job ending in ERROR is not charged. You may still be paying for partial outputs that reached S3 before the failure, and the storage they occupy until something cleans them up.</p>



<h3 class="wp-block-heading">What is a normalized minute?</h3>



<p class="wp-block-paragraph">The billing unit MediaConvert uses so a minute of demanding output counts for more than a minute of simple output. Resolution, frame rate, codec and encoding passes each contribute a multiplier, and those combine into the normalized total your volume discount tier is measured against.</p>



<h3 class="wp-block-heading">Is a reserved queue cheaper than on demand?</h3>



<p class="wp-block-paragraph">Only if utilisation is genuinely steady. You pay for reserved transcode slots regardless of use, across a twelve-month commitment you cannot cancel or shrink. Bursty catalogues usually do better on demand, where volume discounts apply automatically as monthly usage climbs. Reserved pricing also excludes some features, so confirm everything your templates use is available there before committing.</p>



<h3 class="wp-block-heading">Why are my jobs stuck in SUBMITTED?</h3>



<p class="wp-block-paragraph">On an on-demand queue, it is already running its maximum number of concurrent jobs; raise the quota through Service Quotas rather than adding queues. On a reserved queue, check whether the pricing plan has expired, because an expired queue keeps accepting jobs and never processes them.</p>



<h3 class="wp-block-heading">Does adding queues give me more transcoding capacity?</h3>



<p class="wp-block-paragraph">No. Concurrency is bounded by account-level and queue-level quotas in the region. Extra queues isolate workloads from each other, for instance keeping one client&#8217;s jobs from starving another&#8217;s. Capacity increases come from Service Quotas.</p>



<h3 class="wp-block-heading">How do I stop one setting from repricing my whole job?</h3>



<p class="wp-block-paragraph">Split the job. Job-wide professional-tier features apply their tier to every output, so isolate the assets that need them into their own template and keep routine renditions in a template with no job-wide pro-tier features enabled.</p>



<h3 class="wp-block-heading">Should I run FFmpeg on my own servers instead?</h3>



<p class="wp-block-paragraph">Sometimes. Predictable, modest volume on dedicated hardware from a provider like Contabo or InterServer can undercut per-minute pricing, and you get exact control over encoder flags. What you take on is capacity planning, codec licensing questions, and building the retry and observability machinery the managed service hands you for free. That trade favours MediaConvert for spiky workloads and broadcast-grade features, and self-hosting for steady, simple ladders where an engineer already owns the pipeline.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">MediaConvert cost per minute is not a rate you look up. It is a product: output count, times duration, times a stack of feature multipliers, with a single job-wide setting able to reprice the lot. Get that model into whatever you plan from and the bill stops being a monthly surprise.</p>



<p class="wp-block-paragraph">On the failure side, the codes are documented and the events are structured, so there is no excuse for a retry loop that cannot tell a permissions problem from a settings problem. Classify before you retry. And put an alarm on queue standby time today, because the failure that costs you most is the one where nothing errors at all.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your transcoding pipeline?</h2>



<p class="wp-block-paragraph">I work with teams running video-on-demand pipelines on AWS, usually at the point where the bill or the failure rate has become someone&#8217;s problem. What I actually help with here:</p>



<ul class="wp-block-list">
<li>Auditing job templates and ABR ladders for outputs and multipliers costing more than they return.</li>

<li>Building a cost model from your CloudWatch output-duration metrics so you can forecast rather than react.</li>

<li>Designing EventBridge-driven failure handling that classifies by error code and retries only what can succeed.</li>

<li>Working the reserved versus on-demand decision against your real utilisation, including whether queue hopping earns its keep.</li>

<li>Debugging settings failures that only hit certain sources, and loosening the constraints causing them.</li>

<li>Wiring transcoding, S3 lifecycle and CDN delivery into one cost and reliability picture instead of three.</li>
</ul>



<p class="wp-block-paragraph">Send me a job template, an error code, or a month of output-duration metrics and I will tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/mediaconvert-cost-per-minute/">MediaConvert Cost Per Minute: How the Bill Is Built and How Jobs Fail</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Backfilling Historical API Data into S3 Without Silent Gaps</title>
		<link>https://john-nessime.com/blog/devops/backfill-historical-api-data-into-s3/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 24 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Backfill]]></category>
		<category><![CDATA[Boto3]]></category>
		<category><![CDATA[Conditional Requests]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Cursor Pagination]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Exponential Backoff]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Parquet]]></category>
		<category><![CDATA[Partition Projection]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[S3 Inventory]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=268</guid>

					<description><![CDATA[<p>A backfill that exits zero can still be missing a week of data, and nothing will tell you. This is a practical guide to the failure families behind silent gaps: pagination drift under a mutating source, retries that duplicate pages, prefix layouts designed for writes instead of reads, the seam where backfill meets live ingest, and the storage class rules that make mistakes expensive. Includes deterministic key derivation, S3 conditional writes, Athena partition projection, and a per-window manifest pattern that turns completeness into something you can query.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/backfill-historical-api-data-into-s3/">Backfilling Historical API Data into S3 Without Silent Gaps</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The backfill exits zero. Every window processed, nothing red in the logs, the object count in the bucket looks about right for the range. You close the ticket. Weeks later an analyst asks why the chart dips for one week in the middle of the historical range, and nobody can answer, because as far as every system involved is concerned, nothing failed.</p>



<p class="wp-block-paragraph">That&#8217;s the shape of the problem. When you backfill historical API data into S3, the failure to worry about isn&#8217;t the one that throws. It&#8217;s the one that writes fewer records than it should and reports success anyway. Bytes transferred look plausible. Nothing goes red. The data is quietly wrong, and you find out when someone downstream trusts it.</p>



<p class="wp-block-paragraph">This post covers the failure families that produce that outcome and the design decisions that make them impossible rather than merely unlikely: pagination that drifts under a mutating source, retries that write the same page twice under different keys, a prefix layout chosen while thinking about writes instead of reads, the seam where the backfill meets the live pipeline, and the storage class trap that makes a cheap-looking archive expensive to fix.</p>



<h2 class="wp-block-heading">Why a backfill that finishes cleanly can still be wrong</h2>



<p class="wp-block-paragraph">A live pipeline has a natural error signal. Data stops arriving, a lag metric climbs, someone notices. A backfill has none of that. It runs once, against a range nobody is watching in real time, and then it&#8217;s over. The only evidence it leaves behind is a set of objects in a bucket.</p>



<p class="wp-block-paragraph">So the question you need to answer months later isn&#8217;t &#8220;did the job succeed?&#8221; It&#8217;s &#8220;is this window complete, and how do I know?&#8221; If your design can&#8217;t answer that from artifacts on disk, you don&#8217;t have a backfill. You have a large amount of data of unknown provenance.</p>



<h2 class="wp-block-heading">Failure family 1: the source moved while you paginated</h2>



<p class="wp-block-paragraph">This one bites hardest and shows up least. Most REST APIs paginate historical queries with an offset or page number. You ask for page 1, page 2, page 3. Meanwhile the underlying table is still accepting writes.</p>



<p class="wp-block-paragraph">If a record is inserted into the middle of the result set between your page 4 and page 5 requests, everything after it shifts down by one. The row that would have started page 5 is now the last row of page 4, which you already fetched. You request page 5 and that record is gone. Not errored. Gone. Insertions cause skips, deletions cause duplicates, and the job still exits zero.</p>



<h3 class="wp-block-heading">What to do instead</h3>



<ul class="wp-block-list"><li><strong>Prefer keyset or cursor pagination.</strong> If the API offers an opaque cursor or a &#8220;records after ID X&#8221; parameter, use it. The read position is anchored to a value rather than a count, so inserts elsewhere in the set can&#8217;t shift it.</li><li><strong>Sort by an immutable column.</strong> If you&#8217;re stuck with offsets, sort by something that never changes after insert, usually the primary key or the creation timestamp. Sorting by a last-modified column while records are being modified is the worst case: rows physically move within the ordering as you walk it.</li><li><strong>Bound every request on both ends.</strong> Don&#8217;t ask for &#8220;everything before now.&#8221; Ask for a closed interval with an upper bound already in the past. A closed window can&#8217;t grow while you read it.</li><li><strong>Know which timestamp the filter applies to.</strong> Filtering on created-at gives you records that came into existence in that window. Filtering on updated-at gives you records touched in it, which for a mutable source is a different set entirely.</li></ul>



<p class="wp-block-paragraph">That last point causes arguments. Backfill on created-at, then run incremental syncs on updated-at, and your historical range never picks up edits made to old records. That may be correct for your use case. It is not correct by default, and it&#8217;s worth writing down which one you chose.</p>



<h2 class="wp-block-heading">Failure family 2: the retry that wrote the same page twice</h2>



<p class="wp-block-paragraph">Backfills retry. They have to, because you&#8217;re hammering an API for hours and something will time out. The problem is what the retry writes.</p>



<p class="wp-block-paragraph">If your object key contains a UUID, a run identifier or a wall-clock timestamp, a retried page produces a second object with a different key and the same records inside. You now have duplicates that listing will never reveal, because both objects are legitimately present and neither is obviously a copy. Derive the key from the request instead of the run, so the same window plus the same cursor always produces the same key and a retry overwrites rather than accumulates.</p>



<pre class="wp-block-code"><code>import hashlib
import boto3
from botocore.config import Config
from botocore.exceptions import ClientError

s3 = boto3.client(
    "s3",
    config=Config(
        retries={"total_max_attempts": 8, "mode": "standard"},
        max_pool_connections=32,
    ),
)

def page_key(dataset, window, cursor):
    # Deterministic: the same page always maps to the same key.
    token = hashlib.sha256(cursor.encode()).hexdigest()[:16]
    return f"raw/{dataset}/dt={window}/page-{token}.json.gz"

def put_once(bucket, key, body):
    try:
        s3.put_object(Bucket=bucket, Key=key, Body=body, IfNoneMatch="*")
        return "written"
    except ClientError as err:
        if err.response["Error"]["Code"] == "PreconditionFailed":
            return "already_present"
        raise</code></pre>



<p class="wp-block-paragraph">Two things in there are worth explaining rather than copying.</p>



<p class="wp-block-paragraph"><code>IfNoneMatch="*"</code> is a conditional write. S3 accepts the PUT only if no object exists at that key and returns a 412 <code>PreconditionFailed</code> otherwise. It&#8217;s supported on <code>PutObject</code> and <code>CompleteMultipartUpload</code>. That turns &#8220;did I already write this page?&#8221; into a single request with no read-before-write race. If two workers hit the same key concurrently, S3 can instead return a 409 <code>ConditionalRequestConflict</code>, which AWS documents as retryable, so handle the two codes differently: retry on 409, skip on 412.</p>



<p class="wp-block-paragraph">The same thing from the CLI, useful for testing the behaviour by hand:</p>



<pre class="wp-block-code"><code>aws s3api put-object 
  --bucket your-bucket 
  --key raw/orders/dt=WINDOW/page-abc123.json.gz 
  --body page.json.gz 
  --if-none-match "*"</code></pre>



<p class="wp-block-paragraph">The <code>retries</code> block matters for a reason that trips people up. In botocore, <code>max_attempts</code> means different things depending on where you set it: in the AWS config file or the environment variable it counts total requests including the first, but in a <code>Config</code> object it counts retries on top of the first. <code>total_max_attempts</code> exists only in a <code>Config</code> object and always means total, so using it removes the ambiguity. There&#8217;s also an <code>adaptive</code> mode that layers client-side rate limiting on top of <code>standard</code>; AWS still labels it experimental, so I reach for <code>standard</code> plus my own concurrency cap first.</p>



<h2 class="wp-block-heading">Failure family 3: a layout designed for writing, not reading</h2>



<p class="wp-block-paragraph">The prefix structure you pick during the backfill is the one you&#8217;re stuck with. Reorganising later means rewriting every object, and copy requests aren&#8217;t free.</p>



<h3 class="wp-block-heading">Partition on event time, not ingestion time</h3>



<p class="wp-block-paragraph">The tempting layout is the date you ran the job. It&#8217;s easy, it&#8217;s monotonic, and it&#8217;s useless. Every query anyone writes against historical data filters on when the thing happened, not when you happened to fetch it. If your partitions are ingestion dates, every one of those queries becomes a full scan. Use Hive-style keys so query engines prune without extra configuration:</p>



<pre class="wp-block-code"><code>s3://your-bucket/raw/orders/dt=YYYY-MM-DD/page-&lt;token&gt;.json.gz
s3://your-bucket/curated/orders/dt=YYYY-MM-DD/part-&lt;n&gt;.parquet</code></pre>



<p class="wp-block-paragraph">Keep the raw landing zone and the curated zone separate. Raw is whatever the API gave you, unmodified, so you can re-derive everything if your parsing logic turns out to be wrong. Curated is columnar and typed. Merging them saves nothing and costs you the ability to reprocess.</p>



<h3 class="wp-block-heading">Don&#8217;t let the write path throttle itself</h3>



<p class="wp-block-paragraph">S3 documents at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix, with no limit on the number of prefixes. The word doing the work is <em>partitioned</em>. That rate applies once S3 has split the keyspace out for that prefix, and AWS is explicit that the scaling is gradual rather than instant. During the ramp you can see 503 <code>SlowDown</code> responses even when your sustained rate looks well inside the documented ceiling.</p>



<p class="wp-block-paragraph">For a backfill this is mostly good news, since date-partitioned keys spread writes across many prefixes as you work through the range. It becomes a problem when you parallelise <em>within</em> a single day and drive thousands of writes per second into one <code>dt=</code> prefix. Two mitigations: back off with jitter instead of retrying 503s immediately, and shard workers across windows rather than within them.</p>



<h3 class="wp-block-heading">Register partitions without a scan</h3>



<p class="wp-block-paragraph">A backfill can create thousands of partitions at once. <code>MSCK REPAIR TABLE</code> handles that badly. AWS documents that it only adds partitions and never removes them, that it can take a long time on large tables, and that it can time out leaving the catalog partially updated. Partial success with no error is exactly the failure mode this whole post is about.</p>



<p class="wp-block-paragraph">Partition projection sidesteps the catalog. Athena computes the partition list from table properties instead of looking it up, so a backfilled range is queryable the moment the objects exist.</p>



<pre class="wp-block-code"><code>TBLPROPERTIES (
  'projection.enabled'          = 'true',
  'projection.dt.type'          = 'date',
  'projection.dt.format'        = 'yyyy-MM-dd',
  'projection.dt.range'         = 'START_DATE,NOW',
  'projection.dt.interval'      = '1',
  'projection.dt.interval.unit' = 'DAYS',
  'storage.location.template'   = 's3://your-bucket/curated/orders/dt=${dt}/'
)</code></pre>



<p class="wp-block-paragraph">Replace <code>START_DATE</code> with the earliest date in your backfill. The trade-off is worth stating plainly: projection generates every partition in the range whether or not data exists there, so an over-wide range on a sparse dataset wastes planning effort. And if the S3 layout ever stops matching <code>storage.location.template</code>, queries return nothing rather than erroring. Set the range to your actual data, not to a comfortable margin.</p>



<h2 class="wp-block-heading">Failure family 4: the seam between backfill and live pipeline</h2>



<p class="wp-block-paragraph">Two processes write to the same table: a backfill walking backwards through history, and a live ingest walking forwards. They meet somewhere, and that meeting point is where records go missing.</p>



<p class="wp-block-paragraph">The instinct is to make the handoff exact. Backfill up to timestamp T, start streaming from T. Don&#8217;t. Clock skew between your runner and the API, records that arrive late at the source, and rounding in the API&#8217;s own filtering will all conspire to open a hole at exactly T.</p>



<p class="wp-block-paragraph">Overlap deliberately instead. Run the backfill past the point where live ingest started, by a margin generously larger than any plausible source lag, then deduplicate on a natural key during curation. Duplicates you can remove with a query. Gaps you can only fix by going back to the API, and by then the retention window may have closed.</p>



<h2 class="wp-block-heading">Failure family 5: the storage class you can&#8217;t undo</h2>



<p class="wp-block-paragraph">Historical data feels like archive data, so the instinct is to write it straight into a cold class or transition it immediately. Read the mechanics first, because the billing rules for cold classes are structural and pro-rated against you.</p>



<ul class="wp-block-list"><li>S3 Standard-IA and One Zone-IA carry a 30-day minimum storage duration and a 128 KB minimum billable object size.</li><li>Glacier Instant Retrieval and Glacier Flexible Retrieval carry a 90-day minimum duration; Deep Archive carries 180 days. Delete, overwrite or transition earlier and you&#8217;re charged the remainder.</li><li>Objects in Glacier Flexible Retrieval and Deep Archive are billed for 40 KB of extra per-object metadata on top of the object itself.</li></ul>



<p class="wp-block-paragraph">Read those together and the conclusion is uncomfortable. A backfill that writes millions of small JSON pages into an archive class pays a size floor on every one of them, plus metadata overhead, plus a minimum duration that makes fixing the mistake expensive. Those pages are also the objects you&#8217;re most likely to want to reprocess, which is exactly the overwrite that triggers the early-deletion charge.</p>



<p class="wp-block-paragraph">What I&#8217;d actually do: land raw pages in Standard, compact them into larger Parquet files in the curated zone, verify completeness, and only then apply a lifecycle rule to the raw prefix. Compaction is what makes cold storage sane, because it turns a size floor per tiny object into a rounding error on a large one. It also fixes the small-file problem in Athena, which is a separate cost line.</p>



<p class="wp-block-paragraph">One more cost note: if your backfill runs from EC2 in a private subnet, every byte pulled from the API and pushed to S3 may route through a NAT gateway with per-GB processing charges. A gateway VPC endpoint for S3 removes the S3 half of that. It&#8217;s a five-minute change people usually discover after the bill.</p>



<h2 class="wp-block-heading">Proving the backfill is complete</h2>



<p class="wp-block-paragraph">Here&#8217;s the part most backfills skip, and the part that makes everything above verifiable rather than hopeful. Write a manifest object per window, alongside the data, describing what that window contains.</p>



<pre class="wp-block-code"><code>{
  "dataset": "orders",
  "window": "&lt;dt value&gt;",
  "filter_field": "created_at",
  "range_start_inclusive": "&lt;iso8601&gt;",
  "range_end_exclusive": "&lt;iso8601&gt;",
  "pages_written": "&lt;int&gt;",
  "records_written": "&lt;int&gt;",
  "source_reported_total": "&lt;int or null&gt;",
  "last_cursor": "&lt;opaque token&gt;",
  "schema_fingerprint": "&lt;sha256 of sorted field names&gt;"
}</code></pre>



<p class="wp-block-paragraph">The manifest turns completeness into something you can query. A window with no manifest was never finished. A window whose <code>records_written</code> disagrees with <code>source_reported_total</code> is suspect. A window whose <code>schema_fingerprint</code> differs from its neighbours tells you the API changed shape partway through your range, which is otherwise close to undetectable.</p>



<p class="wp-block-paragraph">Write the manifest last, after every page for that window is confirmed in S3. A manifest written optimistically at the start is worse than no manifest at all, because it asserts a completeness nobody checked.</p>



<h2 class="wp-block-heading">Troubleshooting a backfill that already ran</h2>



<ol class="wp-block-list"><li><strong>Count objects per window and look at the shape.</strong> Use S3 Inventory rather than a live <code>ListObjectsV2</code> walk on a large bucket. Plot objects per <code>dt=</code> partition. Real data has a rhythm, weekday peaks and weekend troughs. A window that breaks the rhythm without a business reason is your gap.</li><li><strong>Reconcile record counts against the source.</strong> Many APIs return a total count on the first page of a filtered query. Pull that for a sample of windows and compare with what&#8217;s in S3. If you kept manifests, this is one query.</li><li><strong>Check for duplicate natural keys within a partition.</strong> A <code>GROUP BY</code> on the record ID with <code>HAVING COUNT(*) &gt; 1</code> finds retry duplicates immediately. Duplicates clustered at window boundaries mean pagination drift; duplicates spread evenly mean your keys weren&#8217;t deterministic.</li><li><strong>Look at boundary hours, not whole days.</strong> Cursor drift and clock skew concentrate at the edges of each request window. Aggregate to the hour and the missing slice usually becomes obvious even when daily totals look fine.</li><li><strong>Verify the layout still matches the table definition.</strong> If you&#8217;re using partition projection and a query returns zero rows, the first suspect is <code>storage.location.template</code> not matching the actual prefix, trailing slashes included.</li></ol>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list"><li>Treating an exit code of zero as evidence of completeness.</li><li>Putting a run ID or timestamp in the object key, which makes retries additive instead of idempotent.</li><li>Partitioning by ingestion date because it&#8217;s what the job naturally knows.</li><li>Making the backfill and live-ingest boundary exact instead of overlapping.</li><li>Writing millions of small objects straight into an archive class and meeting the minimum object size and duration rules afterwards.</li><li>Running <code>MSCK REPAIR TABLE</code> over thousands of new partitions and trusting that it completed.</li><li>Backfilling on a last-modified field while assuming you captured creation events.</li><li>Sharding workers within a single day, concentrating writes on one prefix and inviting 503s.</li></ul>



<h2 class="wp-block-heading">Best practices when you backfill historical API data into S3</h2>



<ul class="wp-block-list"><li>Make every window independently re-runnable. If re-running one day is routine rather than scary, you&#8217;ll actually fix problems when you find them.</li><li>Derive object keys deterministically from window plus cursor, and use conditional writes so a retry is a no-op.</li><li>Bound every API request on both ends, and never let the upper bound be &#8220;now&#8221;.</li><li>Write raw responses untouched before parsing anything. Parsing bugs are common; re-fetching a closed API window often isn&#8217;t possible.</li><li>Emit a manifest per window, written last, and reconcile it against source counts.</li><li>Compact to Parquet before applying any lifecycle transition.</li><li>Run the job somewhere it can survive for hours. A long backfill on a laptop over hotel wifi is a bad plan; a small dedicated VPS from a provider like Contabo or InterServer, or a spot-priced EC2 instance, costs little and won&#8217;t sleep halfway through.</li><li>Watch progress somewhere other than the terminal. Pushing records-per-window into CloudWatch or Grafana gives you a curve you can eyeball, and an unusually flat stretch is a gap you can catch while the source data is still retrievable.</li></ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Should I write raw JSON or convert to Parquet during the backfill?</h3>



<p class="wp-block-paragraph">Both, in two zones. Write the raw response body first so the fetch step has no dependency on your schema assumptions, then convert as a separate job. Converting inline couples an expensive, rate-limited API walk to a parsing step that will eventually throw on an unexpected field, and you don&#8217;t want to re-fetch years of data because of a type error.</p>



<h3 class="wp-block-heading">How large should each object be?</h3>



<p class="wp-block-paragraph">In the raw zone, one object per API page is fine and keeps idempotency simple. In the curated zone, aim for files in the tens to low hundreds of megabytes. Query engines pay a fixed overhead per file, so thousands of tiny Parquet files scan slower and cost more than a few large ones holding identical data.</p>



<h3 class="wp-block-heading">Can I run the backfill in parallel across date windows?</h3>



<p class="wp-block-paragraph">Yes, and that&#8217;s the right axis, because each window is independent and lands on its own prefix. The binding constraint is almost always the source API&#8217;s rate limit rather than S3. Add a global concurrency cap and honour any documented limit or <code>Retry-After</code> header. Getting your credentials throttled or suspended mid-backfill is a worse outcome than finishing slowly.</p>



<h3 class="wp-block-heading">Do I need AWS Glue or Airflow for this?</h3>



<p class="wp-block-paragraph">Not to start. A single script that takes a window as an argument, writes pages and a manifest, and is safe to re-run covers most backfills. Orchestration earns its place when you need retries across hundreds of windows, dependency ordering, or a visible run history. Glue and Step Functions fit AWS-native stacks; Airflow gives you more control and more to operate. Pick after the script works, not before.</p>



<h3 class="wp-block-heading">What if the API caps how far back I can query?</h3>



<p class="wp-block-paragraph">Then the retention boundary is your real deadline and it should drive priority. Fetch the oldest reachable data first, not the most recent, because the newest data is still available tomorrow and the oldest may not be. If the vendor offers a bulk export or reporting endpoint separate from the standard API, it&#8217;s usually faster and less rate-limited than paginating.</p>



<h3 class="wp-block-heading">Does any of this change with S3-compatible storage instead of AWS?</h3>



<p class="wp-block-paragraph">The layout, idempotency and manifest patterns carry over unchanged to Cloudflare R2, Backblaze B2, MinIO and similar. What varies is API-level feature support. Conditional writes with <code>If-None-Match</code>, request-rate behaviour and storage class semantics differ by provider, so verify the specific behaviour you&#8217;re depending on against that provider&#8217;s documentation rather than assuming parity with S3.</p>



<h3 class="wp-block-heading">How do I know the backfill is actually finished?</h3>



<p class="wp-block-paragraph">When every expected window has a manifest, every manifest&#8217;s record count reconciles against the source, and a duplicate check on natural keys returns nothing. Not when the script exits.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A backfill isn&#8217;t finished when the job exits. It&#8217;s finished when you can prove the range is complete without re-reading the source. Deterministic keys, conditional writes, closed windows, manifests and deliberate overlap all exist to make that proof cheap.</p>



<p class="wp-block-paragraph">If you&#8217;re about to backfill historical API data into S3, spend the extra hour on the key derivation and the manifest before you start the run. The alternative is learning about a gap from a chart, months later, with the source retention window already closed behind you.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a hand with a backfill?</h2>



<p class="wp-block-paragraph">This is a large part of what I do. If you&#8217;re staring at a historical range and an API with a rate limit, I can help with:</p>



<ul class="wp-block-list"><li>Designing an idempotent, re-runnable backfill job with deterministic keys and per-window manifests, so a failed day is a one-command fix.</li><li>Choosing a partition layout that serves your actual query patterns, and setting up Athena or Glue tables with partition projection so thousands of backfilled partitions become queryable without catalog scans.</li><li>Auditing a backfill that already ran, reconciling record counts against the source and finding the gaps and duplicates before your dashboards do.</li><li>Handling awkward source APIs: offset-only pagination, aggressive rate limits, undocumented result caps, schema drift partway through a historical range.</li><li>Compaction and lifecycle design, so cold storage actually saves money instead of hitting minimum object size and minimum duration charges.</li><li>Wiring the seam between the backfill and your live pipeline so there&#8217;s no gap at the handover, plus the monitoring that would have caught it.</li></ul>



<p class="wp-block-paragraph">Send me something concrete: a sample API response, your current key layout, or the object counts per partition. That&#8217;s usually enough to spot where the gap is.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/backfill-historical-api-data-into-s3/">Backfilling Historical API Data into S3 Without Silent Gaps</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Amazon Textract Data Extraction: What Breaks on Real Contracts and Reports</title>
		<link>https://john-nessime.com/blog/devops/amazon-textract-data-extraction/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 20 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon SNS]]></category>
		<category><![CDATA[Amazon SQS]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Batch Processing]]></category>
		<category><![CDATA[Boto3]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Legal Tech]]></category>
		<category><![CDATA[OCR]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Serverless]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=270</guid>

					<description><![CDATA[<p>Textract rarely fails loudly. It returns a plausible result that is quietly incomplete: a truncated result set, a tick box read as an empty string, a clause split across a page break. A practitioner's guide to the failure modes that actually bite when you point Amazon Textract at contracts, technical reports and correspondence, plus how to choose between sync and async, which feature types are worth paying for, and where Textract stops being the right tool.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/amazon-textract-data-extraction/">Amazon Textract Data Extraction: What Breaks on Real Contracts and Reports</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The pipeline passed every test. Three sample agreements, all the right fields, clean JSON out the other side. Then it ran against the real archive and somebody in legal noticed that every contract longer than about forty pages was missing its termination clause. No errors. No failed jobs. The dashboard was green the whole time.</p>



<p class="wp-block-paragraph">The bug was a missing loop. The job had finished, the results were there, and the code had read the first chunk of them and stopped.</p>



<p class="wp-block-paragraph">That is the shape of most problems with Amazon Textract data extraction on real documents. The service rarely fails loudly. It returns a plausible-looking result that is quietly incomplete, and you find out weeks later when somebody asks a question the data cannot answer. This post covers the failures that actually bite on contracts, technical reports and correspondence: sync versus async, reading the block graph without losing fields, which feature types are worth paying for, and where Textract stops being the right tool.</p>



<h2 class="wp-block-heading">Why Amazon Textract data extraction is not just OCR</h2>



<p class="wp-block-paragraph">Plain OCR gives you words and their positions. Textract gives you a graph. Every response is a flat array of <code>Block</code> objects, and each block carries an ID plus relationships to other block IDs. A page relates to its lines, a line to its words, a key to its value, a table to its cells.</p>



<p class="wp-block-paragraph">That structure is the whole point, and it is where the pain lives. Nothing is nested for you. To read one form field you find the <code>KEY_VALUE_SET</code> block with entity type <code>KEY</code>, follow its <code>VALUE</code> relationship to another block, then follow that block&#8217;s <code>CHILD</code> relationships to the words. Three hops, and getting a hop wrong returns an empty string rather than an exception.</p>



<p class="wp-block-paragraph">Contracts and reports make this harder than invoices do. An invoice has a total. A master services agreement has a liability cap buried in a numbered sub-clause that spans a page break.</p>



<h2 class="wp-block-heading">Synchronous or asynchronous: the choice is made for you</h2>



<p class="wp-block-paragraph">People reach for <code>AnalyzeDocument</code> first because it returns results in the same call and is easy to test in a notebook. Then they hit the quotas, which AWS documents as hard limits you cannot raise. Synchronous operations cap JPEG, PNG, PDF and TIFF at 10 MB in memory, and cap PDF and TIFF at <strong>one page</strong>. Asynchronous operations keep JPEG and PNG at 10 MB but take PDF and TIFF up to 500 MB and 3,000 pages.</p>



<p class="wp-block-paragraph">One page. That single constraint decides your architecture. Any real contract goes through <code>StartDocumentAnalysis</code> and <code>GetDocumentAnalysis</code>, which means it goes through S3, which means you need somewhere to stage documents and a way to learn when the job is done.</p>



<p class="wp-block-paragraph">Three other constraints from the same quota page are worth knowing before you promise anything to a client:</p>



<ul class="wp-block-list">
<li>Text detection covers English, French, German, Italian, Portuguese and Spanish. Query detection is English only.</li>

<li>Vertical text is not supported. Rotation is fine, including odd in-plane angles, but vertically written scripts are not.</li>

<li>Password-protected PDFs are rejected and XFA-based PDFs are unsupported. Both turn up in legal archives more often than you would expect.</li>
</ul>



<p class="wp-block-paragraph">Pass a <code>ClientRequestToken</code> on submission. It is an idempotency token: the same token returns the same <code>JobId</code> instead of starting a duplicate job. If your submitter is a Lambda function behind an S3 event, and S3 events can be delivered more than once, that is the difference between paying once and paying twice.</p>



<pre class="wp-block-code"><code>import boto3

textract = boto3.client("textract")

job = textract.start_document_analysis(
    DocumentLocation={
        "S3Object": {"Bucket": "contracts-intake", "Name": "msa/acme-2.pdf"}
    },
    FeatureTypes=["FORMS", "TABLES"],
    ClientRequestToken="msa-acme-2-v1",
    NotificationChannel={
        "SNSTopicArn": "arn:aws:sns:eu-west-1:111122223333:textract-done",
        "RoleArn": "arn:aws:iam::111122223333:role/TextractSnsPublish",
    },
)</code></pre>



<p class="wp-block-paragraph">The notification channel is optional but you want it. Polling in a loop inside a Lambda function burns billed duration doing nothing, and Lambda&#8217;s execution ceiling will cut you off on long documents anyway. Publish to SNS, fan out to SQS, let a second function do the reading, and you get a natural place to hang a dead letter queue.</p>



<h2 class="wp-block-heading">The result set you never finished reading</h2>



<p class="wp-block-paragraph">This is the one that cost the termination clauses. <code>GetDocumentAnalysis</code> returns results in pages. When there are more blocks than fit in one response, the response carries a <code>NextToken</code> and you have to call again with it. If you do not, you get the beginning of the document and nothing tells you so. <code>JobStatus</code> still reports success, because the job did succeed. Your code just stopped reading.</p>



<p class="wp-block-paragraph">Short test documents fit in a single response. That is exactly why this survives testing and dies in production.</p>



<pre class="wp-block-code"><code>def fetch_all_blocks(textract, job_id):
    blocks, next_token = [], None

    while True:
        kwargs = {"JobId": job_id}
        if next_token:
            kwargs["NextToken"] = next_token

        response = textract.get_document_analysis(**kwargs)

        if response["JobStatus"] == "FAILED":
            raise RuntimeError(response.get("StatusMessage", "job failed"))

        blocks.extend(response["Blocks"])
        next_token = response.get("NextToken")

        if not next_token:
            return blocks</code></pre>



<p class="wp-block-paragraph">Then make the failure detectable. Treat the highest page number present in the blocks as a checksum against the page count of the source file. Submit a 60-page PDF, get blocks that stop at page 12, and something is wrong no matter what the job status says.</p>



<h2 class="wp-block-heading">Choosing feature types without paying for all of them</h2>



<p class="wp-block-paragraph"><code>AnalyzeDocument</code> and <code>StartDocumentAnalysis</code> take a <code>FeatureTypes</code> list: <code>FORMS</code>, <code>TABLES</code>, <code>QUERIES</code>, <code>SIGNATURES</code> and <code>LAYOUT</code>. This is not cosmetic. Textract bills per page and the per-page rate depends on which features you enabled, so turning all five on because you might need them later multiplies your bill across the whole archive. Rates change and vary by region, so check the current pricing page rather than trusting a number in a blog post. The mechanism does not change: each feature adds to the per-page cost, and you pay it on every page you submit, including blank separator sheets.</p>



<ul class="wp-block-list">
<li><strong>FORMS</strong> finds key-value pairs where the document has an explicit label. Good for cover sheets, signature blocks, report headers. Useless for prose clauses with no label.</li>

<li><strong>TABLES</strong> reconstructs rows, columns and cells, and also identifies table titles, footers and merged cells. This is what you want for rate cards and appendix tables.</li>

<li><strong>QUERIES</strong> lets you ask natural-language questions and get answers back under an alias you chose. This is the feature that makes contracts tractable.</li>

<li><strong>SIGNATURES</strong> returns the location and confidence of handwritten signatures, electronic signatures and initials. It tells you something is signed at a coordinate, not whose signature it is.</li>

<li><strong>LAYOUT</strong> groups text into titles, headers, footers, section headers, paragraphs and lists in reading order. Skippable on single-column documents. On two-column reports it is the difference between coherent text and interleaved nonsense.</li>
</ul>



<p class="wp-block-paragraph">A practical pattern: run a cheap text-detection pass to classify each document, then apply the expensive feature set only to the pages that matter. A 200-page appendix of scanned site photographs does not need FORMS and TABLES.</p>



<h2 class="wp-block-heading">Reading the block graph without losing fields</h2>



<p class="wp-block-paragraph">Here is the failure that produces silent empty strings. A key-value pair&#8217;s value is not always text. If the value is a tick box, the value block&#8217;s children include a <code>SELECTION_ELEMENT</code> block, and that block has no text at all. It has a <code>SelectionStatus</code> of <code>SELECTED</code> or <code>NOT_SELECTED</code>. Concatenate only <code>WORD</code> children and a ticked box comes back as an empty string, which looks exactly like a field nobody filled in.</p>



<pre class="wp-block-code"><code>def block_text(block, block_map):
    parts = []
    for rel in block.get("Relationships", []):
        if rel["Type"] != "CHILD":
            continue
        for child_id in rel["Ids"]:
            child = block_map[child_id]
            if child["BlockType"] == "WORD":
                parts.append(child["Text"])
            elif child["BlockType"] == "SELECTION_ELEMENT":
                parts.append(child["SelectionStatus"])
    return " ".join(parts)</code></pre>



<p class="wp-block-paragraph">Build <code>block_map</code> once as a dictionary keyed on block ID before you traverse anything. Scanning the block list linearly for each lookup turns a three-hop traversal into an accidental quadratic, which you will notice the first time a 3,000-page bundle goes through.</p>



<h2 class="wp-block-heading">Queries: asking for clauses that have no label</h2>



<p class="wp-block-paragraph">FORMS works when the document says &#8220;Effective Date:&#8221; next to the date. Contracts frequently do not. The governing law sits in a paragraph of prose halfway down page nine. Queries handles that. You pass questions in plain English, each with an <code>Alias</code> you choose, and the response pairs <code>QUERY</code> blocks with <code>QUERY_RESULT</code> blocks through an <code>ANSWER</code> relationship. The alias is what makes this usable downstream: you match on <code>governing_law</code>, not on the exact wording of the question, so you can reword questions without breaking your schema.</p>



<pre class="wp-block-code"><code>QueriesConfig = {
    "Queries": [
        {"Text": "What is the governing law?",
         "Alias": "governing_law", "Pages": ["*"]},
        {"Text": "What is the termination notice period?",
         "Alias": "termination_notice", "Pages": ["9-*"]},
    ]
}</code></pre>



<p class="wp-block-paragraph">The limits matter. AWS documents a maximum of 15 queries per page for synchronous operations and 30 per page for asynchronous ones. A fifty-field schema cannot be asked in one pass. You either split it across multiple calls, which multiplies per-page spend, or you narrow <code>Pages</code> so each query runs only where the answer plausibly lives. Signature blocks are at the end, definitions near the front. Running every query against every page is the most common way people accidentally triple their bill.</p>



<p class="wp-block-paragraph">Queries are pre-trained on a spread of business documents including paystubs, bank statements, loan applications and mortgage notes. Your niche contract template was not in that set. Expect good results on common concepts and mediocre ones on house-specific terminology.</p>



<h2 class="wp-block-heading">Documents that fight back</h2>



<h3 class="wp-block-heading">Clauses that span a page break</h3>



<p class="wp-block-paragraph">Textract analyses pages. A clause starting at the bottom of page 11 and finishing at the top of page 12 is two disconnected fragments as far as the block graph is concerned. Queries will often return the fragment on one page and miss the qualifier on the other, which is the worst outcome available because the answer looks complete. The fix is not clever: reassemble full text in reading order using LAYOUT, then run clause-level logic over that rather than over per-page answers. Use Queries to <em>locate</em> a clause and the reassembled text to <em>read</em> it.</p>



<h3 class="wp-block-heading">Two-column reports and email threads</h3>



<p class="wp-block-paragraph">Without LAYOUT, a two-column technical report reads as alternating lines from both columns. Everything downstream inherits that corruption and it passes a spot check, because individual lines look fine. LAYOUT sequences elements in reading order and returns block types for titles, headers, footers and section headers, which is also what you need to chunk a report by section.</p>



<p class="wp-block-paragraph">Printed email chains are quoted text inside quoted text, newest message at the top, the same signature block repeated five times. Textract extracts all of it faithfully, which is the problem. Splitting a thread into individual messages is a text-processing job you do afterwards, not something a feature type solves.</p>



<h3 class="wp-block-heading">PDFs that were never scanned</h3>



<p class="wp-block-paragraph">This is the cheapest win available and almost everybody misses it. A large share of contract archives are born-digital PDFs exported from a word processor, and they already contain a text layer. Running OCR over them pays a service to guess at text you could have read directly. Put a triage step in front of the pipeline; <code>pdftotext</code> from poppler-utils pulls the embedded layer if there is one:</p>



<pre class="wp-block-code"><code>pdftotext -layout contract.pdf - | head -c 2000</code></pre>



<p class="wp-block-paragraph">Substantial text back means born-digital: route it down a cheaper path and reserve Textract for genuinely scanned material. Triage across a large archive is an embarrassingly parallel batch job that sits better on a plain VPS from somewhere like Contabo or InterServer than on per-invocation serverless billing, since you are CPU-bound for minutes at a time rather than reacting to events. One caveat: born-digital does not mean clean. Some exporters produce a text layer with broken word spacing or mangled ligatures, so sample before you trust it.</p>



<h2 class="wp-block-heading">Confidence scores you can actually act on</h2>



<p class="wp-block-paragraph">Every block carries a confidence score and the instinct is to threshold on it. That is usually wrong. Confidence tells you how sure the model is about <em>which characters are on the page</em>, not whether the field is semantically right. Textract can read a date with total certainty and hand you the wrong one, because it picked up the printing date in the footer instead of the effective date in the recitals. High confidence, completely wrong answer.</p>



<p class="wp-block-paragraph">It is also per block. A key-value pair has a score on the key, another on the value and separate ones on each word, so there is no single number to gate on. Validate the extracted value instead. Does the date parse and fall in a plausible range? Does the total equal the sum of the line items? Does the party name appear in the signature block as well as the preamble? Route to human review on failed validation and treat low confidence as one input among several.</p>



<h2 class="wp-block-heading">When Custom Queries adapters are worth the effort</h2>



<p class="wp-block-paragraph">If pre-trained Queries keeps getting your house document type wrong, train an adapter. Create it with <code>CreateAdapter</code>, annotate sample documents, train a version with <code>CreateAdapterVersion</code>, then reference it at inference time.</p>



<pre class="wp-block-code"><code>AdaptersConfig = {
    "Adapters": [
        {"AdapterId": ADAPTER_ID, "Version": "1", "Pages": ["1-5"]},
        {"AdapterId": ADAPTER_ID, "Version": "1", "Pages": ["6-*"]},
    ]
}</code></pre>



<p class="wp-block-paragraph">That <code>Pages</code> string takes digits, hyphens and an asterisk with no blank spaces, a page can only have one adapter applied to it, and an asterisk meaning all pages must be the only element in the list.</p>



<p class="wp-block-paragraph">Two things to weigh. AWS sets a minimum of five samples per query level for training or testing and says plainly that more is better, so treat five as a demo floor rather than a number that survives layout variation. And successful trainings per month are capped per account, so the tight annotate-train-evaluate loop you want is not available. Adapters suit one high-volume, house-specific document type with a stable schema. They are a poor fit for a long tail of one-off templates, which is what most legal archives actually are.</p>



<h2 class="wp-block-heading">Where Textract stops and a language model starts</h2>



<p class="wp-block-paragraph">This is a real decision now, so both sides deserve a hearing. Textract&#8217;s case is strong on standardised, high-volume documents. It is deterministic in a way generative models are not, it returns bounding-box geometry so you can point at exactly where a value came from, it gives per-element confidence, and it bills predictably per page. When an auditor asks why a field has the value it does, geometry beats a model&#8217;s reasoning.</p>



<p class="wp-block-paragraph">The generative side, whether that is Amazon Bedrock Data Automation as a managed document processing service or a model called directly through Amazon Bedrock, wins where the task needs reasoning rather than pattern matching. &#8220;Does this agreement contain an assignment restriction, and if so summarise it&#8221; is a comprehension question, and Textract has no answer for it. Generative approaches also cope better with layouts that were in nobody&#8217;s training set.</p>



<p class="wp-block-paragraph">Most production pipelines use both, which is what AWS&#8217;s own reference architectures push: Textract for faithful extraction with geometry and confidence, then a model over the extracted text for classification and comprehension. Choosing from scratch, ask whether your documents are standardised and high-volume. If yes, Textract plus a thin post-processing layer is cheaper and more auditable. If they are varied and volume is modest, a managed generative service gets you working faster. Google Document AI and Azure AI Document Intelligence cover similar ground if you are not committed to AWS.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Extraction stops partway through long documents.</strong> You are not following <code>NextToken</code>. Compare the highest page number in your blocks against the source page count.</li>

<li><strong>Fields come back empty on forms you know were filled in.</strong> The value is a <code>SELECTION_ELEMENT</code> and you are only reading <code>WORD</code> children.</li>

<li><strong>The document is rejected as too large.</strong> You are on the synchronous API. Multi-page PDFs must go through <code>StartDocumentAnalysis</code>, and the 10 MB cap is on in-memory size, not file size on disk.</li>

<li><strong>Textract cannot read the document at all.</strong> Check for password protection and XFA-based forms. Both are documented as unsupported.</li>

<li><strong>The service cannot access the S3 object.</strong> Usually the execution role missing <code>s3:GetObject</code>, or a bucket in a different region from the Textract endpoint you called.</li>

<li><strong>Throttling under batch load.</strong> Transactions-per-second quotas are per account per region. Smooth spiky traffic through a queue and retry with exponential backoff and jitter rather than raising concurrency.</li>

<li><strong>Garbled text on reports that look fine to you.</strong> Multi-column layout without LAYOUT enabled, or a source scan below roughly 150 DPI. Re-scanning beats any amount of post-processing.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Testing only on short documents, which hides the pagination bug entirely.</li>

<li>Enabling every feature type on every page because it is easier than deciding.</li>

<li>Running OCR over born-digital PDFs that already carry a text layer.</li>

<li>Treating a confidence score as a correctness score.</li>

<li>Scanning the block array linearly instead of building an ID map.</li>

<li>Polling for job completion inside a Lambda function instead of using SNS.</li>

<li>Storing only extracted values and discarding the raw JSON, so reprocessing means paying again.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Persist the full Textract JSON to S3 keyed on a hash of the source document. Reprocessing then costs storage, not extraction.</li>

<li>Classify first, extract second. A cheap text-detection pass tells you which expensive feature set each document needs.</li>

<li>Validate on the value, not the score. Parse dates, check ranges, cross-reference names against other parts of the document.</li>

<li>Use aliases on every query and treat them as your schema contract.</li>

<li>Set a per-job page-count assertion and alert on mismatches. That is your canary for silent truncation.</li>

<li>Keep source documents in an encrypted bucket with a lifecycle policy. Contracts and correspondence should not accumulate indefinitely by accident.</li>

<li>Track cost per document rather than per API call. Feature stacking makes the per-call figure meaningless.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can Amazon Textract handle multi-page PDF contracts?</h3>



<p class="wp-block-paragraph">Yes, through the asynchronous API only. Synchronous operations cap PDF and TIFF at one page. Asynchronous operations accept PDF and TIFF up to 500 MB and 3,000 pages, staged through S3.</p>



<h3 class="wp-block-heading">Why is my Textract output missing the end of the document?</h3>



<p class="wp-block-paragraph">Almost certainly because you read the first response from <code>GetDocumentAnalysis</code> and stopped. Results are paginated with a <code>NextToken</code> and the job status still reports success, so nothing signals the truncation. Loop until <code>NextToken</code> is absent.</p>



<h3 class="wp-block-heading">How many Textract Queries can I ask per page?</h3>



<p class="wp-block-paragraph">AWS documents 15 per page for synchronous operations and 30 per page for asynchronous ones. Larger schemas need multiple passes, which costs more, so narrowing each query&#8217;s page range is worth the effort.</p>



<h3 class="wp-block-heading">Does Textract work on handwriting and signatures?</h3>



<p class="wp-block-paragraph">It detects handwriting, and the SIGNATURES feature returns the location and confidence of handwritten signatures, electronic signatures and initials. It does not verify identity: it tells you something was signed and where, not by whom.</p>



<h3 class="wp-block-heading">What languages does Amazon Textract support?</h3>



<p class="wp-block-paragraph">Text detection covers English, French, German, Italian, Portuguese and Spanish. Queries detection is English only, vertically written text is unsupported, and Textract does not return the detected language in its output.</p>



<h3 class="wp-block-heading">Should I use Textract or a generative model?</h3>



<p class="wp-block-paragraph">Textract when documents are standardised, volume is high, and you need auditable geometry and per-field confidence. A generative service when layouts vary widely, volume is modest, or the task needs comprehension rather than extraction. Most production pipelines use both.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Amazon Textract data extraction fails quietly far more often than it fails loudly. A truncated result set, a tick box read as an empty string, a clause split across a page break, a confidently extracted wrong date: none of these throw an exception and none show up on a dashboard.</p>



<p class="wp-block-paragraph">So build the assertions that make silence detectable. Compare expected page counts against extracted ones, validate values rather than trusting scores, and keep the raw JSON so you can prove what the service actually returned. Textract is genuinely good at reading documents. Your job is noticing when it did not read all of them.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a document extraction pipeline that does not lose fields?</h2>



<p class="wp-block-paragraph">I build and fix document processing pipelines on AWS. Typical work looks like this:</p>



<ul class="wp-block-list">
<li>Auditing an existing Textract pipeline for silent truncation, dropped selection elements and mis-scoped queries</li>

<li>Designing the async architecture end to end: S3 intake, SNS and SQS fan-out, Lambda or container workers, dead letter handling and retries</li>

<li>Cutting per-page cost by classifying first, triaging born-digital PDFs out of the OCR path and narrowing feature types per page range</li>

<li>Turning block-graph JSON into a clean schema in CSV, a relational database or a data lake, with validation rules and a review queue for exceptions</li>

<li>Building the hybrid path where Textract handles extraction and a model on Amazon Bedrock handles classification and comprehension</li>

<li>Adding the observability that makes quiet failures loud: page-count assertions, per-document cost tracking, confidence distribution monitoring</li>
</ul>



<p class="wp-block-paragraph">If you have a redacted sample document, a chunk of Textract JSON or an extraction coming back half empty, send it over and I will tell you what is going wrong with it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/amazon-textract-data-extraction/">Amazon Textract Data Extraction: What Breaks on Real Contracts and Reports</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building an Insurance Claims Processing Pipeline on AWS That Fails Loudly</title>
		<link>https://john-nessime.com/blog/cloud-computing/insurance-claims-processing-pipeline-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 20 Aug 2026 09:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Insurance Technology]]></category>
		<category><![CDATA[Workflow Automation]]></category>
		<category><![CDATA[Amazon SNS]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Bedrock Data Automation]]></category>
		<category><![CDATA[Claims Automation]]></category>
		<category><![CDATA[Confidence Scoring]]></category>
		<category><![CDATA[Data Validation]]></category>
		<category><![CDATA[Dead Letter Queue]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Event-Driven Architecture]]></category>
		<category><![CDATA[HIPAA]]></category>
		<category><![CDATA[Human In The Loop]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Insurance Claims]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Step Functions]]></category>
		<category><![CDATA[Straight-Through Processing]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=464</guid>

					<description><![CDATA[<p>Claims pipelines rarely crash. They succeed, emit clean JSON, and hand a wrong number to a payment system. Six failure families in an insurance claims processing pipeline on AWS, with the Textract, Bedrock Data Automation and Step Functions details that decide whether a bad extraction is visible or silent.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/insurance-claims-processing-pipeline-aws/">Building an Insurance Claims Processing Pipeline on AWS That Fails Loudly</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The worst ticket on a claims pipeline is never the one that says the pipeline is down. A stuck queue is loud. It pages somebody, somebody restarts something, and it gets fixed before lunch. The bad ticket arrives three weeks later from finance: a run of claims was auto-approved at amounts nobody can reconcile, and every single execution in the Step Functions console is green.</p>



<p class="wp-block-paragraph">That is the failure mode that defines this problem. An insurance claims processing pipeline on AWS almost never falls over in the way you designed it to fall over. It succeeds. It emits well-formed JSON. It hands a number to a payment system, and the number is wrong, and nothing in the pipeline had any reason to think otherwise.</p>



<p class="wp-block-paragraph">This post is organized by failure family rather than by service. I&#8217;ll walk through the six ways these pipelines go quietly wrong, what each one costs, and what the fix actually looks like in Textract, Bedrock, Step Functions and S3. There&#8217;s a troubleshooting section, the mistakes I see repeated, and an FAQ at the end.</p>



<h2 class="wp-block-heading">The shape most claims pipelines end up with</h2>



<p class="wp-block-paragraph">Before the failure families make sense, the skeleton. Almost every serverless claims pipeline lands on roughly the same set of stages, whatever the vendor deck calls them:</p>



<ol class="wp-block-list">
<li><strong>Intake.</strong> A document lands in S3 from a portal upload, an SFTP drop, or a mail scanning vendor. An S3 event or EventBridge rule starts an execution.</li>

<li><strong>Classification.</strong> Work out what the packet actually contains. A first notice of loss, a CMS-style claim form, a police report, an itemized bill, forty pages of photographs.</li>

<li><strong>Extraction.</strong> Pull the fields you need. Amazon Textract for OCR, forms and tables, or Amazon Bedrock Data Automation with a blueprint that names the fields directly.</li>

<li><strong>Validation.</strong> Check the extracted values against business rules, policy data, and each other.</li>

<li><strong>Routing.</strong> Straight-through processing, human review, or rejection with a reason.</li>

<li><strong>Persistence and audit.</strong> The claim record, the extraction artifacts, and enough evidence to explain a decision months later.</li>
</ol>



<p class="wp-block-paragraph">Nothing controversial there. AWS publishes an open-source GenAI IDP Accelerator that implements exactly this shape, with a Bedrock Data Automation mode and a Textract-plus-foundation-model pipeline mode, and it&#8217;s a reasonable place to start reading. The interesting part is not the boxes. It&#8217;s what happens between them.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: the field that was never there</h2>



<p class="wp-block-paragraph">This is the one that pays out the wrong number, and it is worth more attention than everything else in this post combined.</p>



<p class="wp-block-paragraph">Every extraction service gives you confidence scores. So the obvious design is a gate: if every field scores above some threshold, approve automatically; if anything falls below, send it to a human. That gate is sound reasoning applied to the wrong population.</p>



<p class="wp-block-paragraph">A confidence score only exists for a value that came back. When the extractor doesn&#8217;t find a field at all, there is no low score to catch, because there&#8217;s nothing to score. The gate iterates over four returned fields, finds all four above threshold, and reports a clean pass. The fifth field, the one that determines coordination of benefits or the deductible offset, is simply absent from the response. Downstream code treats absent as zero, or as null, or as &#8220;not applicable,&#8221; and the claim goes through.</p>



<p class="wp-block-paragraph">The fix is structural, not statistical. Validate <em>presence against a schema</em> before you validate confidence, and treat the two as separate gates with separate outcomes:</p>



<pre class="wp-block-code"><code># Two gates, not one. Presence first, then confidence.
# 'extracted' is the flattened field map from Textract Queries
# or a Bedrock Data Automation blueprint result.

REQUIRED = {
    "claim_number",
    "date_of_service",
    "billed_amount",
    "member_id",
    "secondary_payer_indicator",
}

def gate(extracted, scores, threshold=0.95):
    missing = REQUIRED - set(extracted)
    if missing:
        # Never silently default. This is a routing decision.
        return "HUMAN_REVIEW", {"reason": "missing_fields",
                                "fields": sorted(missing)}

    weak = [f for f in REQUIRED if scores.get(f, 0.0) &lt; threshold]
    if weak:
        return "HUMAN_REVIEW", {"reason": "low_confidence",
                                "fields": sorted(weak)}

    return "STRAIGHT_THROUGH", {}
</code></pre>



<p class="wp-block-paragraph">Two details matter here. The set difference is computed against a declared schema, not against whatever keys happen to be in the response, so an absent field becomes a first-class routing reason. And <code>scores.get(f, 0.0)</code> defaults to zero rather than to a passing value, so a field that arrives without a score fails closed.</p>



<p class="wp-block-paragraph">If you&#8217;re on Textract Queries, there&#8217;s a second reason to be explicit: Queries let you attach an alias to each question, which means your schema keys are yours rather than whatever label happened to be printed on the form. That&#8217;s the difference between &#8220;the field is missing&#8221; and &#8220;the field moved and we didn&#8217;t notice.&#8221;</p>



<h2 class="wp-block-heading">Failure family two: confidence scores that answer a different question</h2>



<p class="wp-block-paragraph">Assume you&#8217;ve fixed presence. The next trap is what the confidence number is measuring.</p>



<p class="wp-block-paragraph">OCR confidence is a statement about characters. It says the model is highly sure those pixels read <code>1,240.00</code>. It is not a statement that <code>1,240.00</code> is the billed amount rather than the allowed amount from the box directly above it, or the prior balance from a remittance summary that happened to be stapled into the same packet. Read it as a legibility score, because that&#8217;s closer to what it is.</p>



<p class="wp-block-paragraph">Bedrock Data Automation narrows this gap: blueprints define fields semantically, confidence scores come with bounding boxes, and the visual grounding lets you point at the region a value came from. Textract Queries narrow it too, by asking a question rather than harvesting a label. Neither eliminates the problem, because a high-confidence read of the wrong region still scores high.</p>



<p class="wp-block-paragraph">What actually catches this is cross-field invariants. They cost almost nothing and they fail for reasons a human can read:</p>



<ul class="wp-block-list">
<li><strong>Arithmetic.</strong> Line items sum to the claimed total. If they don&#8217;t, one of the two is wrong and you don&#8217;t yet know which.</li>

<li><strong>Temporal.</strong> Date of service falls inside the policy period and before the date of submission. A service date after the submission date is a parsing error nine times out of ten.</li>

<li><strong>Referential.</strong> The member or policy identifier resolves against your system of record. An identifier that matches the format but not a real record is a strong signal you read the wrong box.</li>

<li><strong>Range.</strong> Amounts within a plausible band for the claim type. Not a fraud model, just a tripwire for a decimal point that moved.</li>

<li><strong>Page provenance.</strong> Fields that must come from the same page or the same document within the packet. Bounding box data makes this checkable rather than assumed.</li>
</ul>



<p class="wp-block-paragraph">An invariant failure is more useful than a low score, because it names a contradiction. &#8220;Line items sum to 1,180 but the claimed total reads 1,240&#8221; is something a reviewer resolves in seconds. &#8220;Confidence 0.91&#8221; is something a reviewer stares at.</p>



<h2 class="wp-block-heading">Failure family three: the claim that stops halfway</h2>



<p class="wp-block-paragraph">Claims documents are multi-page packets, so you&#8217;ll be using Textract&#8217;s asynchronous operations. That means jobs, notifications, and a whole class of orchestration bugs that only show up under load or after a weekend.</p>



<p class="wp-block-paragraph">The asynchronous pattern is: call <code>StartDocumentAnalysis</code>, get a <code>JobId</code> back, and let Textract publish completion to an SNS topic you nominate. A request looks like this:</p>



<pre class="wp-block-code"><code>{
  "DocumentLocation": {
    "S3Object": { "Bucket": "claims-intake", "Name": "packets/abc123.pdf" }
  },
  "FeatureTypes": ["FORMS", "TABLES"],
  "ClientRequestToken": "abc123-v1",
  "JobTag": "fnol-packet",
  "NotificationChannel": {
    "SNSTopicArn": "arn:aws:sns:REGION:ACCOUNT:textract-complete",
    "RoleArn": "arn:aws:iam::ACCOUNT:role/TextractPublishRole"
  },
  "OutputConfig": {
    "S3Bucket": "claims-extraction",
    "S3Prefix": "raw/"
  },
  "KMSKeyId": "alias/claims-cmk"
}
</code></pre>



<p class="wp-block-paragraph">Three of those parameters are doing load-bearing work that is easy to skip.</p>



<p class="wp-block-paragraph"><code>ClientRequestToken</code> is the idempotency token. Reuse the same token and you get the same <code>JobId</code> back instead of a second job. Derive it from the document, not from the invocation, and a Lambda retry or a duplicated S3 event stops turning into a duplicate charge and a duplicate claim record.</p>



<p class="wp-block-paragraph"><code>OutputConfig</code> writes results into a bucket you control. Without it, results stay internal to Textract and the only way to read them is the <code>Get</code> operations, which have their own throttling limits. Under concurrency those limits become the bottleneck: you end up polling more jobs than you&#8217;re allowed to poll, backing off, and watching end-to-end latency climb for reasons that have nothing to do with the documents. Writing to S3 sidesteps the whole path.</p>



<p class="wp-block-paragraph"><code>JobTag</code> shows up in the completion notification. In a mixed pipeline where the same topic carries first notice of loss packets, itemized bills and ID documents, that tag is what lets the notification handler route without a lookup.</p>



<p class="wp-block-paragraph">One expiry to plan around: a Textract <code>JobId</code> is only valid for seven days. If your retry story is &#8220;requeue it and someone will look on Monday,&#8221; a bad weekend turns recoverable failures into full reprocessing. Persist the S3 output location, not the job identifier.</p>



<h3 class="wp-block-heading">Callbacks that never come back</h3>



<p class="wp-block-paragraph">Human review means pausing a workflow for hours or days, which in Step Functions means the callback pattern. You append <code>.waitForTaskToken</code> to the resource ARN, pass <code>$$.Task.Token</code> into the payload, and the execution parks until something calls <code>SendTaskSuccess</code> or <code>SendTaskFailure</code> with that token.</p>



<p class="wp-block-paragraph">The trap is that a callback task with no timeout waits until the execution itself hits its quota, and Standard workflow executions can run for up to a year. A reviewer who leaves, a review UI that drops the token, a queue consumer that crashes after reading the message and before writing it to the review table: all of these produce an execution that is neither failed nor finished. It just sits there. Nobody alerts on it because nothing broke.</p>



<pre class="wp-block-code"><code>"AwaitAdjusterDecision": {
  "Type": "Task",
  "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
  "Parameters": {
    "FunctionName": "enqueue-review-task",
    "Payload": {
      "claimId.$": "$.claimId",
      "taskToken.$": "$$.Task.Token"
    }
  },
  "TimeoutSeconds": 259200,
  "HeartbeatSeconds": 3600,
  "Catch": [{
    "ErrorEquals": ["States.Timeout"],
    "Next": "EscalateStaleReview"
  }],
  "Next": "ApplyDecision"
}
</code></pre>



<p class="wp-block-paragraph"><code>TimeoutSeconds</code> is the maximum total lifetime of the task regardless of heartbeats. <code>HeartbeatSeconds</code> is the maximum gap between <code>SendTaskHeartbeat</code> calls, so a review app that periodically confirms the item is still in someone&#8217;s queue will fail fast when that app dies, rather than at the outer limit. AWS&#8217;s own guidance is to set the heartbeat below the task timeout for exactly this reason: a heartbeat failure tells you the worker died, a timeout tells you the work took too long, and those are different incidents. Catch <code>States.Timeout</code> and route to a real state. An unhandled timeout is just a differently-shaped silence.</p>



<p class="wp-block-paragraph">One constraint worth knowing before you design around it: the callback pattern requires Standard workflows. Express workflows support request-response integrations only, so no <code>.waitForTaskToken</code> and no <code>.sync</code>. If you split your pipeline into a fast Express path and a Standard review path, the boundary between them is where the token has to live.</p>



<h2 class="wp-block-heading">Failure family four: the human review service you can no longer sign up for</h2>



<p class="wp-block-paragraph">This one catches people copying a reference architecture, and it&#8217;s the reason to read publication dates on IDP blog posts.</p>



<p class="wp-block-paragraph">Amazon Augmented AI, known as A2I, was the managed answer to human-in-the-loop review. It plugged directly into Textract&#8217;s <code>AnalyzeDocument</code>, watched confidence conditions, and spun up review tasks with a worker UI for you. It appears in a great many architecture diagrams for claims and lending workflows.</p>



<p class="wp-block-paragraph">Per the AWS documentation, SageMaker A2I is no longer open to new customers. Existing customers can keep using it, and AWS continues security and availability work, but no new features are planned. If you&#8217;re standing up a new account today, that diagram does not deploy.</p>



<p class="wp-block-paragraph">Be fair about what that costs you, because A2I genuinely removed real work: the task assignment logic, the worker UI, the private workforce plumbing through Cognito, result consolidation. Rebuilding it means owning all of that. What you get back is that the review queue becomes yours, which in practice means you can put claim-specific context on the screen instead of a generic key-value editor. For adjusters that difference is not cosmetic.</p>



<p class="wp-block-paragraph">A minimal replacement is not exotic:</p>



<ul class="wp-block-list">
<li>A DynamoDB table of review items, each holding the claim identifier, the extracted values, the bounding boxes, and the Step Functions task token.</li>

<li>A small web app for reviewers that renders the page image with the boxes overlaid, so a reviewer confirms placement rather than retyping values.</li>

<li>An API that writes the corrected values and calls <code>SendTaskSuccess</code> with the stored token.</li>

<li>Authentication in front of it. Amazon Cognito if you want to stay inside AWS, or an identity-aware proxy such as Cloudflare Access if your reviewers are external adjusters you&#8217;d rather not create AWS identities for.</li>

<li>A sweeper that finds review items older than your heartbeat window and escalates them.</li>
</ul>



<p class="wp-block-paragraph">If the review app is a small internal tool with no data residency requirement of its own, it doesn&#8217;t have to live in the same account or even the same provider. A modest VPS from a host like Contabo or InterServer running behind a Cloudflare Tunnel is a legitimate answer for a reviewer console that talks to AWS over scoped API credentials. Just be honest about what crosses that boundary, which brings us to the next family.</p>



<h2 class="wp-block-heading">Failure family five: claim data in places nobody decided to put it</h2>



<p class="wp-block-paragraph">Claims documents carry protected health information, financial identifiers, and often photographs of people and property. The pipeline you drew has three or four places that data lives. The pipeline you deployed has a dozen.</p>



<p class="wp-block-paragraph">The ones that get missed:</p>



<ul class="wp-block-list">
<li><strong>Lambda logs.</strong> One <code>print</code> of an event payload during a debugging session, and CloudWatch Logs is now a claims repository with a different retention policy and a different access model.</li>

<li><strong>Dead letter queues.</strong> A DLQ holds the full failed message. If that message carries extracted values, your DLQ is regulated data, and it is usually the least governed thing in the account.</li>

<li><strong>Step Functions execution history.</strong> State input and output are visible in the console and the history API. Passing extracted fields between states puts them there.</li>

<li><strong>Intermediate extraction output.</strong> The bucket you pointed <code>OutputConfig</code> at holds raw OCR of the whole packet, often with a lifecycle policy nobody wrote.</li>

<li><strong>Model invocation logging.</strong> Bedrock can log inputs and outputs to S3 or CloudWatch. Useful for debugging, and another copy of everything.</li>
</ul>



<p class="wp-block-paragraph">The pattern that keeps this manageable is passing pointers, not payloads. States carry an S3 key and a claim identifier; the values themselves stay in one encrypted bucket with one lifecycle policy and one access policy. It makes debugging marginally more annoying and it makes the data map fit on a page.</p>



<p class="wp-block-paragraph">On regulated workloads, check the current AWS HIPAA-eligible services list and your executed BAA for every service in the path, in the specific region you&#8217;re deploying to. Eligibility is per service and it changes. Textract has long been used for claims workflows on that basis, and Bedrock is listed as HIPAA eligible, but &#8220;I read a blog post&#8221; is not a control. Pull the list yourself before PHI touches anything.</p>



<p class="wp-block-paragraph">Two smaller things worth deciding early. Reviewers working from home should reach the console over something better than the open internet, whether that&#8217;s a corporate tunnel, a business VPN account from a provider like NordVPN or Surfshark, or an identity-aware proxy. And if reviewers ever download claim documents locally, agree what happens to those files afterward, because a deleted file is not an erased file. Tools such as O&amp;O SafeErase exist for exactly that gap on Windows endpoints.</p>



<h2 class="wp-block-heading">Failure family six: paying twice for the same page</h2>



<p class="wp-block-paragraph">Document AI services bill per page. That single fact reshapes how you think about retries, because in most pipelines a retry is free and here it isn&#8217;t.</p>



<p class="wp-block-paragraph">The expensive patterns are all shaped the same way. A poison document fails a downstream parser, gets requeued, and is re-extracted on every attempt. A batch job re-runs over an entire prefix instead of a delta. A misconfigured S3 event delivers twice. An operator reprocesses a day&#8217;s intake to fix a mapping bug in the transform stage, when the extraction stage was fine all along.</p>



<p class="wp-block-paragraph">The structural fix is separating extraction from interpretation. Extract once, write the raw result to S3 keyed by a content hash of the document, and let every downstream stage read from that. When the mapping bug shows up, you re-run interpretation over stored output and pay nothing. Combined with <code>ClientRequestToken</code>, most accidental double-charges disappear.</p>



<p class="wp-block-paragraph">Also route the packet before you extract it. Forty pages of accident photographs do not need forms and tables analysis. Classification is cheaper than extraction, and page-level routing is often the single largest lever on the bill.</p>



<p class="wp-block-paragraph">For attributing that spend, cost allocation tags on the buckets and functions give you the AWS-native view, and platforms like Vantage or CloudZero are worth a look if you need per-claim or per-client unit costs rather than per-service totals. Whatever you use, the metric that matters is cost per claim processed, split by straight-through versus reviewed. Those two numbers tell you whether the automation is earning its keep.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting an insurance claims processing pipeline on AWS</h2>



<p class="wp-block-paragraph">Symptoms you&#8217;ll actually see, and where to look first.</p>



<ul class="wp-block-list">
<li><strong>Executions succeed but downstream amounts are wrong.</strong> Check whether required fields are present, not just confident. Diff the schema against the response keys for a sample of recent claims. This is failure family one until proven otherwise.</li>

<li><strong>Executions stuck in Running for days.</strong> A callback task with no timeout. List running executions ordered by start time and look for the state name of your review task.</li>

<li><strong>Throttling on the extraction stage under load.</strong> If you&#8217;re polling <code>Get</code> operations, move to <code>OutputConfig</code> and SNS notification and stop polling. If you&#8217;re already there, check the start-operation limits rather than assuming the whole service is slow.</li>

<li><strong>The same claim appearing twice.</strong> Look for a missing or per-invocation <code>ClientRequestToken</code>, and check whether your S3 event handler is idempotent. Delivery is at-least-once.</li>

<li><strong>Extraction quality dropped for one document type.</strong> Usually the form changed, not the model. Compare bounding boxes for the affected field against an older sample. If the box moved, that&#8217;s a layout change, and query aliases or a blueprint update is the fix.</li>

<li><strong>Review queue growing faster than reviewers clear it.</strong> Break the routing reasons apart. If most items are low confidence on one field, that&#8217;s an extraction problem wearing a staffing problem&#8217;s clothes.</li>

<li><strong>SNS notification arrives, handler can&#8217;t find the results.</strong> Confirm the notification role has permission to publish and the handler is reading the S3 prefix rather than calling <code>Get</code> with an expired job identifier.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Gating only on confidence, so a missing field is indistinguishable from a clean extraction.</li>

<li>Defaulting absent values to zero or null in the transform layer instead of raising a routing decision.</li>

<li>Copying an architecture diagram that includes A2I into a new AWS account.</li>

<li>Callback tasks with no <code>TimeoutSeconds</code> and no heartbeat.</li>

<li>Passing extracted claim values through Step Functions state rather than passing an S3 pointer.</li>

<li>Treating a single global confidence threshold as adequate for every field. A name and a dollar amount do not carry the same downstream risk.</li>

<li>Running forms and tables analysis over every page of a packet including the photographs.</li>

<li>No metric for straight-through rate, so nobody notices when it quietly drops.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li><strong>Declare the schema, then validate presence, then confidence, then invariants.</strong> Four gates, four distinct rejection reasons, four things a reviewer can act on.</li>

<li><strong>Set per-field thresholds by consequence.</strong> Get the payable amount wrong and money moves. Get a street suffix wrong and a letter is slightly odd.</li>

<li><strong>Keep a labeled regression set.</strong> A few dozen real packets with known-correct values, run on every blueprint or query change. Bedrock Data Automation can use ground-truth examples to refine blueprint instructions, which only works if you maintain the ground truth.</li>

<li><strong>Store bounding boxes alongside values.</strong> They make review faster and they turn &#8220;quality dropped&#8221; from a guess into a comparison.</li>

<li><strong>Alarm on rates, not just errors.</strong> Straight-through rate, review-queue age, and cost per claim. A pipeline that stops approving anything is broken even though nothing threw.</li>

<li><strong>Make every stage idempotent on a content hash.</strong> Reprocessing is normal. It should be safe and cheap.</li>

<li><strong>Instrument the pipeline like a pipeline.</strong> CloudWatch covers the AWS surface; if you&#8217;re consolidating with on-premises claims systems, a platform like Grafana Cloud gives you one place to correlate both sides.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Should I use Amazon Textract or Bedrock Data Automation for claims extraction?</h3>



<p class="wp-block-paragraph">Textract is the sharper tool when your documents are standardized forms and you want deterministic OCR with forms, tables and targeted queries. Bedrock Data Automation is stronger on mixed packets, because it splits along logical document boundaries, classifies each part, and applies a blueprint per document type, with confidence scores and visual grounding on the output. Claims intake is usually mixed packets, which tilts toward Data Automation, but the honest answer is to run both against a sample of your real documents. The evaluation costs a day and it decides your architecture.</p>



<h3 class="wp-block-heading">What replaces Amazon A2I for human review?</h3>



<p class="wp-block-paragraph">For new AWS accounts, a custom review path: a queue or table of review items, a reviewer UI, and the Step Functions callback pattern to resume the workflow. It&#8217;s more code than A2I but not a large amount, and it gives you a review screen designed around claims rather than around generic key-value pairs. Existing A2I customers can continue as they are, though building on a service with no planned features is a decision to make deliberately rather than by default.</p>



<h3 class="wp-block-heading">What straight-through processing rate should I expect?</h3>



<p class="wp-block-paragraph">Anyone quoting you a number without seeing your documents is guessing. It depends almost entirely on document quality and how many fields you require. What&#8217;s reliable is the method: measure your current rate, split failures by reason, and fix the largest reason. Requiring one rarely-present field can dominate everything else, and that&#8217;s a policy decision as much as an engineering one.</p>



<h3 class="wp-block-heading">How do I keep an insurance claims processing pipeline on AWS HIPAA-aligned?</h3>



<p class="wp-block-paragraph">Start from the AWS HIPAA-eligible services list and an executed BAA, and confirm eligibility for each service in your specific region. Then do the unglamorous work: customer-managed KMS keys, no PHI in logs or state payloads, scoped IAM roles per stage, VPC endpoints where the service supports them, retention policies on every bucket and queue including dead letter queues, and CloudTrail configured so you can answer who accessed which claim. Eligibility is permission to build; the controls are yours.</p>



<h3 class="wp-block-heading">Can I run the whole pipeline with Step Functions Express workflows?</h3>



<p class="wp-block-paragraph">Not the part that waits for a human. Express workflows support request-response integrations only, so the callback pattern requires Standard. A common split is Express for the high-volume deterministic stages and Standard for anything holding a task token, with the two connected by an event or a queue.</p>



<h3 class="wp-block-heading">How do I stop duplicate claims from duplicate events?</h3>



<p class="wp-block-paragraph">Treat every trigger as at-least-once. Derive an idempotency key from the document itself, usually a content hash plus a version marker, pass it as <code>ClientRequestToken</code> to the extraction call, and use it as the conditional write key when you create the claim record. Then a duplicate event is a no-op rather than a second claim.</p>



<h3 class="wp-block-heading">Is it worth starting from the AWS GenAI IDP Accelerator?</h3>



<p class="wp-block-paragraph">As a reference for structure and as a way to get a working pipeline in front of stakeholders quickly, yes. As a production system you inherit wholesale, be careful: you&#8217;re adopting someone else&#8217;s opinions about classification, review and storage, and you&#8217;ll be reading that code anyway the first time something behaves oddly. Read it, borrow the patterns, own what you deploy.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">An insurance claims processing pipeline on AWS is not hard to build. Textract, Bedrock Data Automation, Step Functions and S3 will get you a working pipeline in a couple of weeks. What&#8217;s hard is making it fail in ways you can see.</p>



<p class="wp-block-paragraph">Every expensive failure in this space shares one shape: the pipeline had no opinion about what it did not receive. A field that didn&#8217;t come back scored nothing, a callback that never fired errored nothing, a duplicate event failed nothing. If you take one design rule from this, take that one. Declare what a complete claim looks like, check for its absence explicitly, and route anything incomplete to a human with a reason attached.</p>



<p class="wp-block-paragraph">Green executions are not evidence. Reconciled numbers are.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with your claims pipeline?</h2>



<p class="wp-block-paragraph">I work with teams building document-heavy workflows on AWS, and claims pipelines are one of the places where a small amount of design care prevents a large amount of reconciliation work. Things I can help with:</p>



<ul class="wp-block-list">
<li>Reviewing an existing extraction pipeline for silent-failure paths, particularly missing-field handling and default values in the transform layer</li>

<li>Designing the routing logic: schema gates, per-field thresholds, cross-field invariants, and the escalation rules that sit behind them</li>

<li>Building a human review path with the Step Functions callback pattern, including timeouts, heartbeats and a sweeper for stale tasks</li>

<li>Running a structured evaluation of Amazon Textract against Bedrock Data Automation on your actual documents, with a labeled regression set you keep afterward</li>

<li>Tracing where claim data actually lands across logs, queues, execution history and intermediate buckets, then shrinking that footprint</li>

<li>Instrumenting straight-through rate, review-queue age and cost per claim so regressions surface before finance finds them</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;d like a second opinion, send me a state machine definition, a sample extraction response with the values redacted, or the routing code that decides what goes to review. That&#8217;s usually enough to spot the gap.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/insurance-claims-processing-pipeline-aws/">Building an Insurance Claims Processing Pipeline on AWS That Fails Loudly</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</title>
		<link>https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 17 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Cursor Pagination]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[GraphQL]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Orchestration]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[Step Functions]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=215</guid>

					<description><![CDATA[<p>A GraphQL source can hand you a 200 OK, a populated data block, and a quietly broken column in the same response. Here is how to build a GraphQL data ingestion pipeline on AWS that catches partial errors, respects cost-based rate limits, resumes cleanly from a cursor, and notices when the schema moves under you.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/">Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The row counts looked fine. That was the problem.</p>



<p class="wp-block-paragraph">A nightly sync from a partner&#8217;s GraphQL API had run for weeks without a single failed execution. Green across the board in the console. Then someone asked why one column had been empty since the previous Tuesday. Not sparse. Empty. Every row, every night, for six nights.</p>



<p class="wp-block-paragraph">The API had started returning a field-level error on that one field. Status code 200. A populated <code>data</code> block. An <code>errors</code> array sitting right next to it. The fetcher parsed <code>data</code>, wrote the records to S3, exited zero, and the orchestrator logged a success. Nothing in the stack was wrong, exactly. Everything in the stack was checking the wrong thing.</p>



<p class="wp-block-paragraph">This post covers how to build a GraphQL data ingestion pipeline on AWS that fails honestly. It is organised by failure family rather than by service, because the AWS wiring is the easy part. The hard part is that GraphQL breaks four assumptions that REST ingestion quietly relies on, and every one of them fails silently by default.</p>



<h2 class="wp-block-heading">Why GraphQL ingestion breaks the habits REST taught you</h2>



<p class="wp-block-paragraph">Most ETL code carries three inherited reflexes from years of REST work: the status code tells you whether it worked, the response shape is the server&#8217;s business, and rate limits are counted in requests. GraphQL invalidates all three.</p>



<ul class="wp-block-list">
<li><strong>The status code is not the contract.</strong> The GraphQL over HTTP specification says a server should return 200 even when field errors are raised during execution, because a partial response is still a successful execution.</li>

<li><strong>You own the response shape.</strong> The selection set is written by you, which means schema drift shows up as a query that stops being valid, not as a payload that changes underneath you.</li>

<li><strong>Cost is query-shaped, not endpoint-shaped.</strong> One endpoint can cost almost nothing or blow your entire budget depending on how deeply you nested it.</li>
</ul>



<p class="wp-block-paragraph">Each of those becomes a distinct failure family. Let&#8217;s take them in the order they bite.</p>



<h2 class="wp-block-heading">Failure one: the 200 OK that quietly ate a column</h2>



<p class="wp-block-paragraph">This is the one that costs the most and shows up the latest. GraphQL splits errors into two categories, and they behave completely differently on the wire.</p>



<ul class="wp-block-list">
<li><strong>Request errors</strong> happen before execution starts: syntax errors, validation failures, bad variable coercion. No <code>data</code> key is returned at all. Nothing executed.</li>

<li><strong>Field errors</strong> happen during execution: a resolver threw, a downstream service timed out, a permission check failed. Execution continues, the failed field goes null, and you get <code>data</code> and <code>errors</code> together.</li>
</ul>



<p class="wp-block-paragraph">Field errors are the dangerous ones for ingestion, because they look exactly like success to anything that only inspects the transport layer. Worse, GraphQL&#8217;s null propagation rules mean a single failing non-null field bubbles up to the nearest nullable ancestor, so one broken resolver can hollow out an entire nested object while the surrounding rows land normally.</p>



<p class="wp-block-paragraph">The fix is to make the response body the contract. Classify before you commit anything:</p>



<pre class="wp-block-code"><code>import json, urllib.request

class RequestError(Exception): pass
class FieldError(Exception): pass

def graphql(endpoint, token, query, variables):
    body = json.dumps({"query": query, "variables": variables}).encode()
    req = urllib.request.Request(
        endpoint,
        data=body,
        headers={
            "Content-Type": "application/json",
            "Accept": "application/graphql-response+json, application/json",
            "Authorization": f"Bearer {token}",
        },
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        payload = json.loads(resp.read())

    # The status code told us nothing. The body is the contract.
    if "data" not in payload:
        # Nothing executed. Query or auth problem. Do not retry blindly.
        raise RequestError(payload.get("errors"))

    if payload.get("errors"):
        # Partial response. data is populated but incomplete.
        raise FieldError(payload["errors"])

    return payload["data"]</code></pre>



<p class="wp-block-paragraph">Two things worth explaining. The <code>Accept</code> header asks for <code>application/graphql-response+json</code>, the media type the GraphQL over HTTP spec recommends, which lets a compliant server return a real 4xx for pre-execution failures instead of burying them in a 200. Servers that don&#8217;t support it fall back to <code>application/json</code>, so including both is free.</p>



<p class="wp-block-paragraph">The second is the split between the two exception types. They need different handling. A request error is a bug in your query or a credential problem, and retrying it just burns budget on the same failure. A field error is transient often enough to be worth retrying, but never worth committing.</p>



<h3 class="wp-block-heading">Deciding what &#8220;partial&#8221; means for your data</h3>



<p class="wp-block-paragraph">Failing the whole page on any error is the safe default and the one I reach for first. But it&#8217;s not always right. If you are pulling a hundred fields and one optional enrichment field is flaky, refusing every page forever is its own outage.</p>



<p class="wp-block-paragraph">The workable middle ground is to classify by path. Every error object carries a <code>path</code> array pointing at the exact field that failed. Maintain a small allowlist of paths that are permitted to be null, fail the page on anything outside it, and emit the error count as a CloudWatch metric so a rising trend is visible before someone in finance notices.</p>



<h2 class="wp-block-heading">Failure two: rate limits that never send a 429</h2>



<p class="wp-block-paragraph">Your retry logic almost certainly triggers on 429 and 5xx. Several major GraphQL APIs will throttle you without ever sending one.</p>



<p class="wp-block-paragraph">GitHub&#8217;s GraphQL API is explicit about this: exceeding the primary rate limit returns a 200 response with an error message in the body and <code>x-ratelimit-remaining</code> set to zero. Your Step Functions retrier sees a success. Your Lambda sees no data. The run completes, having ingested nothing, and reports green.</p>



<p class="wp-block-paragraph">The other half of the problem is that request counting is the wrong unit. Shopify&#8217;s GraphQL Admin API and several others price queries by <em>calculated cost</em>, using a leaky bucket. The server statically analyses your query before executing it, charges the bucket the worst-case <em>requested</em> cost, executes, then refunds the difference between requested and <em>actual</em> cost. That has a consequence people miss: a query asking for the maximum connection size gets charged as if every page were full, even when the last page returns three records.</p>



<p class="wp-block-paragraph">The good news is that these APIs tell you the state of the bucket on every response, under the <code>extensions</code> key:</p>



<pre class="wp-block-code"><code>{
  "extensions": {
    "cost": {
      "requestedQueryCost": 72,
      "actualQueryCost": 38,
      "throttleStatus": {
        "maximumAvailable": 1000.0,
        "currentlyAvailable": 962.0,
        "restoreRate": 100.0
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Read those numbers, never hardcode them. Bucket size and restore rate vary by plan tier, and a query that works fine against a large merchant&#8217;s store will fail instantly against a small one. The pattern that holds up is a pre-flight check: before dispatching, compare your estimated cost against <code>currentlyAvailable</code>, and if you&#8217;re short, sleep for the shortfall divided by <code>restoreRate</code>.</p>



<p class="wp-block-paragraph">On APIs that expose their own budget as a queryable field, ask for it in the same round trip rather than spending a second call to find out:</p>



<pre class="wp-block-code"><code>query BudgetCheck {
  rateLimit {
    limit
    cost
    remaining
    resetAt
  }
}</code></pre>



<p class="wp-block-paragraph">Two more things about concurrency. First, if you fan out across Lambda invocations, each worker&#8217;s local view of the bucket is wrong, because the bucket is shared. Multiple workers independently deciding they have headroom is the classic way to get throttled while every individual worker believes it&#8217;s behaving. Either centralise the token accounting or cap concurrency low enough that the arithmetic can&#8217;t go wrong.</p>



<p class="wp-block-paragraph">Second, if the source offers a bulk or async export operation, use it for backfills. Paginating fifty thousand records through the interactive endpoint burns the same budget your incremental syncs need, and it competes with itself.</p>



<h2 class="wp-block-heading">Failure three: cursors that don&#8217;t survive a restart</h2>



<p class="wp-block-paragraph">Most production GraphQL schemas follow the Relay cursor connections convention: <code>first</code> and <code>after</code> arguments, a <code>pageInfo</code> object with <code>hasNextPage</code> and <code>endCursor</code>, and edges carrying opaque per-item cursors.</p>



<pre class="wp-block-code"><code>query Orders($cursor: String) {
  orders(first: 100, after: $cursor) {
    pageInfo {
      hasNextPage
      endCursor
    }
    edges {
      cursor
      node {
        id
        updatedAt
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">The filter argument for incremental sync is not standardised. Some schemas take a query-string filter, some take a typed timestamp argument, some expose nothing at all. Run introspection against the schema and find out rather than guessing, because a wrong argument name is a request error and your run will fail before it starts.</p>



<p class="wp-block-paragraph">The failure here is subtle. Cursors are opaque and their validity is entirely up to the server. Some are stable positional pointers. Some encode a sort key and a snapshot identifier and expire. If a page-47 fetch fails and you retry an hour later with the stored cursor, you may get an error, or you may get results from a shifted window with a hole in the middle. The second outcome is worse, because it looks like it worked.</p>



<p class="wp-block-paragraph">Three habits make restarts safe:</p>



<ol class="wp-block-list">
<li><strong>Checkpoint after the write, not after the fetch.</strong> A cursor recorded before the page lands in S3 is a promise you can&#8217;t keep.</li>

<li><strong>Store a high watermark alongside the cursor.</strong> The cursor is your fast path. The watermark, taken from the newest record you actually committed, is your fallback when the cursor is rejected.</li>

<li><strong>Overlap the watermark deliberately.</strong> Restart a few minutes before the last committed timestamp and deduplicate on the primary key downstream. Re-reading a handful of rows is cheap. Missing rows that arrived during a clock skew is not.</li>
</ol>



<p class="wp-block-paragraph">A DynamoDB item per source stream is enough state for all of this:</p>



<pre class="wp-block-code"><code>{
  "source": "orders",
  "run_id": "01hq7k...",
  "cursor": "eyJsYXN0X2lkIjoxMjM0fQ==",
  "high_watermark": "&lt;updatedAt of newest committed record&gt;",
  "pages_committed": 47,
  "status": "IN_PROGRESS"
}</code></pre>



<p class="wp-block-paragraph">Overlap only works if downstream deduplication is real. Write raw pages with an ingest-time partition and a run identifier in the object key, then let the curated layer resolve duplicates by primary key and latest <code>updatedAt</code>. If you write straight into a merged table with no dedupe step, overlapping restarts will double-count and you&#8217;ll have traded silent loss for silent inflation.</p>



<h2 class="wp-block-heading">Failure four: the schema moved and nobody told you</h2>



<p class="wp-block-paragraph">GraphQL&#8217;s deprecation story is genuinely better than REST&#8217;s. Fields carry a <code>@deprecated</code> directive with a reason string, and introspection exposes it. The problem is that nothing forces you to look.</p>



<p class="wp-block-paragraph">The failure mode is a field that gets deprecated, stops being populated, then eventually gets removed. Stage one is invisible: your query still validates, the field returns null, and your warehouse fills with nulls. Stage two is loud but late: the query becomes invalid and the whole ingest breaks at 3am.</p>



<p class="wp-block-paragraph">Put an introspection diff in CI. Fetch the schema on a schedule, store it as an artifact, and diff against the last known good version. Fail the build on removals or type changes to fields your queries actually select, and warn on new deprecations. Tools in the Apollo and Hasura ecosystems ship schema-diff tooling for this, and rolling your own against the introspection result is not much work either. Either way, the point is that the check runs on a schedule rather than when the pager goes off.</p>



<p class="wp-block-paragraph">The complementary habit is to keep queries narrow. Every field you select is a field that can break you, a field that costs budget, and a field that widens your Glue table. Selecting a whole object because it was convenient during development is a liability you carry indefinitely.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Wiring the GraphQL data ingestion pipeline on AWS</h2>



<p class="wp-block-paragraph">With the failure families understood, the AWS assembly is mostly a set of trade-offs about where the fetch loop lives.</p>



<h3 class="wp-block-heading">Where the fetch loop runs</h3>



<p class="wp-block-paragraph">Step Functions can call an HTTPS endpoint directly with an HTTP Task, no Lambda involved. It uses an EventBridge connection to hold the credentials, which means your token lives in a Secrets Manager secret managed by EventBridge rather than in an environment variable:</p>



<pre class="wp-block-code"><code>"FetchPage": {
  "Type": "Task",
  "Resource": "arn:aws:states:::http:invoke",
  "Parameters": {
    "ApiEndpoint": "https://api.example.com/graphql",
    "Method": "POST",
    "InvocationConfig": {
      "ConnectionArn": "arn:aws:events:region:account-id:connection/name/id"
    },
    "Headers": { "Content-Type": "application/json" },
    "RequestBody": {
      "query.$": "$.query",
      "variables.$": "$.variables"
    }
  },
  "Retry": [{
    "ErrorEquals": [
      "States.Http.StatusCode.429",
      "States.Http.StatusCode.502",
      "States.Http.StatusCode.503",
      "States.Http.StatusCode.504"
    ],
    "IntervalSeconds": 1,
    "BackoffRate": 2,
    "MaxAttempts": 3,
    "JitterStrategy": "FULL"
  }],
  "Next": "InspectBody"
}</code></pre>



<p class="wp-block-paragraph">The execution role needs four permissions: <code>states:InvokeHTTPEndpoint</code>, <code>events:RetrieveConnectionCredentials</code>, and both <code>secretsmanager:GetSecretValue</code> and <code>secretsmanager:DescribeSecret</code> scoped to the EventBridge-managed secret.</p>



<p class="wp-block-paragraph">It&#8217;s a clean pattern, and I like it for small, high-value syncs. Two constraints decide whether it fits. HTTP Task requests time out after 60 seconds, which a deeply nested GraphQL query can exceed without being pathological. And Step Functions reads the response within its standard state payload ceiling of roughly 256 KB, so a page of a hundred rich records will not fit. Also note the retrier above: it fires on HTTP status codes, so it will not catch a throttle that arrives as a 200 with an error in the body. That check has to happen in the state after it.</p>



<p class="wp-block-paragraph">The alternative is a Lambda driver that paginates internally and writes pages straight to S3, returning only the cursor and a count to the state machine. This dodges both limits, gives you a real HTTP client with real backoff, and keeps large payloads out of the workflow&#8217;s state. It costs you per-page retry granularity, which you buy back with the checkpoint record.</p>



<p class="wp-block-paragraph">Distributed Map is worth mentioning for the fan-out case, such as one child execution per tenant or per shard. It gives you high concurrency and a per-item execution history, which is genuinely useful for debugging. Be aware of two things before committing: express child workflows cap at five minutes rather than Lambda&#8217;s fifteen, and practitioners have published benchmarks showing throughput well below what the headline concurrency numbers suggest for small, fast items. For a lot of ingestion work, SQS plus a DynamoDB job table remains the simpler answer.</p>



<p class="wp-block-paragraph">One case genuinely falls outside serverless: sources that require a long-lived session, a persistent websocket subscription, or a synchronisation window longer than fifteen minutes with no resumable cursor. A small always-on worker on a modest VPS from a provider like InterServer or Hetzner, feeding the same S3 landing zone, is less clever and more honest than contorting Lambda around the constraint.</p>



<h3 class="wp-block-heading">The landing zone</h3>



<p class="wp-block-paragraph">Land raw responses first, unmodified, including the <code>errors</code> and <code>extensions</code> keys. Storage is the cheapest thing in this architecture and the raw layer is what lets you reprocess after you discover a parsing bug six weeks later.</p>



<ul class="wp-block-list">
<li><strong>Partition by ingest time, not event time.</strong> Event-time partitioning requires reading the record before you know where to put it, which means you can&#8217;t write until you&#8217;ve parsed. Ingest-time keys let the raw write be dumb and fast.</li>

<li><strong>Put the run identifier in the key.</strong> When you need to unwind one bad run, being able to delete or ignore by prefix beats writing a surgical delete.</li>

<li><strong>Define Glue tables explicitly rather than crawling.</strong> You wrote the selection set, so you already know the schema. A crawler inferring types from JSON will surprise you the first time an optional field is absent from an entire page.</li>

<li><strong>Convert to columnar in the curated layer.</strong> Athena over raw JSON works and is fine for debugging. Parquet is what you want for anything anyone queries repeatedly.</li>
</ul>



<p class="wp-block-paragraph">If your source pushes rather than being polled, such as webhook deliveries carrying GraphQL payloads, Amazon Data Firehose (renamed from Kinesis Data Firehose, with APIs and IAM policies unchanged) will buffer, convert to Parquet and partition on the way to S3 without you running a consumer. For scheduled pull-based syncs it adds a hop you don&#8217;t need.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Runs succeed but a column is all null.</strong> Field errors. Pull a raw response from the landing zone and look at the <code>errors</code> array. The <code>path</code> tells you exactly which field.</li>

<li><strong>Runs succeed but ingest zero rows.</strong> Likely a throttle returned as a 200. Log the response body and the rate-limit headers on empty results, always.</li>

<li><strong>Intermittent failures partway through pagination.</strong> Cursor expiry. Fall back to the watermark and check whether the source documents a cursor lifetime.</li>

<li><strong>Query fails immediately with no <code>data</code> key.</strong> Request error. Validate against current introspection; something in the schema moved.</li>

<li><strong>Throttled despite staying under the documented limit.</strong> Either concurrent workers are sharing a bucket they each think they own, or you&#8217;re being charged requested cost on near-empty pages.</li>

<li><strong>HTTP Task fails on large pages.</strong> The response exceeds the state payload ceiling. Reduce page size or move the fetch into Lambda.</li>

<li><strong>Row counts drift upward over time.</strong> Overlap without deduplication. Check that the curated layer actually resolves on the primary key.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating a 200 as success and never reading the body.</li>

<li>Retrying request errors. They are deterministic; retrying just spends budget to fail identically.</li>

<li>Hardcoding bucket sizes and restore rates that differ per plan tier and change without notice.</li>

<li>Checkpointing the cursor before the data is durable.</li>

<li>Requesting the maximum page size on every call while being charged the worst-case cost for it.</li>

<li>Selecting entire objects because it was easier during development.</li>

<li>Alerting only on execution failure, so a job that succeeds while ingesting nothing never pages anyone.</li>

<li>Overlapping restart windows with no downstream deduplication.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Classify every response into clean, partial, or request-failed before anything downstream sees it.</li>

<li>Persist raw responses with <code>errors</code> and <code>extensions</code> intact, then parse from the raw layer.</li>

<li>Alarm on row-count deltas and partial-error rate, not just on execution status.</li>

<li>Read the throttle state the API hands you and back off before you hit the wall, not after.</li>

<li>Keep both a cursor and a watermark, and know which one is authoritative on restart.</li>

<li>Diff the schema on a schedule so deprecations reach you before they reach your data.</li>

<li>Hold credentials in Secrets Manager or an EventBridge connection, never in a Lambda environment variable.</li>

<li>Trace the pipeline end to end. CloudWatch covers the basics; Grafana or Datadog earn their keep once you&#8217;re correlating child executions across a fan-out.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can Step Functions call a GraphQL API without a Lambda function?</h3>



<p class="wp-block-paragraph">Yes. An HTTP Task using the <code>arn:aws:states:::http:invoke</code> resource posts your query directly, with an EventBridge connection handling authentication. It works well for small pages. The 60-second request timeout and the state payload size ceiling are what push most real ingestion work back into Lambda.</p>



<h3 class="wp-block-heading">How do I detect GraphQL errors when the status code is always 200?</h3>



<p class="wp-block-paragraph">Inspect the body. If there&#8217;s no <code>data</code> key, nothing executed and you have a request error. If <code>data</code> is present alongside a non-empty <code>errors</code> array, you have a partial response and the data is incomplete. Only a response with <code>data</code> and no errors is unambiguously clean.</p>



<h3 class="wp-block-heading">Does AWS AppSync fit into this?</h3>



<p class="wp-block-paragraph">Only if you&#8217;re on the serving side. AppSync is a managed GraphQL API for exposing your own data, so it&#8217;s what you&#8217;d reach for if you wanted clients to push events in via mutations. For pulling data out of somebody else&#8217;s GraphQL API, AppSync isn&#8217;t part of the picture.</p>



<h3 class="wp-block-heading">Should I store raw GraphQL responses or parsed records in S3?</h3>



<p class="wp-block-paragraph">Both, in separate layers. Raw responses are your audit trail and your reprocessing path, and they&#8217;re the only way to answer &#8220;was this field null in the source, or did my parser drop it?&#8221; weeks after the fact. Parsed and columnar output goes in a curated prefix that analysts query.</p>



<h3 class="wp-block-heading">How do I run an incremental sync from a GraphQL API?</h3>



<p class="wp-block-paragraph">Find the schema&#8217;s filter argument for a modification timestamp using introspection, pass your stored watermark into it, and page forward with cursors. Overlap the window slightly on each run and deduplicate on the primary key downstream. If the schema exposes no such filter, you&#8217;re limited to full pulls or webhook-driven change capture.</p>



<h3 class="wp-block-heading">Why am I throttled when I&#8217;m making very few requests?</h3>



<p class="wp-block-paragraph">Cost-based APIs charge for query complexity, not call count. A single deeply nested query pulling a connection inside a connection can consume a large fraction of the bucket by itself. Read the cost figures the API returns and flatten the query if the requested cost is consistently much higher than the actual cost.</p>



<h3 class="wp-block-heading">Is GraphQL a worse ingestion source than REST?</h3>



<p class="wp-block-paragraph">Different, not worse. You control the payload, which cuts transfer and lets you avoid the N+1 fan-out REST forces on nested data. Deprecation metadata is machine-readable, which REST rarely offers. The cost is that error signalling and rate limiting both moved into the body, so naive clients fail silently instead of loudly.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A GraphQL data ingestion pipeline on AWS fails differently from a REST one, and the difference is almost entirely about where the truth lives. In REST, the transport tells you what happened. In GraphQL, the transport tells you the request arrived, and the body tells you what happened. Every failure family in this post traces back to that: partial errors hidden behind a 200, throttles hidden behind a 200, cursors that fail by returning the wrong rows rather than an error.</p>



<p class="wp-block-paragraph">So build the pipeline so that a green run means something. Parse before you commit, alarm on row deltas rather than exit codes, and keep the raw responses so you can prove what the source actually said. Everything else is ordinary AWS plumbing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with a GraphQL ingestion pipeline?</h2>



<p class="wp-block-paragraph">This is the kind of work I take on regularly. Specifically:</p>



<ul class="wp-block-list">
<li>Auditing an existing GraphQL sync for silent partial-error loss and telling you which columns are already affected</li>

<li>Building the fetch, checkpoint and land loop in Step Functions and Lambda with cursors that survive restarts</li>

<li>Fitting a cost-aware throttle to whatever budget model your source actually uses, including multi-worker coordination</li>

<li>Designing the raw and curated S3 layout, Glue table definitions and Athena partitioning so queries stay cheap</li>

<li>Adding introspection diffing to CI so schema deprecations surface in a pull request, not in a dashboard</li>

<li>Wiring CloudWatch metrics and alarms on row-count deltas and partial-error rates, so a green run is actually green</li>
</ul>



<p class="wp-block-paragraph">Send me a run log, a state machine definition, or a raw response with an <code>errors</code> array in it, and I&#8217;ll tell you what it&#8217;s costing you.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/">Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
