<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Schema Design | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/schema-design/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/schema-design/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Thu, 12 Nov 2026 07:09:08 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Schema Design | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/schema-design/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>GPS Data Gets Ugly Fast: A Fleet GPS Data Pipeline That Stays Queryable</title>
		<link>https://john-nessime.com/blog/data-engineering/fleet-gps-data-pipeline/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 12 Nov 2026 07:09:08 +0000</pubDate>
				<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Databases]]></category>
		<category><![CDATA[Logistics Technology]]></category>
		<category><![CDATA[Continuous Aggregates]]></category>
		<category><![CDATA[Data Partitioning]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Fleet Telematics]]></category>
		<category><![CDATA[Geofencing]]></category>
		<category><![CDATA[GNSS]]></category>
		<category><![CDATA[Hypertables]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Late-Arriving Data]]></category>
		<category><![CDATA[Map Matching]]></category>
		<category><![CDATA[MobilityDB]]></category>
		<category><![CDATA[PostGIS]]></category>
		<category><![CDATA[PostgreSQL]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Time-Series Data]]></category>
		<category><![CDATA[TimescaleDB]]></category>
		<category><![CDATA[Track and Trace]]></category>
		<category><![CDATA[Trajectory Data]]></category>
		<category><![CDATA[Valhalla]]></category>
		<category><![CDATA[Vehicle Data]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=714</guid>

					<description><![CDATA[<p>Telematics devices lie in specific, documented ways: stale fixes reported as new, zero coordinates after reboot, and buffered records replayed days late. Here is how to build a fleet GPS data pipeline on PostgreSQL, PostGIS and TimescaleDB that survives all three and still answers trip and geofence queries fast.</p>
<p>The post <a href="https://john-nessime.com/blog/data-engineering/fleet-gps-data-pipeline/">GPS Data Gets Ugly Fast: A Fleet GPS Data Pipeline That Stays Queryable</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">It usually starts with a screenshot. Someone in operations sends the live map with a van sitting in the middle of a field, or off the coast entirely, and asks whether the tracker is broken. It isn&#8217;t. The tracker is doing exactly what its firmware says it should. Nobody decided what a bad point looks like before the data was written down, so now there are eleven million rows and no way to tell the real ones from the noise.</p>



<p class="wp-block-paragraph">By the time it reaches you it looks like a query problem. The trip report takes ninety seconds. The mileage total disagrees with the odometer. A geofence alert fires for a vehicle that never moved. Those are symptoms. The cause sits upstream, in the shape of the table and in the decision, made by default, to store every record the device sent as though it were true.</p>



<p class="wp-block-paragraph">This post covers how I&#8217;d build a fleet GPS data pipeline that stays queryable as it grows: what the dirty data looks like at the source, how to split storage so cleaning stays reversible, how to keep the raw table from drowning in its own volume, and why nearly every question your users ask is a question about trips rather than points.</p>



<h2 class="wp-block-heading">The four ways GPS data goes ugly before you query it</h2>



<p class="wp-block-paragraph">Organise your defences by failure family, not by field. Each of these produces rows that are individually well-formed and collectively wrong.</p>



<h3 class="wp-block-heading">The device sends its last known position and calls it new</h3>



<p class="wp-block-paragraph">This is the one that bites hardest, because it is invisible. When a unit loses its fix, plenty of firmware does not send a null. Teltonika&#8217;s FMB documentation is explicit: when a record has no valid coordinates because there was no fix at acquisition time, the longitude, latitude and altitude fields carry the <em>last valid fix</em>, while angle, satellites and speed are set to zero. The row looks normal. It has a timestamp, it has coordinates, it parses.</p>



<p class="wp-block-paragraph">That is worse than a gap. A van parked underground for six hours emits hundreds of records claiming the same coordinates, and your &#8220;last seen&#8221; logic reports it sitting at the entrance the whole time. Then it drives out, gets a fix elsewhere, and the trip builder draws a straight line through everything in between.</p>



<p class="wp-block-paragraph">The signal you want is the satellite count and the dilution of precision, not the coordinates. Store both. Teltonika exposes a GPS Accuracy setting expressed as a minimum HDOP, PDOP or VDOP before the device refreshes its GPS data, plus a minimum satellite count, so the hardware can be told to be pickier before it transmits. Configure that, and still keep the fields, because you will want to re-derive the threshold later without re-collecting the fleet.</p>



<h3 class="wp-block-heading">Null Island, and the reboot that gets you there</h3>



<p class="wp-block-paragraph">Zero latitude and zero longitude is a real place off the coast of Ghana, and it is where a surprising share of any fleet appears to live. The same Teltonika documentation notes that after a reboot the device sends zero coordinates. A naive pipeline stores that as a point, and every map, bounding box and distance calculation now carries an outlier several thousand kilometres from the fleet. Trivially filterable; it survives into production because nobody looks until the map is built, and by then the filter has to be retrofitted into six queries.</p>



<h3 class="wp-block-heading">Records that arrive days late, in the wrong order</h3>



<p class="wp-block-paragraph">Telematics units buffer. A vehicle drives through a tunnel or a rural dead zone, keeps recording, and dumps the backlog on reconnect. Devices in this class also let you choose whether records are saved after a position fix, after a time sync, or always, which means you can receive records whose device clock was never synchronised at all.</p>



<p class="wp-block-paragraph">So the timestamp on the record and the moment you received it are two different facts, and you need both. Storing one is the most common design mistake I see in fleet ingest. Keep only the device timestamp and you cannot answer &#8220;what did the dashboard show at 4pm yesterday&#8221;, which is what every dispute becomes. Keep only the receive time and your trips are wrong.</p>



<h3 class="wp-block-heading">Parked-vehicle drift</h3>



<p class="wp-block-paragraph">A stationary receiver with a marginal sky view reports a slow random walk of a few metres at a time, all night, in a depot. Sum the distance between consecutive points and you have invented a three-kilometre trip. Multiply by a fleet and the mileage report is one nobody trusts.</p>



<p class="wp-block-paragraph">Drift is not caught by the same rule as teleporting. A jump filter catches physically impossible movement. Drift is physically possible, it just isn&#8217;t real. You need ignition state, reported speed or an accelerometer movement flag to tell &#8220;not moving&#8221; from &#8220;moving very slowly in traffic&#8221;.</p>



<h2 class="wp-block-heading">Split the fleet GPS data pipeline into three tables</h2>



<p class="wp-block-paragraph">The instinct is one positions table with cleaning applied on write. Resist it. Once you discard a record you cannot revisit the decision, and you will change your mind about the thresholds at least twice.</p>



<ol class="wp-block-list">
<li><strong>Raw.</strong> Everything the device sent, as sent, plus a receive timestamp. Append-only, never cleaned in place.</li>



<li><strong>Clean.</strong> Points that passed your quality rules, with a geometry column. Derived from raw and fully rebuildable from it.</li>



<li><strong>Trips.</strong> One row per journey, with the whole path as a single geometry. This is what nearly every user-facing query reads.</li>
</ol>



<p class="wp-block-paragraph">Raw is what makes the rest safe. When a mileage figure is disputed eight months from now, you can re-run the cleaning under the old rules and the new ones and show the difference. Two timestamps, quality fields as first-class columns, and anything device-specific in JSONB so new hardware doesn&#8217;t force a migration:</p>



<pre class="wp-block-code"><code>CREATE TABLE gps_raw (
    device_id   text        NOT NULL,
    recorded_at timestamptz NOT NULL,   -- the device's clock
    received_at timestamptz NOT NULL DEFAULT now(),
    lon         double precision,
    lat         double precision,
    speed_kph   double precision,
    heading     double precision,
    satellites  smallint,
    hdop        double precision,
    ignition    boolean,
    payload     jsonb       NOT NULL DEFAULT '{}'::jsonb
);</code></pre>



<p class="wp-block-paragraph">No primary key on device and time, deliberately. Devices resend. A unique constraint here turns a harmless duplicate into an ingest failure at three in the morning. Deduplicate on the way into the clean table, where a conflict is a no-op rather than an incident.</p>



<h2 class="wp-block-heading">Making the raw table survive its own growth</h2>



<p class="wp-block-paragraph">A few hundred vehicles reporting every few seconds produces a table that ordinary PostgreSQL handles for a while and then doesn&#8217;t, usually right after someone adds a date filter to a dashboard. Time partitioning is the fix, either through native declarative partitioning or the TimescaleDB extension.</p>



<pre class="wp-block-code"><code>SELECT create_hypertable('gps_raw', 'recorded_at',
                         chunk_time_interval =&gt; INTERVAL '1 day');</code></pre>



<p class="wp-block-paragraph">Pick the chunk interval from your query pattern, not your ingest rate. If dashboards ask for &#8220;today&#8221; and &#8220;this week&#8221;, daily chunks let the planner exclude almost everything. If reporting is monthly, daily chunks leave a lot of small chunks to plan across.</p>



<p class="wp-block-paragraph">Compression is where the storage argument is won, and the segment key is what people get wrong. Segmenting by vehicle groups each vehicle&#8217;s rows together inside a compressed batch, which is what keeps &#8220;one vehicle, one day&#8221; fast afterwards:</p>



<pre class="wp-block-code"><code>ALTER TABLE gps_raw SET (
    timescaledb.compress,
    timescaledb.compress_orderby   = 'recorded_at',
    timescaledb.compress_segmentby = 'device_id'
);</code></pre>



<h3 class="wp-block-heading">The licensing detail that catches you at runtime</h3>



<p class="wp-block-paragraph">TimescaleDB ships in two editions. The Apache 2.0 build gives you hypertables. Compression, continuous aggregates and retention policies live in the Community edition under the Timescale License, which is source-available rather than OSI-open-source. The vendor states the practical boundary plainly: as long as you are not offering TimescaleDB itself as a hosted database service, you can run the Community features free and self-hosted in production.</p>



<p class="wp-block-paragraph">The trap is that this fails at runtime, not at install. If your distribution or provider ships the Apache-licensed module, <code>CREATE EXTENSION</code> succeeds, the hypertable works, and then the <code>ALTER TABLE</code> above returns an error saying the functionality is not supported under the current licence. Zabbix users have filed bugs about exactly this. Check the edition before you design around compression, especially on managed PostgreSQL where you don&#8217;t choose the build.</p>



<p class="wp-block-paragraph">If that blocks you, native partitioning plus a scheduled job rolling old partitions into archive storage is a fine answer. More code, less magic, no licence question. The related fork is whether to self-host at all: a telemetry database is steady, disk-heavy and predictable, which is where a large VPS from Hetzner, Contabo or InterServer compares well against a managed hourly bill, and also where you inherit backups, upgrades and the extension build. My <a href="https://john-nessime.com/tools/vps-vs-managed-aws/">VPS versus managed AWS calculator</a> prices your own admin hours against the managed bill and returns a break-even rather than an opinion.</p>



<h3 class="wp-block-heading">Rollups, and the late records that quietly miss them</h3>



<p class="wp-block-paragraph">Continuous aggregates pre-compute the hourly numbers so dashboards stop scanning points:</p>



<pre class="wp-block-code"><code>CREATE MATERIALIZED VIEW vehicle_hourly
WITH (timescaledb.continuous) AS
SELECT device_id,
       time_bucket(INTERVAL '1 hour', recorded_at) AS bucket,
       count(*)        AS fixes,
       max(speed_kph)  AS max_speed_kph
FROM gps_clean
GROUP BY device_id, bucket;</code></pre>



<p class="wp-block-paragraph">Now connect that to buffering. The refresh policy has a window, and anything older than its start is never revisited:</p>



<pre class="wp-block-code"><code>SELECT add_continuous_aggregate_policy('vehicle_hourly',
    start_offset      =&gt; INTERVAL '7 days',
    end_offset        =&gt; INTERVAL '1 hour',
    schedule_interval =&gt; INTERVAL '1 hour');</code></pre>



<p class="wp-block-paragraph">A device out of coverage for ten days reconnects, replays its backlog, the rows land in the clean table, and the hourly rollup for those ten days stays exactly as wrong as it was. Raw is right, the dashboard is wrong, nothing errors. Set the start offset wider than your worst realistic backlog, and graph the gap between <code>recorded_at</code> and <code>received_at</code> per device model so you know what that worst case actually is. If Grafana already points at this database, it costs you one panel.</p>



<h2 class="wp-block-heading">Cleaning: two filters that do most of the work</h2>



<p class="wp-block-paragraph">Promotion from raw to clean is where the quality rules live. Make it an idempotent batch over a time window so any period can be re-run without duplicating anything.</p>



<pre class="wp-block-code"><code>INSERT INTO gps_clean (device_id, recorded_at, geom, speed_kph)
SELECT device_id,
       recorded_at,
       ST_SetSRID(ST_MakePoint(lon, lat), 4326),
       speed_kph
FROM gps_raw
WHERE recorded_at &gt;= $1
  AND recorded_at &lt;  $2
  AND lat IS NOT NULL
  AND lon IS NOT NULL
  AND NOT (lat = 0 AND lon = 0)
  AND satellites &gt;= 4
  AND hdop &lt;= 5
ON CONFLICT (device_id, recorded_at) DO NOTHING;</code></pre>



<p class="wp-block-paragraph">Treat those numbers as starting points to tune against your own hardware, not constants. Four satellites is the minimum for a three-dimensional fix; the dilution threshold depends on your receivers and your operating environment. Write the chosen values somewhere a human will find them, because in a year someone will ask why a specific point was dropped.</p>



<p class="wp-block-paragraph">The second filter is the jump test, which needs the previous row. Cast to <code>geography</code> so distance comes back in metres rather than degrees:</p>



<pre class="wp-block-code"><code>WITH stepped AS (
  SELECT device_id, recorded_at, geom,
         LAG(geom)        OVER w AS prev_geom,
         LAG(recorded_at) OVER w AS prev_at
  FROM gps_clean
  WHERE device_id = $1
    AND recorded_at &gt;= $2
  WINDOW w AS (PARTITION BY device_id ORDER BY recorded_at)
)
SELECT device_id,
       recorded_at,
       ST_Distance(geom::geography, prev_geom::geography) AS metres,
       EXTRACT(epoch FROM recorded_at - prev_at)          AS seconds
FROM stepped
WHERE prev_geom IS NOT NULL
  AND ST_Distance(geom::geography, prev_geom::geography)
      / NULLIF(EXTRACT(epoch FROM recorded_at - prev_at), 0) &gt; 60;</code></pre>



<p class="wp-block-paragraph">Sixty metres per second is a little over 200 km/h. No delivery van does that, so anything above it is a bad fix, a clock problem, or two vehicles sharing a device identifier. Flag them rather than deleting them; the pattern tells you which of the three you have.</p>



<h2 class="wp-block-heading">Trips are the abstraction that makes queries fast</h2>



<p class="wp-block-paragraph">Users don&#8217;t ask point questions. They ask trip questions: how far did this vehicle go, when did it enter the site, how long was it stopped, which vehicles passed through this area. Answering those from a point table means aggregating thousands of rows per answer, every time.</p>



<p class="wp-block-paragraph">PostGIS has a native representation for this. A trajectory is a LINESTRING whose M dimension carries time, with M increasing from each vertex to the next, and <code>ST_IsValidTrajectory</code> checks exactly that. Once your paths are valid trajectories the spatio-temporal functions come free: <code>ST_ClosestPointOfApproach</code> returns the measure at which two moving objects were nearest, <code>ST_DistanceCPA</code> returns how near, and <code>ST_CPAWithin</code> gives the boolean. That turns &#8220;were these two vehicles ever within fifty metres of each other&#8221; into one function call instead of a self-join across a point table.</p>



<pre class="wp-block-code"><code>CREATE TABLE trips (
    trip_id     bigserial PRIMARY KEY,
    device_id   text        NOT NULL,
    started_at  timestamptz NOT NULL,
    ended_at    timestamptz NOT NULL,
    path        geometry(LineStringM, 4326) NOT NULL,
    distance_m  double precision NOT NULL,
    point_count integer     NOT NULL
);

CREATE INDEX trips_path_gix ON trips USING GIST (path);
CREATE INDEX trips_device_time_idx ON trips (device_id, started_at DESC);</code></pre>



<p class="wp-block-paragraph">Building the path is an aggregate over the clean points, with epoch seconds as the M value:</p>



<pre class="wp-block-code"><code>SELECT ST_MakeLine(
         ST_MakePointM(ST_X(geom), ST_Y(geom),
                       EXTRACT(epoch FROM recorded_at))
         ORDER BY recorded_at
       ) AS path
FROM gps_clean
WHERE device_id = $1
  AND recorded_at &gt;= $2
  AND recorded_at &lt;  $3;</code></pre>



<p class="wp-block-paragraph">Segmenting points into trips is a business decision dressed as a technical one. Ignition on to ignition off is cleanest where the wiring supports it; otherwise speed below a floor for longer than a gap threshold ends the trip. Whichever you pick, store the rule as a column on the trip, because someone will change it and you&#8217;ll need to know which rows were built under which version.</p>



<p class="wp-block-paragraph">For rendering, keep a simplified two-dimensional copy. <code>ST_SimplifyPreserveTopology</code> runs Douglas-Peucker without collapsing the line into nonsense, and a route drawn at city zoom doesn&#8217;t need a vertex every two seconds. Simplify for display only, never for the geometry you measure distance from.</p>



<h2 class="wp-block-heading">Map matching: when to snap, and when to skip it</h2>



<p class="wp-block-paragraph">Snapping traces to the road network fixes a cosmetic problem, paths wandering across the pavement, and a real one, distance summed from raw points overstating the journey. Valhalla&#8217;s Meili implements the well-known hidden Markov approach to matching noisy, sparse traces. Its API exposes two actions: <code>trace_route</code> returns the shape snapped to the road network with directions, and <code>trace_attributes</code> returns per-segment attribution along the matched path. Requests should be POSTed, because an encoded trace gets large. OSRM&#8217;s match service solves the same problem with a similar model.</p>



<pre class="wp-block-code"><code>curl -s -X POST https://valhalla.internal.example/trace_attributes 
  -H 'Content-Type: application/json' 
  -d '{
        "shape": [
          {"lat": 52.0907, "lon": 5.1214},
          {"lat": 52.0912, "lon": 5.1229}
        ],
        "costing": "auto",
        "shape_match": "map_snap"
      }'</code></pre>



<p class="wp-block-paragraph">Two things decide whether it&#8217;s worth running. First, trace density: Valhalla&#8217;s own guidance is to keep points roughly between one per second and one per ten seconds, and warns the match degrades as that spacing widens. If your devices report every two minutes to save data, matching will confidently snap you to plausible roads you never used, which is worse than an obviously wobbly line because it looks correct.</p>



<p class="wp-block-paragraph">Second, what the number is for. Billing, driver disputes and compliance want the matched path. An internal ops map does not. Matching is either a per-trip network call or a self-hosted routing service with a tile build to maintain, and it&#8217;s the most operationally expensive piece in this design. If no money rides on the number, skip it and say so.</p>



<p class="wp-block-paragraph">Self-hosting, Valhalla and OSRM both run fine in containers against an OpenStreetMap extract, on a separate box from the one serving queries because the tile build is memory-hungry. Worth pasting that stack into my <a href="https://john-nessime.com/tools/docker-compose-production-checker/">Docker Compose production checker</a> first, since routing containers are exactly the kind of thing that quietly exposes a port or writes tiles to a volume that dies on restart.</p>



<h2 class="wp-block-heading">Geofences that actually use the index</h2>



<p class="wp-block-paragraph">Geofence queries are where a working platform turns slow, and the cause is nearly always a function that can&#8217;t use the spatial index. <code>ST_Intersects</code>, <code>ST_Within</code> and <code>ST_DWithin</code> are index-assisted. <code>ST_Distance</code> is not. Writing <code>WHERE ST_Distance(a, b) &lt; 500</code> gets the right answer by computing distance for every candidate row.</p>



<pre class="wp-block-code"><code>-- Polygon fences: which trips crossed a depot this week
SELECT g.name, t.trip_id, t.device_id
FROM trips t
JOIN geofences g ON ST_Intersects(t.path, g.area)
WHERE t.started_at &gt;= now() - INTERVAL '7 days';

-- Radius fences: within 500 m of a point, using the index
SELECT t.trip_id
FROM trips t
WHERE ST_DWithin(t.path::geography, $1::geography, 500);</code></pre>



<p class="wp-block-paragraph">Running fences against trips rather than points is the other half of the win. A hundred fences against ten million points is a hundred million comparisons. A hundred fences against fifty thousand trip geometries, each with a bounding box the index rejects in one step, is a different query entirely.</p>



<p class="wp-block-paragraph">The catch is latency. Trips only exist once they end, so live entry and exit alerts still run against incoming points. Keep that path separate and deliberately small: active fences only, evaluated as records arrive. Don&#8217;t make one query serve both the live alert and the monthly report.</p>



<h2 class="wp-block-heading">Troubleshooting a fleet data platform that has gone wrong</h2>



<p class="wp-block-paragraph"><strong>Mileage disagrees with the odometer.</strong> Compare summed point-to-point distance against the matched-path distance for the same trip. Consistently higher means drift or noise, so tighten the quality filter and check whether stationary periods are included. Lower means missing points, with gaps bridged by straight lines.</p>



<p class="wp-block-paragraph"><strong>A dashboard number is wrong but the raw data is right.</strong> Suspect the rollup window first. Query the raw table and the continuous aggregate for the same bucket and compare. A difference means late data outside the refresh offset.</p>



<p class="wp-block-paragraph"><strong>A vehicle sits still for hours, then teleports.</strong> Check satellites and HDOP across the stationary block. Zero satellites throughout means repeated stale fixes, not a parked vehicle.</p>



<p class="wp-block-paragraph"><strong>Queries slowed suddenly with no traffic change.</strong> Check whether the planner is excluding chunks. A predicate wrapped in a function on the time column, or a parameter type that doesn&#8217;t match, prevents partition pruning and silently turns a one-day scan into a full-table scan. Run <code>EXPLAIN</code> and count the chunks in the plan.</p>



<p class="wp-block-paragraph"><strong>Two vehicles occupy the same position.</strong> Almost always a device moved between vehicles without the assignment history being updated. Model the device-to-vehicle link as a table with validity intervals, not a column on the vehicle.</p>



<p class="wp-block-paragraph">One failure worth rehearsing before it happens is the ingest endpoint going down. Devices buffer, so a short outage survives and a long one becomes a thundering replay on recovery. If you&#8217;ve never asked what breaks when the database node dies mid-shift, my <a href="https://john-nessime.com/tools/cloud-architecture-assessment/">cloud architecture assessment</a> scores where the single points of failure sit.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Cleaning on ingest and keeping nothing else. The rules will change and you&#8217;ll have no way to apply them retroactively.</li>



<li>Storing latitude and longitude as plain numeric columns. Fine until the first spatial query, then you&#8217;re building indexes on a table you can&#8217;t afford to lock.</li>



<li>Treating the device timestamp as authoritative. Unsynchronised clocks and buffered replays make it a claim, not a fact.</li>



<li>Using <code>ST_Distance</code> in a WHERE clause instead of <code>ST_DWithin</code>. Same answer, no index.</li>



<li>Designing around compression or continuous aggregates without checking which TimescaleDB edition is installed.</li>



<li>Map matching a sparse trace and trusting the output. Confident wrong answers are harder to catch than obvious ones.</li>



<li>Putting live alerting and historical reporting through the same queries.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Keep raw append-only and every downstream table rebuildable from it. Test that rebuild before you need it.</li>



<li>Store both timestamps on every record, and alert on the lag between them per device model.</li>



<li>Keep the quality fields after filtering on them. Satellites and HDOP are how you re-derive a threshold later.</li>



<li>Match the chunk interval to how people query, and revisit it when the reporting cadence changes.</li>



<li>Answer user questions from trips. Reach for points only when you genuinely need sub-trip resolution.</li>



<li>Record the segmentation rule and quality thresholds as data, not as code comments.</li>



<li>Retire raw data on a written retention schedule rather than when the disk fills.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need TimescaleDB, or is plain PostgreSQL enough?</h3>



<p class="wp-block-paragraph">Plain PostgreSQL with native partitioning and PostGIS carries a small fleet a long way, with no licensing question attached. TimescaleDB earns its place when you want compression and continuous aggregates without maintaining that machinery yourself. It saves you code, not capability.</p>



<h3 class="wp-block-heading">Should I use PostGIS geometry or geography for GPS points?</h3>



<p class="wp-block-paragraph">Store as <code>geometry</code> in SRID 4326 and cast to <code>geography</code> for distance and radius work. Geometry has broader function coverage and cheaper operations; geography gives metres on a sphere instead of degrees. Casting at the point of use is the pragmatic middle.</p>



<h3 class="wp-block-heading">How long should I keep raw GPS points?</h3>



<p class="wp-block-paragraph">Long enough to cover the dispute window your business actually faces, which is driven by employment law, insurance or customer contracts rather than by engineering. Decide it with whoever owns that risk, write it down, enforce it with a retention policy. Trips are far smaller and usually worth keeping much longer.</p>



<h3 class="wp-block-heading">What about ClickHouse or InfluxDB instead?</h3>



<p class="wp-block-paragraph">Both are genuinely strong at high-volume ingest and time-range scans, and for a purely time-series workload I&#8217;d take either seriously. The reason I reach for PostgreSQL here is that fleet data isn&#8217;t purely time-series: it joins constantly to vehicles, drivers, contracts and sites, and it needs spatial predicates in the same query as the time filter. One database beats two and a reconciliation job. If your spatial needs are trivial and your volume is enormous, that flips.</p>



<h3 class="wp-block-heading">Is MobilityDB worth adding?</h3>



<p class="wp-block-paragraph">MobilityDB extends PostgreSQL and PostGIS with proper moving-object types, <code>tgeompoint</code> and <code>tgeogpoint</code>, that interpolate between observations so you can query a position at any instant. It&#8217;s an OSGeo project and the modelling is better than hand-rolled trajectories. The cost is another extension to install and upgrade, including <code>shared_preload_libraries</code> and locking configuration on the server. I&#8217;d add it when trajectory analysis is the product, not when it&#8217;s a reporting feature.</p>



<h3 class="wp-block-heading">How do I stop parked vehicles generating fake mileage?</h3>



<p class="wp-block-paragraph">Don&#8217;t rely on distance alone, because drift is physically plausible. Use ignition state or a movement flag to mark stationary periods, exclude those points from distance summation, and keep them in raw so the stop duration stays auditable.</p>



<h3 class="wp-block-heading">Where does a fleet GPS data pipeline break first at scale?</h3>



<p class="wp-block-paragraph">The dashboard queries, and nearly always for the same reason: something in the predicate stopped the planner pruning partitions. Before adding hardware, run <code>EXPLAIN</code> and confirm how many chunks the query touches. Adding vehicles rarely breaks a partitioned table. Bad predicates always do.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A fleet GPS data pipeline stays queryable because of two decisions made early, not because of the database you picked. Keep the raw record untouched so every cleaning rule stays reversible, and answer questions from trips rather than points so the expensive aggregation happens once instead of on every page load.</p>



<p class="wp-block-paragraph">Everything else is tuning. Chunk intervals, compression settings and index choices all matter, and all of them are recoverable. Throwing away the record of what the device actually sent is the only mistake here you can&#8217;t undo later.</p>



<h2 class="wp-block-heading">Where I can help</h2>



<p class="wp-block-paragraph">Most fleet data work arrives as a number somebody disputes, and the fix lives three layers upstream of the report. Things I&#8217;d take on here:</p>



<ul class="wp-block-list">
<li>Reviewing an existing positions schema and working out what it takes to make it partition-friendly without a full rewrite.</li>



<li>Designing the raw, clean and trip split, including the idempotent promotion job so any window can be safely re-run.</li>



<li>Tracking down why a rollup and the raw data disagree, which is usually late records falling outside a refresh window.</li>



<li>Building quality filters against your actual hardware, so thresholds fit your receivers rather than someone else&#8217;s blog post.</li>



<li>Getting geofence and trip queries onto the spatial index, with the EXPLAIN output to show what changed.</li>



<li>Standing up self-hosted map matching, and deciding honestly if you need it at all.</li>
</ul>



<p class="wp-block-paragraph">If any of that is live for you, send the concrete thing: a table definition, the <code>EXPLAIN</code> output from the slow query, or a day of records from one device. That&#8217;s usually enough to say something useful straight away.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://john-nessime.com/projects/data-pipeline-review/">Data pipeline rerun review</a></div>



<div class="wp-block-button is-style-outline"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/data-engineering/fleet-gps-data-pipeline/">GPS Data Gets Ugly Fast: A Fleet GPS Data Pipeline That Stays Queryable</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building an Ecommerce Recommendation Pipeline on AWS: The Parts That Bite</title>
		<link>https://john-nessime.com/blog/cloud-computing/aws-ecommerce-recommendation-pipeline/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 26 Oct 2026 06:37:42 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Machine Learning]]></category>
		<category><![CDATA[Retail Technology]]></category>
		<category><![CDATA[Amazon Personalize]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Batch Processing]]></category>
		<category><![CDATA[Clickstream Ingestion]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Ecommerce Analytics]]></category>
		<category><![CDATA[Idle Resources]]></category>
		<category><![CDATA[Item Cold Start]]></category>
		<category><![CDATA[Kinesis Data Streams]]></category>
		<category><![CDATA[Model Serving]]></category>
		<category><![CDATA[Recommendation Systems]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Training-Serving Skew]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=894</guid>

					<description><![CDATA[<p>An idle Amazon Personalize setup does not throw errors. It just bills. A practical walkthrough of the five decisions that shape an AWS ecommerce recommendation pipeline: domain recommenders versus custom recipes, bulk versus streaming ingestion, schema limits, training cadence, and real-time versus batch serving, with the failure modes that stay invisible until the invoice arrives.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/aws-ecommerce-recommendation-pipeline/">Building an Ecommerce Recommendation Pipeline on AWS: The Parts That Bite</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question usually arrives from finance rather than from engineering. Something along the lines of: why is there an Amazon Personalize charge on an account where nobody has shipped a recommendation carousel yet?</p>



<p class="wp-block-paragraph">The answer is almost always the same. Someone followed a tutorial, created a dataset group, trained a solution, deployed a campaign to see if the output looked reasonable, then got pulled onto something else. The campaign is still holding a throughput floor. The solution is still retraining on a schedule. Neither is wired to a storefront. Both are billing.</p>



<p class="wp-block-paragraph">I want that failure mode first because it is invisible. A broken AWS ecommerce recommendation pipeline throws errors and gets fixed by Friday. An idle one throws nothing. It sits there being technically correct and quietly expensive.</p>



<p class="wp-block-paragraph">This post covers the five decisions that actually shape the pipeline: which resource type to train, how events get in, what the schema can hold, how fresh the model needs to be, and how a recommendation reaches the page. I will flag the spots where the default behavior is not the behavior you want.</p>



<h2 class="wp-block-heading">The bill arrives before the carousel does</h2>



<p class="wp-block-paragraph">Two things in Personalize cost money on a clock rather than on demand, and both default toward spending.</p>



<p class="wp-block-paragraph"><strong>Solution training.</strong> New custom solutions use automatic training by default. Personalize creates a new solution version every seven days unless you say otherwise, and the cadence is configurable between one and thirty days. The first run starts within an hour of the solution going active. AWS is explicit that you incur training costs while the solution is active, and that switching automatic training off is how you stop them.</p>



<p class="wp-block-paragraph"><strong>Inference capacity.</strong> A campaign has <code>minProvisionedTPS</code>, a domain recommender has <code>minRecommendationRequestsPerSecond</code>. Both default to 1. Both set a billing floor. Personalize scales above the floor as traffic climbs but never below it, and you are billed on the higher of the floor and your actual rate. A campaign serving zero requests still bills its floor for every hour it stays active.</p>



<p class="wp-block-paragraph">Two habits fix most of this. While you are evaluating rather than serving, create solutions with automatic training off and trigger training yourself. And delete campaigns you are not calling instead of leaving them parked.</p>



<pre class="wp-block-code"><code>import boto3

personalize = boto3.client('personalize')

# Automatic training off while evaluating.
# You then call create_solution_version() yourself when you want a model.
response = personalize.create_solution(
    name='storefront-user-personalization',
    datasetGroupArn='arn:aws:personalize:REGION:ACCOUNT:dataset-group/NAME',
    recipeArn='arn:aws:personalize:::recipe/aws-user-personalization-v2',
    performAutoTraining=False
)
print(response['solutionArn'])</code></pre>



<h2 class="wp-block-heading">Decision one: domain recommender or custom solution</h2>



<p class="wp-block-paragraph">A domain dataset group created with the ECOMMERCE domain exposes pre-configured use cases. A custom dataset group makes you pick a recipe and configure the solution yourself.</p>



<p class="wp-block-paragraph">The ECOMMERCE use cases map onto the modules you were going to build anyway: Recommended for you, Frequently bought together, Customers who viewed X also viewed, Best sellers, Most viewed. Each carries an event-type requirement. The purchase-driven ones need at minimum a thousand Purchase events; the view-driven ones need at minimum a thousand View events. That is a useful early check on your event taxonomy, before you spend a training run finding out.</p>



<p class="wp-block-paragraph">Where the domain route wins: you skip the recipe argument, the defaults are sensible, and you reach a testable result faster. For a catalog under a few thousand SKUs with a normal retail event mix, this is the one I reach for first.</p>



<p class="wp-block-paragraph">Where it hurts: you are constrained to the shape of the provided use cases, you get less control over hyperparameters, and you cannot create next best action resources inside a domain dataset group at all. If you need to recommend actions rather than products, you are in custom territory.</p>



<p class="wp-block-paragraph">On the custom side the recipes worth knowing are User-Personalization-v2 for a personalized feed, Similar-Items for a related-products rail, Personalized-Ranking-v2 for reordering a category or search page, Trending-Now for what is moving, and Popularity-Count as the baseline you should measure against. The v2 recipes use a transformer architecture, handle much larger catalogs, and return item metadata in the inference response, saving a catalog lookup on the render path. They also bill on a different model from the older recipes, so do not carry a cost estimate across.</p>



<p class="wp-block-paragraph">One thing to watch while reading around: the legacy HRNN recipes are no longer available. Their documentation stays online for reference and plenty of third-party tutorials still use them. If a guide points a solution at <code>aws-hrnn</code>, its advice about cold items and training cadence is stale too.</p>



<h2 class="wp-block-heading">Decision two: how interaction data gets in</h2>



<h3 class="wp-block-heading">Bulk imports replace unless you tell them not to</h3>



<p class="wp-block-paragraph">This is the one that costs people a month of confused model quality.</p>



<p class="wp-block-paragraph">A dataset import job defaults to <code>FULL</code>, which overwrites all existing bulk data in the dataset. It leaves records you sent individually through the event tracker alone, but everything previously loaded from S3 is replaced by whatever is in this job. The alternative is <code>INCREMENTAL</code>, which appends and only replaces records sharing an ID.</p>



<p class="wp-block-paragraph">Picture a nightly export producing a rolling thirty-day extract, fed into an import job left on the default. Every night the dataset is truncated to thirty days. Nothing errors. The job reports success. Model quality just degrades as the training window shrinks, and it is not obvious why, because the pipeline is green.</p>



<pre class="wp-block-code"><code>import boto3

personalize = boto3.client('personalize')

# INCREMENTAL appends to what is already there.
# FULL, the default, replaces all previously bulk-imported records.
personalize.create_dataset_import_job(
    jobName='interactions-nightly',
    datasetArn='arn:aws:personalize:REGION:ACCOUNT:dataset/NAME/INTERACTIONS',
    dataSource={'dataLocation': 's3://your-bucket/interactions/'},
    roleArn='arn:aws:iam::ACCOUNT:role/PersonalizeS3Role',
    importMode='INCREMENTAL'
)</code></pre>



<p class="wp-block-paragraph">Two limits shape the schedule. An incremental interactions import is capped well below a full one, so treat it as a delta mechanism and not a backfill. And an interactions import job needs a minimum number of new records to be accepted, so a low-traffic store cannot run one every fifteen minutes and expect success. Batch the deltas into a cadence that clears the floor.</p>



<h3 class="wp-block-heading">Streaming events through an event tracker</h3>



<p class="wp-block-paragraph">The live path is a separate API. Create an event tracker, take the tracking ID it returns, and call PutEvents against the <code>personalize-events</code> client.</p>



<pre class="wp-block-code"><code>import boto3
import time

personalize_events = boto3.client('personalize-events')

personalize_events.put_events(
    trackingId='YOUR_TRACKING_ID',
    userId='user-8842',
    sessionId='sess-19f3ac',
    eventList=[{
        'eventId': 'evt-1',
        'eventType': 'View',
        'itemId': 'SKU-4471',
        'sentAt': int(time.time())
    }]
)</code></pre>



<p class="wp-block-paragraph">Do not put that call inline in the request path of a product page. Recommendation telemetry is not worth a network round trip on your time to first byte, and it is certainly not worth a page error when the API has a bad minute. Push events onto Kinesis Data Streams or Amazon Data Firehose, drain with Lambda, call PutEvents from there.</p>



<p class="wp-block-paragraph">That indirection buys a second thing you want anyway: Firehose can land a durable copy of the same events in S3, which becomes the source for bulk imports and for analysis outside Personalize. Events that only ever existed inside a managed service are events you cannot re-examine when the model does something odd.</p>



<p class="wp-block-paragraph">The <code>sessionId</code> field matters more than it looks. It lets you record anonymous browsing before login and have Personalize associate that session with the user afterward. Skip it and you train on your smallest, least representative slice of traffic.</p>



<h2 class="wp-block-heading">Decision three: schema and event types, decided once</h2>



<p class="wp-block-paragraph">Personalize schemas are Avro. An interactions schema needs a user ID, an item ID and a timestamp. An event type field is optional, but the domain use cases key off it, so you want it.</p>



<pre class="wp-block-code"><code>{
  "type": "record",
  "name": "Interactions",
  "namespace": "com.amazonaws.personalize.schema",
  "fields": [
    { "name": "USER_ID",    "type": "string" },
    { "name": "ITEM_ID",    "type": "string" },
    { "name": "EVENT_TYPE", "type": "string" },
    { "name": "TIMESTAMP",  "type": "long" }
  ],
  "version": "1.0"
}</code></pre>



<p class="wp-block-paragraph">The budget for extra fields is tighter than people expect. An interactions dataset allows only a handful of metadata columns beyond the reserved fields, and the count of distinct event types plus those metadata columns is itself capped. That is a hard design constraint. You cannot ship twenty-eight granular event types and also carry rich per-event context.</p>



<p class="wp-block-paragraph">Spend it deliberately. View, AddToCart and Purchase carry most retail pipelines. Wishlist adds earn a slot if customers use them. Hover, scroll depth and image zoom do not, because they inflate volume without changing what the model learns about intent. Once you have several event types you can weight them differently at training time, which beats the alternative people reach for first, which is dropping views entirely.</p>



<p class="wp-block-paragraph">Before any of that, settle the item ID question. Variants and parent products are different objects in Shopify and WooCommerce alike, and a model trained on variant IDs will happily suggest the blue medium of a shirt the customer already owns in blue medium. Pick a level, write it down, use the same one in your bulk extract and your live payload. Getting those out of step is the most common reason a technically healthy pipeline produces recommendations that feel broken.</p>



<p class="wp-block-paragraph">On volume: the documented floor is a thousand item interactions from at least twenty-five unique users with two or more interactions each, plus a minimum share of your user base clearing that two-interaction bar. AWS recommends considerably more for quality. Near the minimum, the honest read is that you are not building a recommendation pipeline yet. You are building the event capture that lets you build one later, and a popularity-ranked rail will serve customers better meanwhile.</p>



<p class="wp-block-paragraph">Timestamps are Unix time. Mixed timezones, or milliseconds where seconds are expected, will not throw. They will scramble the ordering the model learns from, which is the entire signal.</p>



<h2 class="wp-block-heading">Decision four: freshness without a retraining bill</h2>



<p class="wp-block-paragraph">Two mechanisms get conflated constantly, and the difference sets your monthly cost.</p>



<p class="wp-block-paragraph"><strong>Automatic training</strong> creates a new solution version. The model learns from your latest data. It costs a training run each time.</p>



<p class="wp-block-paragraph"><strong>Automatic updates</strong> are something else entirely. For User-Personalization-v2, User-Personalization and Next-Best-Action, and for the Top picks for you and Recommended for you domain use cases, Personalize refreshes the deployed model every two hours so new items can surface through exploration. That is not retraining. The model does not learn from your recent interactions. It just stops new products from being structurally invisible.</p>



<p class="wp-block-paragraph">So automatic updates solve the cold item problem and do nothing about model drift. AWS&#8217;s own guidance is that even with them, training cadence should stay at least weekly.</p>



<p class="wp-block-paragraph">Trending-Now is the interesting exception. It evaluates interactions over a configurable interval and identifies what is climbing, without waiting for a training run. For a store with promotional spikes or seasonal churn, a Trending-Now rail beside a personalized rail covers a lot of ground for very little retraining.</p>



<p class="wp-block-paragraph">The trap is Similar-Items, which has no automatic updates. Add products daily, run a related-products rail on Similar-Items, and new products only appear after the next training. Same-day visibility means daily training. That is a real budget line, and it is the point where catalog velocity stops being a merchandising detail and becomes an infrastructure decision.</p>



<h2 class="wp-block-heading">Decision five: how the recommendation reaches the page</h2>



<p class="wp-block-paragraph">Real-time means a campaign or a recommender answering GetRecommendations. Batch means an inference job reading input from S3 and writing recommendations back to S3, with no campaign involved and therefore no throughput floor.</p>



<p class="wp-block-paragraph">Most small and mid-sized stores reach for real-time by reflex and do not need it. Ask what the module actually is. A homepage rail refreshed daily, an email send, a merchandising export: those are batch problems. Run the batch overnight, load results into DynamoDB or a Redis instance, serve the carousel from a key lookup. Single-digit millisecond reads, no per-request inference charge, and nothing sitting idle at three in the morning.</p>



<p class="wp-block-paragraph">Real-time earns its cost where the recommendation has to react inside the session, which in practice means product detail and cart pages.</p>



<p class="wp-block-paragraph">Two things belong in the serving layer from day one either way. Filters, so you can exclude out-of-stock items and things the customer already bought; note that Personalize considers a bounded number of interactions per user per event type when filtering, and a user with no imported interactions for several months stops counting toward filtering at all. And a timeout with a fallback, because a carousel must never be able to delay a product page. Fall back to a cached bestseller list and log the fallback rate as a metric, so a rise shows up in CloudWatch or Grafana rather than in a conversion report a fortnight later.</p>



<h2 class="wp-block-heading">Troubleshooting an AWS ecommerce recommendation pipeline</h2>



<ul class="wp-block-list">
<li><strong>Everyone gets the same recommendations.</strong> Usually too few interactions per user, or user IDs that are not stable across sessions. Check that anonymous activity is recorded with a session ID and reconciled at login.</li>



<li><strong>New products never show up.</strong> Check that your recipe supports automatic updates. If it does not, this is a training cadence problem, not a data problem.</li>



<li><strong>Training fails on data volume.</strong> Compare against the documented minimums. The share of users with two or more interactions catches people out: plenty of users is not the same as plenty of returning users.</li>



<li><strong>The dataset shrank and nothing errored.</strong> A bulk import running in FULL mode against a rolling extract. Check the import mode on every scheduled job.</li>



<li><strong>Recommendations ignore what the user just clicked.</strong> Confirm events reach the event tracker, and that the user ID sent to PutEvents matches the one sent to GetRecommendations exactly.</li>



<li><strong>The bill rose during a quiet week.</strong> Automatic training runs on a calendar, not on traffic. Check cadence, and check for campaigns nobody is calling.</li>



<li><strong>Filters are not excluding purchased items.</strong> Filtering looks at a bounded slice of each user&#8217;s history per event type. Heavy users push older purchases outside it.</li>
</ul>



<p class="wp-block-paragraph">Auditing an account you did not build starts with listing what exists, not with reading the application code.</p>



<pre class="wp-block-code"><code>aws personalize list-dataset-groups
aws personalize list-solutions --dataset-group-arn ARN
aws personalize list-campaigns --solution-arn ARN
aws personalize list-recommenders --dataset-group-arn ARN

# Then see what each campaign is holding open
aws personalize describe-campaign --campaign-arn ARN</code></pre>



<p class="wp-block-paragraph">Orphaned campaigns from an abandoned proof of concept show up in that output in about ninety seconds.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Leaving automatic training on for a solution that exists only to satisfy curiosity.</li>



<li>Raising the provisioned throughput floor for a traffic spike, then never lowering it afterward.</li>



<li>Modelling on variant IDs in the bulk extract and product IDs in the live events, or the reverse.</li>



<li>Recording events only for authenticated users, starving the model of most of your traffic.</li>



<li>Designing a rich event taxonomy that collides with the cap on event types plus metadata columns.</li>



<li>Calling PutEvents synchronously on the render path.</li>



<li>Following an implementation guide that still references the retired HRNN recipes.</li>



<li>Shipping with no fallback, so the first slow inference call becomes a slow product page.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Keep a raw copy of every interaction event in S3, independent of Personalize. Managed services are easier to reason about when you can replay what you sent them.</li>



<li>Start with a domain dataset group. Move to custom recipes when you can name the specific thing the domain route will not do.</li>



<li>Measure against a popularity baseline before celebrating anything. If a personalized rail cannot beat bestsellers, the pipeline is not earning its cost yet.</li>



<li>Set training cadence from catalog velocity, not from habit. A stable catalog of a few hundred products does not need what a weekly fashion drop needs.</li>



<li>Prefer batch inference plus a cache for anything that does not have to react within the session.</li>



<li>Alarm on idle inference resources. Utilization near zero for days is a cost bug, and it is the only bug here that never pages anyone.</li>



<li>Keep the staging storefront on cheap infrastructure. The application side of this runs fine on a modest VPS from a provider like Contabo or InterServer while you are still shaping the event schema, and it keeps experiments off the production account.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How much data do I need before Personalize is worth it?</h3>



<p class="wp-block-paragraph">The documented training floor is a thousand item interactions from at least twenty-five unique users with two or more interactions each, and AWS recommends substantially more for quality recommendations. Below that, popularity ranking and hand-curated rails serve customers better and cost nothing to run.</p>



<h3 class="wp-block-heading">Can I build this without Amazon Personalize?</h3>



<p class="wp-block-paragraph">Yes, and for some shops you should. Item-to-item collaborative filtering over purchase history is a modest amount of code and runs fine as a scheduled job writing to a cache. You give up cold-start handling, exploration and session-level personalization. You gain a near-zero cost and full visibility into why an item was recommended.</p>



<h3 class="wp-block-heading">What does an idle Personalize setup cost?</h3>



<p class="wp-block-paragraph">More than zero, which is the surprising part. An active campaign bills its provisioned throughput floor whether or not anyone calls it, and a solution with automatic training on bills a training run on its cadence regardless of traffic. Rates change, so apply current pricing yourself, but the shape is a floor plus a recurring training charge.</p>



<h3 class="wp-block-heading">Real-time or batch recommendations?</h3>



<p class="wp-block-paragraph">Batch unless the module has to respond to the current session. Homepage rails, email and merchandising exports are batch problems. Product detail and cart pages are where real-time inference genuinely changes what the customer sees.</p>



<h3 class="wp-block-heading">Do new products appear automatically?</h3>



<p class="wp-block-paragraph">Only with recipes that support automatic updates, or with Trending-Now. Automatic updates refresh the deployed model every couple of hours so new items can be explored, but they do not retrain it. With a recipe like Similar-Items, a new product waits for the next training run.</p>



<h3 class="wp-block-heading">How do I connect a Shopify or WooCommerce store?</h3>



<p class="wp-block-paragraph">There is no first-party connector. Emit events from the storefront to your own endpoint, push them onto a stream, and call PutEvents from a consumer. For history, export orders and product views to S3 and load them with a dataset import job. Ordinary plumbing; the hard part is settling the item ID question first.</p>



<h3 class="wp-block-heading">How often should I retrain?</h3>



<p class="wp-block-paragraph">At least weekly is the documented baseline, even for recipes with automatic updates. Go faster only when your catalog turns over faster, and price the extra training runs before committing to a daily cadence.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">An AWS ecommerce recommendation pipeline runs two clocks, and they are easy to mistake for one. The freshness clock decides whether a customer can see the product you added this morning. The billing clock runs on training cadence and provisioned throughput, and it keeps running when nobody is looking at the site at all.</p>



<p class="wp-block-paragraph">Get the event schema right, keep a raw copy of everything in S3, stay on batch until the session genuinely demands otherwise, and set training cadence from how fast your catalog actually moves. Do that and the rest is straightforward. Skip it and you end up with something that works, costs more than the revenue it lifts, and nobody can say why.</p>



<h2 class="wp-block-heading">Need a second pair of eyes on your recommendation pipeline?</h2>



<p class="wp-block-paragraph">This is the kind of work I do for ecommerce teams building on AWS:</p>



<ul class="wp-block-list">
<li>Auditing an existing Personalize setup for orphaned campaigns, over-provisioned throughput floors and training cadences that outrun the catalog.</li>



<li>Designing the event schema and item ID strategy before you commit, including the variant-versus-product decision for Shopify and WooCommerce catalogs.</li>



<li>Building the ingestion path: storefront events through Kinesis or Firehose into Lambda, with a durable S3 copy for bulk imports and replay.</li>



<li>Choosing between domain recommenders and custom recipes for your catalog size and event mix, and modelling what each costs to run.</li>



<li>Moving eligible modules from real-time campaigns to batch inference plus a cache layer, then measuring what that takes off the bill.</li>



<li>Setting up monitoring so idle inference capacity and rising fallback rates surface as alerts rather than as a surprise invoice.</li>
</ul>



<p class="wp-block-paragraph">If something is already running, send me your dataset import job configuration and a list of your active campaigns, and I will tell you what I would change first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/aws-ecommerce-recommendation-pipeline/">Building an Ecommerce Recommendation Pipeline on AWS: The Parts That Bite</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Parquet vs JSON vs CSV: What Each Format Actually Costs You</title>
		<link>https://john-nessime.com/blog/cloud-computing/parquet-vs-json-vs-csv-cost-performance/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 24 Oct 2026 06:02:47 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[BigQuery]]></category>
		<category><![CDATA[Columnar Storage]]></category>
		<category><![CDATA[Compression]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[CSV]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Partitioning]]></category>
		<category><![CDATA[DuckDB]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[JSON]]></category>
		<category><![CDATA[Parquet]]></category>
		<category><![CDATA[Schema Design]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=339</guid>

					<description><![CDATA[<p>Format choice shows up as a bill, not a performance complaint. A working breakdown of where the money actually goes across Parquet, JSON and CSV: column pruning, row-group skipping, the gzip splittability trap, and request overhead. Includes a procedure for measuring the difference on your own data, and an honest account of where Parquet costs you.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/parquet-vs-json-vs-csv-cost-performance/">Parquet vs JSON vs CSV: What Each Format Actually Costs You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said &#8220;Athena spend tripled last month, nothing changed.&#8221; Nothing had changed on the query side. What changed was that an upstream team started dropping a wider export into the same S3 prefix, still as gzipped CSV, and every dashboard query that touched that table started reading every column of every file to answer a question about two of them.</p>



<p class="wp-block-paragraph">That is the failure mode that makes this comparison worth writing about. The <strong>Parquet vs JSON vs CSV</strong> question almost never shows up as a performance complaint first. It shows up as a bill, weeks later, from a decision somebody made in an afternoon because CSV was easy to write.</p>



<p class="wp-block-paragraph">This post breaks down where the money and the milliseconds actually go: what each format costs you on read, on write, and on the request path in between. Then a procedure for measuring it on your own data instead of trusting somebody else&#8217;s benchmark, and an honest look at where CSV and JSON still beat the columnar option.</p>



<h2 class="wp-block-heading">The billing model is the whole story</h2>



<p class="wp-block-paragraph">Most serverless query engines bill on bytes read, not bytes returned. That one sentence explains almost every surprise bill I have seen on a data lake.</p>



<p class="wp-block-paragraph">Amazon Athena charges for bytes scanned, rounded up to the nearest megabyte, with a 10 MB minimum per query. DDL statements are free. Cancelled queries are charged for whatever was scanned before the cancel landed. BigQuery&#8217;s on-demand model works the same way in spirit: charges round up to the nearest MB, with a 10 MB minimum per table referenced and a 10 MB minimum per query, and you are billed on the columns you select even when you put a LIMIT on the result.</p>



<p class="wp-block-paragraph">That last part catches people. <code>LIMIT 10</code> does not make a query cheap; the engine still reads enough to know which ten rows to give you, and on a row-based format that means reading whole rows.</p>



<p class="wp-block-paragraph">So the question is not &#8220;which format is smaller on disk.&#8221; It is &#8220;how few bytes can the engine get away with reading.&#8221; Those diverge fast.</p>



<h2 class="wp-block-heading">Four levers, not one</h2>



<p class="wp-block-paragraph">When people compare formats they usually compare compressed file sizes and stop. There are four levers that move real cost, and file size is the weakest of them.</p>



<h3 class="wp-block-heading">1. Column pruning</h3>



<p class="wp-block-paragraph">Row formats store a whole record contiguously. To get one field you read the record. Columnar formats store each column separately, so an engine reads only the column chunks your query names. Query two columns out of forty and you skip most of the file. This is the single biggest lever and it only exists in columnar formats.</p>



<h3 class="wp-block-heading">2. Block skipping</h3>



<p class="wp-block-paragraph">Parquet splits data into row groups, typically sized in the tens to hundreds of megabytes, and stores per-column statistics in the file footer: min, max, null count. An engine reads the footer first and skips any row group whose min/max range cannot possibly match your filter. Newer Parquet writers also emit page-level indexes and optional Bloom filters, which push the same idea down to a finer granularity.</p>



<p class="wp-block-paragraph">The catch: this only works if your data is sorted or clustered on the column you filter by. Write rows in random order and every row group&#8217;s min/max spans the whole range, so nothing gets skipped and you paid for the statistics for nothing. Sorting before write is what turns the feature on.</p>



<h3 class="wp-block-heading">3. Splittability</h3>



<p class="wp-block-paragraph">This is the one that bites silently. A gzip stream cannot be split at an arbitrary byte offset, so a distributed reader cannot hand different chunks of one .gz file to different workers. One gzipped CSV file means one task, on one core, no matter how large your cluster is. Spark has carried a long-standing issue about exactly this confusion, and the standard workarounds are ugly: decompress first, split the file up beforehand, or recompress with something splittable like bzip2 and eat the CPU cost.</p>



<p class="wp-block-paragraph">Parquet sidesteps this entirely. Compression is applied per page inside the file, not across the whole stream, so a Parquet file stays splittable at row-group boundaries regardless of which codec you chose. Snappy is the usual default; ZSTD is the common pick when you want a better ratio and can spend a bit more CPU.</p>



<p class="wp-block-paragraph">If your job takes an hour and one executor is doing all the work while the rest idle, check the file extension before you check anything else.</p>



<h3 class="wp-block-heading">4. Request overhead</h3>



<p class="wp-block-paragraph">Object storage bills per request as well as per byte, and listing is slow. Ten thousand tiny files cost more in GET and LIST calls than one properly sized file, and those charges never appear under your query engine&#8217;s line item. It hits Parquet users hardest, because each file carries footer metadata that must be read separately before any data is. A thousand 1 MB Parquet files can be slower and dearer than a hundred 10 MB ones holding the same rows. The format is not a substitute for file sizing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">CSV: still the right answer more often than people admit</h2>



<p class="wp-block-paragraph"><strong>Where it wins.</strong> Every tool on earth reads it. A human can open it, diff it, grep it, and eyeball it in a terminal. It appends trivially: concatenate two files and the result is still valid. For interchange with a client, a finance team, or a vendor whose ingest spec you do not control, CSV is the format that never generates a support ticket. And for datasets small enough that the 10 MB query minimum dominates anyway, the whole debate is theatre.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t.</strong> No types, no schema, no statistics. Every value is text until something guesses otherwise, and the guesses are where pipelines break: leading zeros stripped off ZIP codes, dates parsed in the wrong locale, a free-text field with an embedded comma. There is no column pruning, so a two-column query pays for a forty-column file. And gzipped CSV, the most common shape in the wild, is the splittability trap above.</p>



<p class="wp-block-paragraph">CSV is an excellent transport format and a poor storage format. That distinction is the useful one.</p>



<h2 class="wp-block-heading">JSON: the most expensive way to store a table</h2>



<p class="wp-block-paragraph"><strong>Where it wins.</strong> Nested and irregular data. If records genuinely differ, if fields come and go, if there are arrays of objects inside objects, JSON models that honestly and CSV cannot. It is self-describing per record, so a consumer that has never seen your schema can still make sense of a line. Newline-delimited JSON also appends cleanly, which is why it dominates log shipping and webhook capture. Most event pipelines are right to land raw JSON first.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t.</strong> Every record repeats every key name. Store ten million events with a field called <code>customer_identifier</code> and you have stored that string ten million times. Compression claws a lot of that back, but the engine still pays to decompress and parse it, and parsing JSON costs meaningfully more per byte than splitting on a comma. No column pruning, no statistics, no skipping.</p>



<p class="wp-block-paragraph">JSON as a landing format is fine. JSON as the format your analysts query every day is a recurring charge you are choosing to pay.</p>



<h2 class="wp-block-heading">Parquet: cheap reads, real obligations</h2>



<p class="wp-block-paragraph"><strong>Where it wins.</strong> Everything above. Column pruning, footer statistics, per-column encoding chosen to suit the data (dictionary encoding for low-cardinality strings, run-length and delta encoding for sorted integers), and splittability regardless of codec. Compressing a string column and an integer column separately beats compressing rows that mix both. On a wide table with selective queries, the reduction in bytes scanned is not marginal.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t.</strong> This is the part the enthusiastic posts skip.</p>



<ul class="wp-block-list"><li><strong>You cannot append.</strong> Adding rows means rewriting a file or adding another one. Streaming small batches straight to Parquet is how you end up with the small-files problem, which is why table formats like Apache Iceberg exist on top of it.</li><li><strong>Schema mismatches across files are a runtime error, not a warning.</strong> Write one file with a column as INT32 and another as a string, point a reader at the directory, and the job fails on type conversion. Debugging means finding which file in a prefix is the odd one out.</li><li><strong>Timestamps are a compatibility minefield.</strong> The old INT96 physical type carried nanosecond timestamps for historical reasons, is deprecated, and is still what some writers emit for compatibility with older engines. Readers expecting the INT64 logical timestamp types will refuse it. If timestamps arrive as garbage, check this first.</li><li><strong>It is opaque.</strong> You cannot grep it, tail it, or read it without tooling, and that costs real debugging time.</li><li><strong>Writing costs more.</strong> Encoding, statistics, and compression are work. If you write far more than you read, that trade may not pay off.</li></ul>



<p class="wp-block-paragraph">None of these are reasons to avoid Parquet for analytics. They are reasons to stop treating the conversion as free.</p>



<h2 class="wp-block-heading">Measure it on your data, in about twenty minutes</h2>



<p class="wp-block-paragraph">Published benchmarks tell you about somebody else&#8217;s data. Yours has different cardinality, different width, different query patterns. Here is the procedure I would run before committing to a conversion.</p>



<ol class="wp-block-list"><li><strong>Get a real byte count for what you have now.</strong> Not an estimate, the actual prefix.</li></ol>



<pre class="wp-block-code"><code>aws s3 ls s3://your-bucket/raw/events/ --recursive --summarize --human-readable</code></pre>



<p class="wp-block-paragraph">The summary at the end gives you total objects and total size. Object count matters as much as size here, because it tells you whether you have a small-files problem before you add to it.</p>



<ol start="2" class="wp-block-list"><li><strong>Convert a representative slice, not the whole thing.</strong> One partition is enough. In Athena, a CTAS statement writes the output for you.</li></ol>



<pre class="wp-block-code"><code>CREATE TABLE events_parquet_sample
WITH (
  format = 'PARQUET',
  write_compression = 'ZSTD',
  external_location = 's3://your-bucket/experiments/events-parquet/'
) AS
SELECT * FROM events_csv WHERE dt = '2020-01-01';</code></pre>



<p class="wp-block-paragraph">Pick a date that exists in your data. Athena can read Parquet compressed with LZ4 or LZO but will not write those codecs, so stick to Snappy, GZIP or ZSTD on the write path.</p>



<ol start="3" class="wp-block-list"><li><strong>Run your three most common queries against both tables and compare bytes scanned.</strong> Athena reports this per query in the console and in the query execution statistics. This number, not the wall-clock time, is your bill.</li><li><strong>On BigQuery, price it before you run it.</strong> A dry run costs nothing and returns the byte estimate.</li></ol>



<pre class="wp-block-code"><code>bq query --use_legacy_sql=false --dry_run 
  'SELECT user_id, amount FROM `project.dataset.events` WHERE dt = "2020-01-01"'</code></pre>



<ol start="5" class="wp-block-list"><li><strong>Inspect the Parquet file you just wrote.</strong> Check row group count and sizes, because a file full of tiny row groups will not perform the way the marketing says.</li></ol>



<pre class="wp-block-code"><code>python3 -c "
import pyarrow.parquet as pq
f = pq.ParquetFile('part-0000.parquet')
print(f.metadata)
print(f.metadata.row_group(0))
"</code></pre>



<p class="wp-block-paragraph">That prints the number of row groups, rows per group, and compressed and uncompressed sizes. If you see dozens of row groups in a small file, your writer is flushing per batch and you need to buffer more before writing.</p>



<ol start="6" class="wp-block-list"><li><strong>Put a guardrail up before anyone else touches it.</strong> Athena workgroups support a per-query data scan limit that cancels a query which exceeds it. You still pay for what was scanned before the cancel, but it turns an unbounded mistake into a bounded one.</li></ol>



<p class="wp-block-paragraph">Running the comparison locally works too. DuckDB reads CSV, JSON and Parquet directly from a path, and a laptop or a small VPS from a provider like Contabo or InterServer will happily handle datasets that people reflexively push into a cluster.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<ul class="wp-block-list"><li><strong>&#8220;We gzip our CSVs, so we already save the money.&#8221;</strong> Compression reduces bytes scanned, and that is real. It does not give you column pruning, and it costs you parallelism. You have optimised one lever and broken another.</li><li><strong>&#8220;Parquet is always faster.&#8221;</strong> Not for full-table scans of narrow tables, not for tiny files where footer reads dominate, and not on the write path. Selectivity and table width are what make it win.</li><li><strong>&#8220;We&#8217;ll just convert everything to Parquet.&#8221;</strong> Converting a landing zone that receives appended events all day means fighting the format&#8217;s biggest weakness. Land raw, convert on a schedule, query the converted copy.</li><li><strong>&#8220;Storage is cheap so format doesn&#8217;t matter.&#8221;</strong> Storage is cheap. Scanning is not, and scanning is billed per query, repeatedly, forever. The recurring cost is the one to model.</li><li><strong>&#8220;Our BI tool caches, so nobody re-scans.&#8221;</strong> Until the cache misses, someone changes a filter, or a scheduled refresh runs hourly against a table nobody remembers owning.</li></ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<p class="wp-block-paragraph">Not a scorecard. A short sequence of questions, in order.</p>



<ul class="wp-block-list"><li><strong>Is this data leaving your organisation?</strong> CSV, unless the recipient asked for something else. Interoperability beats efficiency at a boundary you do not control.</li><li><strong>Is it arriving continuously, one record at a time?</strong> Newline-delimited JSON or CSV into a landing prefix. Do not stream into Parquet.</li><li><strong>Is it read more than once, by more than one person, with filters?</strong> Parquet, partitioned on whatever people filter by most, sorted within partitions on the next most common filter.</li><li><strong>Is the whole dataset smaller than a few hundred megabytes?</strong> Use whatever is convenient. At that size the per-query minimums and human time dominate the byte costs.</li><li><strong>Are you fighting append semantics, concurrent writers, or schema evolution?</strong> You have outgrown bare files. That is the problem a table format like Iceberg solves, and it will still store its data as Parquet underneath.</li></ul>



<p class="wp-block-paragraph">The pattern most teams land on is not one format. It is raw JSON or CSV in a landing zone with a short lifecycle policy, Parquet in a curated zone that analysts actually query, and a scheduled job in between. Storage on the cheap tier, scanning on the cheap format.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How much smaller is Parquet than CSV?</h3>



<p class="wp-block-paragraph">It depends entirely on your data, and anyone quoting a single multiplier is guessing about yours. Wide tables with repetitive low-cardinality strings compress dramatically because dictionary encoding does most of the work. Narrow tables of high-entropy values barely improve. Convert one partition and measure it; that takes minutes and gives you a number you can defend.</p>



<h3 class="wp-block-heading">Does Parquet vs JSON vs CSV matter if I use BigQuery native tables?</h3>



<p class="wp-block-paragraph">Less than you would think, because once data is loaded into a BigQuery native table it lives in Google&#8217;s own columnar storage and on-demand billing is calculated from the logical size of the columns you select, based on their data types, rather than from whatever format you loaded from. The format question applies to your source files and to external tables over Cloud Storage. Batch loading itself is not charged.</p>



<h3 class="wp-block-heading">Is ORC better than Parquet?</h3>



<p class="wp-block-paragraph">They are close enough that ecosystem support should decide it for you. ORC carries statistics at file, stripe and row-group levels; Parquet&#8217;s page indexes and Bloom filters have narrowed that gap. ORC has deeper roots in the Hive world, Parquet has broader support across Spark, Trino, DuckDB, Polars and the Python data stack. Pick the one your engines read best and stop optimising the choice.</p>



<h3 class="wp-block-heading">Why did my query cost the same after I converted to Parquet?</h3>



<p class="wp-block-paragraph">Usually one of three things. You are running <code>SELECT *</code>, which reads every column and throws away the main advantage. Your data is unsorted, so no row group can be skipped. Or the table is not partitioned, so the engine still lists and touches everything. Parquet gives the engine the ability to skip work; your query and layout have to give it a reason to.</p>



<h3 class="wp-block-heading">Should I use Snappy, GZIP or ZSTD for Parquet?</h3>



<p class="wp-block-paragraph">Snappy is the common default and favours speed over ratio. ZSTD gives a better ratio at moderate CPU cost and is a sensible pick for data you keep a long time and read occasionally. GZIP compresses well but decompresses slowly. Because Parquet compresses at page level, none of these break splittability, so this really is just a CPU-versus-bytes trade.</p>



<h3 class="wp-block-heading">Can I query CSV and Parquet in the same table?</h3>



<p class="wp-block-paragraph">Not within one table definition, since a table declares a single input format. What you can do is keep them as separate tables over separate prefixes and union them in a view during a migration, then drop the CSV side once the backfill is verified. Mixing compression codecs within a single format is fine; Athena can read a Parquet table where some files are Snappy and others GZIP.</p>



<h3 class="wp-block-heading">What about the cost of the conversion job itself?</h3>



<p class="wp-block-paragraph">Real, and worth modelling. A conversion reads all the data once and writes it again, whether through Athena CTAS, a Glue job, or a script on a box you own. If a table is queried a handful of times a month, the conversion may never pay for itself. If it backs a dashboard that refreshes hourly, it pays for itself almost immediately. Divide the one-off cost by the per-query saving and you have your break-even in queries.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The <strong>Parquet vs JSON vs CSV</strong> decision is not about disk space. It is about how many bytes an engine is forced to read to answer a question, how many workers it can put on the job, and how many requests it makes getting there. CSV and JSON force the engine to read everything. Parquet lets it read almost nothing, provided you sorted, partitioned and sized the files so it can.</p>



<p class="wp-block-paragraph">Land raw in whatever arrives. Curate to Parquet on a schedule. Point people at the curated copy. Then go and look at your actual bytes-scanned numbers, because that is where the argument gets settled.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help cutting a data lake bill?</h2>



<p class="wp-block-paragraph">Most of the savings on a lake are unglamorous: layout, file sizes, and a handful of queries nobody has looked at since they were written. Things I can help with:</p>



<ul class="wp-block-list"><li>Auditing which queries and tables drive your bytes-scanned spend, and which conversions would actually pay for themselves</li><li>Designing partition and sort layouts around your real filter patterns instead of a default date partition nobody uses</li><li>Building the raw-to-curated conversion pipeline, whether that is Athena CTAS, a Glue job, or something running on a box you control</li><li>Fixing small-files and row-group sizing problems that quietly cap your read throughput and inflate request charges</li><li>Diagnosing Parquet schema and timestamp compatibility failures across engines that disagree about types</li><li>Setting up scan limits, cost alerting, and guardrails so the next surprise bill is caught in hours rather than at month end</li></ul>



<p class="wp-block-paragraph">If you want a concrete second opinion, send me a query execution history, a sample file, or the output of a bytes-scanned report and I will tell you where the money is going.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/parquet-vs-json-vs-csv-cost-performance/">Parquet vs JSON vs CSV: What Each Format Actually Costs You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>The Join Is the Hard Part: Combining Structured and Unstructured Data on AWS</title>
		<link>https://john-nessime.com/blog/technical-guides/combining-structured-unstructured-data-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 15 Oct 2026 06:06:48 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon S3 Vectors]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Bedrock Data Automation]]></category>
		<category><![CDATA[Bedrock Knowledge Bases]]></category>
		<category><![CDATA[Bitemporal Modeling]]></category>
		<category><![CDATA[Confidence Scoring]]></category>
		<category><![CDATA[Data Catalog]]></category>
		<category><![CDATA[Data Governance]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Lineage]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Lake Formation]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[Metadata Filtering]]></category>
		<category><![CDATA[Provenance]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[S3 Tables]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Table Compaction]]></category>
		<category><![CDATA[Vector Database]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=500</guid>

					<description><![CDATA[<p>S3 holds the documents, Iceberg holds the rows, and everything works right up until a number disagrees with its source PDF. Combining structured and unstructured data on AWS is an identity problem, not a storage one. Here are the five failure families that cause it, why each stays invisible until it isn't, and the ingestion order that avoids most of them.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/combining-structured-unstructured-data-aws/">The Join Is the Hard Part: Combining Structured and Unstructured Data on AWS</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said &#8220;the contract value in the dashboard is wrong.&#8221; It wasn&#8217;t. The number in the warehouse table was right for the version of the PDF that had been extracted eleven months earlier. Someone had uploaded an amended contract to the same S3 prefix, the extraction job had picked it up, written a new row, and nothing downstream knew which of the two rows was current. Both were true. Only one was correct.</p>



<p class="wp-block-paragraph">That is what combining structured and unstructured data on AWS actually looks like when it goes wrong. Not a service outage. Not a permissions error. A quiet correctness bug that survives every test you wrote, because every individual component did exactly what it was told.</p>



<p class="wp-block-paragraph">The storage side of this problem is solved. S3 holds the PDFs, Iceberg tables hold the rows, Bedrock Data Automation pulls fields out of the documents, S3 Vectors or OpenSearch holds the embeddings. None of that is hard any more. The hard part is the join, and the join is where nearly all the failures live.</p>



<h2 class="wp-block-heading">What this post covers</h2>



<p class="wp-block-paragraph">Five failure families I keep running into on mixed-data architectures, why each one is invisible until it isn&#8217;t, and a working shape that avoids most of them. Then troubleshooting, mistakes, and the questions people actually ask before they build this.</p>



<p class="wp-block-paragraph">Not a service tour. AWS documentation already covers what Glue, Athena, Lake Formation and Bedrock do. What it covers badly is what happens at the seams between them.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: there is no join key, only a filename</h2>



<p class="wp-block-paragraph">Structured data arrives with identity built in. A row has a primary key, a foreign key, a surrogate key from the CDC pipeline. Unstructured data arrives with an object key, which is a path, which is a decision someone made about folder layout six months ago.</p>



<p class="wp-block-paragraph">Object keys are not identity. They are location. The difference matters the moment anyone overwrites an object, moves a prefix, or uploads the same document twice under two names. Your join silently changes meaning and nothing raises an error, because to Athena a string is a string.</p>



<h3 class="wp-block-heading">What to use instead</h3>



<p class="wp-block-paragraph">Mint a document ID at ingestion, before extraction, and carry it everywhere. Store it three places: as S3 object metadata on the source object, as a column on every extraction output row, and as vector metadata on every chunk you embed. That ID is the only thing that ties the three planes together.</p>



<p class="wp-block-paragraph">Version the object too. S3 versioning gives you a version ID per object, and a document ID plus a version ID is a genuinely stable identity for &#8220;this exact bytes-on-disk artefact.&#8221; If you skip versioning on a bucket that feeds extraction, an overwrite destroys the evidence for every downstream row derived from it. You will be asked to reproduce that extraction eventually. Auditors, disputes, model retraining, all three.</p>



<p class="wp-block-paragraph">S3 Metadata is worth knowing about here. It maintains queryable Apache Iceberg tables describing the objects in a general purpose bucket, so you can join object-level facts against your business tables in Athena without writing an inventory crawler. Two limits shape how you use it: metadata tables live in an AWS managed table bucket rather than one you own, and the journal table only records change events that happen after you create the configuration. Enable it on day one or you inherit a permanent blind spot for everything already in the bucket. The inventory table covers the existing objects, which is why you want both.</p>



<h2 class="wp-block-heading">Failure family two: extraction confidence gets flattened into a fact</h2>



<p class="wp-block-paragraph">Bedrock Data Automation, Textract, and any model-based extraction return confidence alongside values. Every pipeline I have reviewed does the same thing with it: reads it once, applies a threshold, writes the value into a typed column, and drops the score.</p>



<p class="wp-block-paragraph">Now a decimal that the model was 62% sure about sits in the same column as one it was 99% sure about, and every dashboard, every SQL aggregate, every model feature treats them identically. The uncertainty didn&#8217;t go away. You just stopped being able to see it.</p>



<p class="wp-block-paragraph">Keep the score as a column. It costs one float per field and it buys three things: the ability to filter low-confidence rows out of a report without rerunning extraction, a review queue that ranks by expected damage rather than arrival order, and a way to measure whether a blueprint change actually improved anything. Add two more columns while you are there, one for the extractor or blueprint identifier and one for the timestamp. When a field goes wrong at scale the first question is always &#8220;which extractor produced these,&#8221; and without that column the answer is a week of archaeology.</p>



<pre class="wp-block-code"><code>-- Upsert extraction output into an Iceberg table, keyed on document
-- identity rather than object key. Athena performs merge-on-read for
-- MERGE INTO, so the write is cheap and the read pays the cost until
-- compaction catches up.

MERGE INTO contracts_extracted AS t
USING extraction_staging AS s
  ON t.document_id = s.document_id
 AND t.source_version_id = s.source_version_id
WHEN MATCHED THEN UPDATE SET
  contract_value       = s.contract_value,
  contract_value_conf  = s.contract_value_conf,
  extractor_id         = s.extractor_id,
  extracted_at         = s.extracted_at
WHEN NOT MATCHED THEN INSERT (
  document_id, source_version_id, contract_value,
  contract_value_conf, extractor_id, extracted_at
) VALUES (
  s.document_id, s.source_version_id, s.contract_value,
  s.contract_value_conf, s.extractor_id, s.extracted_at
);</code></pre>



<p class="wp-block-paragraph">The merge-on-read detail matters more than it looks. Athena writes delete files rather than rewriting data files, so a chatty extraction pipeline that merges every few minutes will quietly degrade query performance until compaction runs. On S3 Tables that maintenance is handled for you. On self-managed Iceberg in a general purpose bucket it is your job, and forgetting it is one of the more common reasons a lakehouse gets slower every week for no visible reason.</p>



<h2 class="wp-block-heading">Failure family three: two catalogs, two permission models</h2>



<p class="wp-block-paragraph">This is the one that bites teams who did everything else right.</p>



<p class="wp-block-paragraph">Your tables are governed by Lake Formation, with grants at table and column level. Your documents are governed by S3 bucket policies and IAM. Your vectors are governed by IAM on the vector bucket or index ARN. Three planes, three models, no shared enforcement point.</p>



<p class="wp-block-paragraph">The result is predictable and unpleasant. An analyst who is correctly denied the <code>salary</code> column in Athena runs the same question against the RAG endpoint and gets the number read straight out of the source PDF, because nobody applied column-level thinking to the chunks. The retrieval layer had no idea it was quoting a restricted field.</p>



<p class="wp-block-paragraph">There is no single AWS switch that fixes this. What works is applying the same classification at ingestion and enforcing it independently in each plane:</p>



<ul class="wp-block-list">

<li>Classify the document once, at ingestion, and write the classification into the sidecar metadata that the knowledge base ingests.</li>


<li>Use metadata filters on retrieval so a caller only ever searches the chunks their tier permits.</li>


<li>Keep restricted documents in a separate index or knowledge base where the classification is coarse and the blast radius matters.</li>


<li>Redact at extraction time for anything you genuinely cannot let leak, rather than relying on a filter you might misconfigure later.</li>

</ul>



<p class="wp-block-paragraph">Bedrock Knowledge Bases reads per-document metadata from a sidecar JSON file that sits beside the source object and carries the same filename with <code>.metadata.json</code> appended. That file is where classification, tenant, document ID and effective dates belong.</p>



<pre class="wp-block-code"><code>{
  "metadataAttributes": {
    "document_id": "doc_01HX9K4RZP",
    "classification": "restricted",
    "tenant_id": "acme",
    "effective_year": 2031
  }
}</code></pre>



<p class="wp-block-paragraph">Filterable metadata is what makes tenant isolation possible on a shared index. It is also the thing most likely to be wrong on the first attempt, which brings us to the next failure family.</p>



<h2 class="wp-block-heading">Failure family four: index decisions you cannot take back</h2>



<p class="wp-block-paragraph">S3 Vectors makes vector storage cheap enough that people stop thinking about it, and then hit a wall that has nothing to do with cost. When you create a vector index you fix three things permanently: the dimension, the distance metric, and the list of non-filterable metadata keys. Changing any of them means creating a new index and re-embedding everything.</p>



<p class="wp-block-paragraph">By default every metadata key you attach is filterable. Filterable metadata is subject to a tighter size budget than the total metadata per vector, so if you push a large chunk of text through as a filterable key, writes start failing with validation errors rather than degrading gracefully. Query a key you declared non-filterable and you get a 400 back, not an empty result set.</p>



<p class="wp-block-paragraph">If you are pointing Bedrock Knowledge Bases at an S3 Vectors index you created yourself, declare the Bedrock-managed text and metadata keys as non-filterable at creation time. They hold chunk text and are large. Leaving them filterable is the single most common reason a knowledge base sync fails partway through with validation exceptions and no obvious cause.</p>



<pre class="wp-block-code"><code># Non-filterable keys are set once, at index creation, and cannot be
# changed afterwards. Keep the list short: the cap is small, and every
# key you spend here is one you can never filter on.

aws s3vectors create-index 
  --vector-bucket-name my-vector-bucket 
  --index-name documents 
  --data-type float32 
  --dimension 1024 
  --distance-metric cosine 
  --metadata-configuration '{"nonFilterableMetadataKeys":["AMAZON_BEDROCK_TEXT","AMAZON_BEDROCK_METADATA"]}'</code></pre>



<p class="wp-block-paragraph">The dimension has to match your embedding model&#8217;s output exactly, and the same model has to be used for ingestion and for query. Swapping embedding models mid-life is not a config change, it is a migration. Plan the index around the model you intend to keep, not the one that is convenient today.</p>



<p class="wp-block-paragraph">Worth being even-handed here: S3 Vectors is the cheap, durable option and it is the right default for large corpora with modest query rates. If you need heavy filtering, hybrid keyword-plus-vector search, or very high query throughput, OpenSearch Serverless or a dedicated store like pgvector on Aurora will serve you better, and AWS supports tiering between them for exactly that reason. Pick on query pattern, not on storage price.</p>



<h2 class="wp-block-heading">Failure family five: reprocessing rewrites history</h2>



<p class="wp-block-paragraph">You improve the extraction blueprint. You rerun it across the corpus. Numbers in last quarter&#8217;s board pack change.</p>



<p class="wp-block-paragraph">This is the failure that ends trust in the platform, and it is entirely avoidable. Extraction output is a derived opinion about a document at a point in time. It is not a fact about the world. Model it that way.</p>



<p class="wp-block-paragraph">Keep two timestamps on every extraction row: when the source document became effective, and when this extraction ran. Reports pin to the first, audits use the second. Now a reprocess adds rows rather than changing answers, and &#8220;why did this number move&#8221; has an answer you can query instead of guess at.</p>



<p class="wp-block-paragraph">Iceberg gives you snapshot isolation and time travel for free on the table side, which is genuinely useful for reconciliation. It does not, on its own, tell you that the value changed because a different extractor ran. Only your columns do that.</p>



<h2 class="wp-block-heading">A shape that works for combining structured and unstructured data on AWS</h2>



<p class="wp-block-paragraph">Nothing exotic. The ordering is what matters, because several of these decisions are hard to reverse.</p>



<ol class="wp-block-list">

<li><strong>Land raw, enable versioning, enable S3 Metadata.</strong> Do this before you ingest anything you care about. Both are cheap to turn on and impossible to backfill properly.</li>


<li><strong>Mint the document ID at landing.</strong> Write it to object metadata and to the sidecar metadata file in the same step. One writer, one source of truth.</li>


<li><strong>Classify before you extract.</strong> Classification drives which extraction path runs, which index the chunks land in, and which Lake Formation grants apply downstream.</li>


<li><strong>Extract to a staging table, not straight to the curated table.</strong> Staging gives you somewhere to run validation before anything touches a table an analyst can query.</li>


<li><strong>Merge into Iceberg on document identity, carrying confidence and provenance columns.</strong> Never merge on object key.</li>


<li><strong>Embed separately, keyed by the same document ID.</strong> The vector plane and the table plane are different systems with different refresh rates. Treat them as such.</li>


<li><strong>Join at query time in Athena.</strong> Retrieval returns document IDs, SQL turns those into rows. Resist the urge to materialise one giant denormalised table containing both.</li>

</ol>



<p class="wp-block-paragraph">Step seven is the one people argue with. The argument for denormalising is query simplicity. The argument against is that you now have a third copy that drifts from both parents, and drift in a table nobody owns is how you get back to the ticket at the top of this post.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Athena says the principal has no privilege on the resource</h3>



<p class="wp-block-paragraph">This is a Lake Formation grant, not an IAM one. IAM policy alone is not enough for catalog-governed tables. Grant on the catalog and database first, then the table. For S3 Metadata tables and S3 Tables specifically, the grant needs to reach the table bucket catalog and namespace, not just the table.</p>



<h3 class="wp-block-heading">Queries against metadata tables fail with a syntax error</h3>



<p class="wp-block-paragraph">Metadata table namespaces need to be quoted with double quotes or backticks in Athena and Redshift. It looks like a parser bug and it is not.</p>



<h3 class="wp-block-heading">Knowledge base sync fails partway with validation errors</h3>



<p class="wp-block-paragraph">Almost always filterable metadata size. Check whether the Bedrock-managed text keys were declared non-filterable when the index was created. If they were not, you need a new index, because that setting is immutable.</p>



<h3 class="wp-block-heading">Retrieval returns documents the caller should not see</h3>



<p class="wp-block-paragraph">Check that the filter is actually being applied at query time and that the metadata key is filterable. A filter on a non-filterable key returns an error rather than silently failing, so if you are getting results, the filter is probably absent from the request entirely.</p>



<h3 class="wp-block-heading">Athena gets slower week over week on a table that isn&#8217;t growing much</h3>



<p class="wp-block-paragraph">Delete files accumulating from merge-on-read writes, or small file proliferation from frequent micro-batches. Check snapshot and manifest counts before you start tuning queries.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">

<li>Joining on the S3 object key because it was there and it worked in the prototype.</li>


<li>Enabling S3 Metadata after the bucket is full, then wondering why the journal table is missing most of the history.</li>


<li>Applying a confidence threshold and discarding the score.</li>


<li>Creating the vector index before deciding on the embedding model.</li>


<li>Assuming Lake Formation column grants protect the same field when it is quoted out of a PDF by a retrieval endpoint.</li>


<li>Reprocessing in place, with no record of which extractor produced which value.</li>


<li>Turning off S3 versioning on the raw bucket to save on storage, then being asked to reproduce an extraction.</li>

</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">

<li>One ingestion writer that mints identity. Everything downstream reads that ID and never invents its own.</li>


<li>Confidence and provenance columns are not optional. They are the difference between debugging and guessing.</li>


<li>Treat extraction output as append-mostly and bitemporal. Effective time and extraction time, always both.</li>


<li>Decide the embedding model before the index, and budget non-filterable keys deliberately. The cap is small and permanent.</li>


<li>Enforce classification in all three planes independently. There is no single choke point.</li>


<li>Watch table maintenance the way you watch disk. Compaction and snapshot expiry on self-managed Iceberg; a monitoring stack such as Amazon Managed Grafana or Grafana Cloud pointed at query runtime and file counts will show the trend long before an analyst complains.</li>


<li>Keep a small, cheap reprocessing rig. A modest VPS from a provider like Contabo or InterServer is enough to dry-run a blueprint change against a sample corpus before you spend Bedrock invocations on the full set.</li>

</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Do I need a lakehouse to combine structured and unstructured data on AWS?</h3>



<p class="wp-block-paragraph">No. You need a stable join key and a place to put extraction output. Athena over Iceberg tables in S3 is enough for most workloads. The lakehouse framing matters when you have several engines and warehouses to unify, not when you have documents and one analytics database.</p>



<h3 class="wp-block-heading">Should extraction output live in the same table as the structured source data?</h3>



<p class="wp-block-paragraph">Separate tables, joined at query time. They have different refresh cadences, different failure modes and different trust levels. Merging them into one table means a bad extraction run corrupts data that had nothing wrong with it.</p>



<h3 class="wp-block-heading">Is S3 Vectors a replacement for OpenSearch?</h3>



<p class="wp-block-paragraph">For cost-sensitive storage of large vector sets with moderate query volume, yes. For high query throughput, rich filtering or hybrid search, OpenSearch or pgvector remains the stronger fit. Tiering between them is a supported pattern and often the honest answer.</p>



<h3 class="wp-block-heading">How do I keep the vector index in sync with the documents?</h3>



<p class="wp-block-paragraph">Drive both from the same ingestion event. S3 event notifications into a queue, then fan out to extraction and embedding. Reconcile periodically by comparing document IDs present in the table against those present in the index, because &#8220;mostly in sync&#8221; degrades quietly.</p>



<h3 class="wp-block-heading">What breaks first when this architecture is done badly?</h3>



<p class="wp-block-paragraph">Trust. Long before anything errors, someone finds a number that disagrees with the source document and stops using the dashboard. The technical failure is downstream of the social one.</p>



<h3 class="wp-block-heading">Can I use Textract instead of Bedrock Data Automation?</h3>



<p class="wp-block-paragraph">Yes, and for pure OCR and form or table extraction with predictable layouts it is still a good fit. Bedrock Data Automation is aimed at mixed modalities and schema-shaped custom output through blueprints. The architectural advice here does not change either way, because the failure families are about identity and governance, not about which extractor you picked.</p>



<h3 class="wp-block-heading">Does Lake Formation govern my documents as well as my tables?</h3>



<p class="wp-block-paragraph">It governs the catalog and the tables registered in it. Raw objects in a general purpose bucket are governed by S3 and IAM, and vectors by IAM on the vector resources. Assume three enforcement points until you have tested otherwise.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Combining structured and unstructured data on AWS is not a storage problem and it has not been one for a while. Every service in the path works. What fails is identity: the assumption that a file path is a key, that a confidence score is a fact, and that a rerun is idempotent.</p>



<p class="wp-block-paragraph">Mint an ID at the door, carry it into every plane, keep the score and the provenance, and model extraction output as an opinion with a timestamp. Do that and the rest of the architecture is ordinary engineering. Skip it and you will spend a year explaining numbers that were never wrong, exactly.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help wiring documents and tables together?</h2>



<p class="wp-block-paragraph">This is a large part of what I do. If you are standing one of these up, or unpicking one that has drifted, I can help with:</p>



<ul class="wp-block-list">

<li>Designing document identity and provenance so extraction output survives reprocessing, overwrites and audits.</li>


<li>Building the Glue, Athena and Iceberg side of a mixed-data pipeline, including merge strategy and table maintenance that does not rot.</li>


<li>Setting up Bedrock Data Automation or Textract extraction with confidence handling and a human review path that ranks by risk.</li>


<li>Getting metadata filtering and tenant isolation right on S3 Vectors, OpenSearch or Bedrock Knowledge Bases before the index is locked in.</li>


<li>Reconciling Lake Formation grants against what a retrieval endpoint can actually surface, and closing the gaps.</li>


<li>Reviewing an existing lakehouse for the failure families above and giving you a prioritised list rather than a rewrite.</li>

</ul>



<p class="wp-block-paragraph">Send me a table schema, a sample extraction output, or the query that is returning the wrong number, and I will tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/combining-structured-unstructured-data-aws/">The Join Is the Hard Part: Combining Structured and Unstructured Data on AWS</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>RDS vs DynamoDB: The Benchmark Isn&#8217;t the Decision</title>
		<link>https://john-nessime.com/blog/technical-guides/rds-vs-dynamodb-access-patterns/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 29 Sep 2026 06:05:28 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Databases]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Aurora]]></category>
		<category><![CDATA[Amazon RDS]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Capacity Planning]]></category>
		<category><![CDATA[Connection Pooling]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Modeling]]></category>
		<category><![CDATA[DynamoDB]]></category>
		<category><![CDATA[Hot Partition]]></category>
		<category><![CDATA[Over-Engineering]]></category>
		<category><![CDATA[PostgreSQL]]></category>
		<category><![CDATA[Provisioned Throughput]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Throttling]]></category>
		<category><![CDATA[Workload Shape]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=738</guid>

					<description><![CDATA[<p>Latency charts don't decide this. The query nobody wrote down at the start does. How to build an access pattern inventory, where DynamoDB and RDS each genuinely win and hurt, a six-question decision procedure, and the arguments on both sides that fall apart under a real workload.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/rds-vs-dynamodb-access-patterns/">RDS vs DynamoDB: The Benchmark Isn&#8217;t the Decision</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone drops a benchmark link into the design review. Flat latency line for DynamoDB, a rising one for Postgres, and the data store is chosen before anybody has described what the application does.</p>



<p class="wp-block-paragraph">That argument settles architecture decisions all the time, and it&#8217;s the wrong argument. Both services return a primary-key lookup faster than your network hop and your ORM overhead. What hurts six months later is the query nobody wrote down at the start, and no benchmark will warn you about it.</p>



<p class="wp-block-paragraph">So this post skips the latency charts. It covers the access pattern inventory you write before picking anything, where each service genuinely wins and genuinely hurts, a decision procedure you can run in an afternoon, and the arguments on both sides that fall apart on contact with a real workload.</p>



<h2 class="wp-block-heading">Why the benchmark answers a question you don&#8217;t have</h2>



<p class="wp-block-paragraph">A benchmark measures a fixed operation under fixed conditions. Your application is not a fixed operation. It&#8217;s a set of reads and writes that grows every time somebody asks a new question about the data.</p>



<p class="wp-block-paragraph">The failure mode that decides RDS vs DynamoDB is almost never throughput. It&#8217;s a product manager asking, eight months in, for every account that hasn&#8217;t logged in this quarter, broken down by plan tier. On Postgres that&#8217;s an awkward query and maybe an index. On DynamoDB, if you didn&#8217;t model for it, it&#8217;s a full table scan, a new global secondary index, a backfill, and a standing write-capacity tax to keep it in sync.</p>



<p class="wp-block-paragraph">That asymmetry is the whole decision. A relational database defers the modeling cost and charges a little on every query for the flexibility. DynamoDB front-loads it: design keys around known patterns, get cheap predictable reads for exactly those, and pay dearly for anything else.</p>



<h2 class="wp-block-heading">Write the access pattern inventory first</h2>



<p class="wp-block-paragraph">Before comparing anything, write the list. Every read and write the application performs, in plain language, with the shape of the operation and how fast it needs to be. This much is enough:</p>



<pre class="wp-block-code"><code>PATTERN                                SHAPE           FREQ    LATENCY
get order by id                        point read      high    p99 &lt; 50ms
list a customer's orders, newest first range read      high    p99 &lt; 100ms
orders by status, across customers     cross-entity    low     seconds fine
revenue by product category, monthly   aggregate       low     minutes fine
find orders by partial address         text match      medium  p99 &lt; 300ms
mark order shipped + write audit row   multi-item write high   p99 &lt; 100ms</code></pre>



<p class="wp-block-paragraph">Now read the SHAPE column. If nearly every row is a point or range read, and the exceptions tolerate latency, DynamoDB is a live option. If the column is full of cross-entity reads, aggregates and text matching, you&#8217;re describing a relational workload and can stop shopping.</p>



<h3 class="wp-block-heading">The rows people leave out</h3>



<p class="wp-block-paragraph">The inventory is only useful if it&#8217;s honest, and the omissions are predictable:</p>



<ul class="wp-block-list">
<li>The admin back office. Support staff search by email, phone, partial name, anything. Ad hoc queries, and they never appear in the first inventory.</li>



<li>Reporting. Someone will want a monthly rollup. Decide now whether it lives in the operational store or streams out to a warehouse.</li>



<li>Deletion. Regulatory deletes cascade across entities. In a normalized schema that&#8217;s a transaction. In a denormalized single table it&#8217;s a fan-out you have to write and test yourself.</li>



<li>Backfills. Every schema change becomes a migration job when no schema is enforced at the storage layer.</li>
</ul>



<p class="wp-block-paragraph">Write them down even if the answer is &#8220;we&#8217;ll handle it elsewhere.&#8221; A pattern you consciously exile to a warehouse is fine. One you forgot is a rewrite.</p>



<h2 class="wp-block-heading">Where DynamoDB wins, and where it doesn&#8217;t</h2>



<p class="wp-block-paragraph">Give it its genuine case first, because it&#8217;s a strong one.</p>



<h3 class="wp-block-heading">Where it wins</h3>



<ul class="wp-block-list">
<li><strong>Connection model.</strong> It&#8217;s an HTTPS API with SigV4 auth, so there&#8217;s no pool to exhaust. That&#8217;s why it pairs cleanly with Lambda. A relational database behind high Lambda concurrency needs RDS Proxy or a pooler, which is another component you operate.</li>



<li><strong>Predictable reads at any size.</strong> A query against a well-chosen partition key costs the same at ten thousand items or ten billion. Nothing in Postgres gives you that for free.</li>



<li><strong>Idle cost.</strong> On-demand mode bills per request unit consumed, so a table with no traffic costs storage and nothing else. Useful for tenant-per-table designs and low-traffic side services.</li>



<li><strong>Operational surface.</strong> No minor version upgrades, no vacuum, no failover window, no parameter groups to argue about.</li>
</ul>



<p class="wp-block-paragraph">The query API is narrow on purpose. Give it an exact partition key plus an optional sort key condition, and it returns an ordered slice of an item collection:</p>



<pre class="wp-block-code"><code>aws dynamodb query 
  --table-name AppData 
  --key-condition-expression "PK = :pk AND begins_with(SK, :sk)" 
  --expression-attribute-values '{":pk":{"S":"CUSTOMER#42"},":sk":{"S":"ORDER#"}}'</code></pre>



<p class="wp-block-paragraph">One customer&#8217;s orders, in sort order, in one request, no join. If that call shape covers most of your inventory, you&#8217;re looking at the right service.</p>



<h3 class="wp-block-heading">Where it doesn&#8217;t</h3>



<ul class="wp-block-list">
<li><strong>Anything you didn&#8217;t model.</strong> New patterns mean new keys or a new index, and a new global secondary index means a backfill plus write capacity on every future write.</li>



<li><strong>Filtering is not querying.</strong> A <code>FilterExpression</code> runs after the read, so you pay capacity for every item examined and only save bandwidth. A filter that discards most of what it reads is a design smell.</li>



<li><strong>Aggregates.</strong> There is no <code>GROUP BY</code> and no <code>COUNT(*)</code> over a table. You maintain aggregates yourself, usually with DynamoDB Streams feeding a Lambda that updates a counter item, which means you own the correctness problem when that consumer fails.</li>



<li><strong>Hard physical limits.</strong> An item caps at 400 KB. A Query or Scan page returns at most 1 MB before you paginate. A single partition sustains roughly 3,000 read and 1,000 write units per second, so a hot key throttles while the table sits well under its quota.</li>



<li><strong>Index budget.</strong> Global secondary indexes default to twenty per table; local secondary indexes cap at five and can only be defined at table creation. You can&#8217;t index your way out of a bad key design forever.</li>
</ul>



<h2 class="wp-block-heading">Where RDS wins, and where it doesn&#8217;t</h2>



<h3 class="wp-block-heading">Where it wins</h3>



<ul class="wp-block-list">
<li><strong>Unknown future queries.</strong> The entire point. Joins, aggregates, window functions, ad hoc analysis, and a new index in one statement when something turns out slow.</li>



<li><strong>Constraints the database enforces.</strong> Foreign keys, uniqueness, check constraints, transactions across arbitrary tables. In DynamoDB each of those is application code you write and maintain.</li>



<li><strong>Extensions.</strong> PostGIS for geospatial, pgvector for embeddings, full-text search built in, all against the same rows in the same transaction.</li>



<li><strong>Portability.</strong> Postgres and MySQL run on RDS, in a container, or on a plain VPS from a provider like Contabo or InterServer, so migrating off is an export and an import. DynamoDB&#8217;s data model doesn&#8217;t port without rewriting your data access layer.</li>



<li><strong>Everyone already knows SQL.</strong> Underrated on a small team. Teaching four developers single-table design costs real review time.</li>
</ul>



<h3 class="wp-block-heading">Where it doesn&#8217;t</h3>



<ul class="wp-block-list">
<li><strong>Connections under serverless concurrency.</strong> Each Lambda execution environment opens its own connection, and the database refuses new ones once <code>max_connections</code> is reached. RDS Proxy multiplexes to fix that, but some operations pin a client to a backend connection and pinned connections aren&#8217;t reused. Watch <code>DatabaseConnectionsCurrentlySessionPinned</code> before trusting the pool.</li>



<li><strong>Instances are always on.</strong> A provisioned instance bills whether or not anyone queries it. Aurora Serverless v2 changes that with a minimum of zero capacity units on a supported engine version, but resuming from paused is slow enough that the first user after an idle period feels it. AWS documents resume as taking roughly fifteen seconds.</li>



<li><strong>Vertical scaling has a ceiling.</strong> Writes go to one primary and read replicas help reads only. Sharding is a project, not a setting.</li>



<li><strong>Maintenance is yours.</strong> Version upgrades, failover testing, long transactions blocking vacuum, connection storms on restart. Managed does not mean absent.</li>
</ul>



<p class="wp-block-paragraph">If you want a database that hibernates properly, it lives in the scaling configuration and the idle timeout is yours to set:</p>



<pre class="wp-block-code"><code>aws rds create-db-cluster 
  --db-cluster-identifier app-cluster 
  --engine aurora-postgresql 
  --serverless-v2-scaling-configuration 
      MinCapacity=0,MaxCapacity=4,SecondsUntilAutoPause=600 
  --manage-master-user-password</code></pre>



<p class="wp-block-paragraph">A minimum of zero only works on engine versions supporting automatic pause and resume, so check yours before planning a budget around it. If the cluster refuses the setting, that&#8217;s usually why.</p>



<h2 class="wp-block-heading">A decision procedure for RDS vs DynamoDB</h2>



<p class="wp-block-paragraph">Run these in order against the inventory you wrote. The first clear answer wins.</p>



<ol class="wp-block-list">
<li><strong>Can you enumerate the access patterns, and are they stable?</strong> If the product is still moving and nobody can say what the admin tools will need, pick the store that tolerates being wrong. That&#8217;s RDS.</li>



<li><strong>Does any pattern need a cross-entity aggregate on the hot path?</strong> A user-facing screen needing a count or sum across entities in under a second, with no way to precompute it, means RDS.</li>



<li><strong>Is there a natural partition key with even distribution?</strong> Tenant ID, user ID, device ID. If traffic concentrates on one key, the per-partition ceiling finds you regardless of capacity mode. No good key means no DynamoDB.</li>



<li><strong>What&#8217;s the concurrency shape of the compute tier?</strong> Thousands of short-lived Lambda invocations argue for DynamoDB. A few long-running containers with a warm pool remove the strongest argument against RDS.</li>



<li><strong>Does the workload go quiet?</strong> Bursty and mostly idle favors on-demand DynamoDB or a scale-to-zero Aurora cluster. Steady traffic makes provisioned capacity or a right-sized instance cheaper.</li>



<li><strong>Who maintains this in a year?</strong> If it&#8217;s one part-time developer, weight familiarity heavily. A design nobody on the team can debug is a liability whatever the benchmark said.</li>
</ol>



<h2 class="wp-block-heading">The cost model nobody actually models</h2>



<p class="wp-block-paragraph">The two bill on axes that don&#8217;t compare. RDS charges instance hours, storage, and depending on engine and storage type, I/O, busy or not. DynamoDB charges request units and storage, so an idle table is nearly free and a chatty one isn&#8217;t.</p>



<p class="wp-block-paragraph">The number that matters is request units per user action, and almost nobody calculates it. Reads bill in 4 KB increments and writes in 1 KB increments, strongly consistent reads cost twice what eventually consistent ones do, and transactional operations double again. Every index you project into consumes write units on every write that touches it. A screen firing six queries plus an audit write can cost many times the naive estimate.</p>



<p class="wp-block-paragraph">Work it out per action, multiply by real traffic, and sanity check the instance side with my <a href="https://john-nessime.com/tools/aws-cost-estimator/">AWS cost estimator</a> before the decision hardens. AWS recommends on-demand as the default capacity mode for most workloads, but steady predictable traffic is still cheaper provisioned, and you can flip between them:</p>



<pre class="wp-block-code"><code># which mode is this table on?
aws dynamodb describe-table --table-name AppData 
  --query "Table.BillingModeSummary.BillingMode"

# what is it actually consuming?
aws cloudwatch get-metric-statistics 
  --namespace AWS/DynamoDB 
  --metric-name ConsumedWriteCapacityUnits 
  --dimensions Name=TableName,Value=AppData 
  --statistics Sum --period 3600 
  --start-time $(date -u -d '1 day ago' +%Y-%m-%dT%H:%M:%SZ) 
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ)</code></pre>



<p class="wp-block-paragraph">Switching billing mode is rate limited, so treat it as a planned change rather than an incident-time knob.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<ul class="wp-block-list">
<li><strong>&#8220;DynamoDB scales infinitely.&#8221;</strong> The service does. Your partition key might not. Uneven access concentrates load on one partition, and adaptive capacity only helps within limits.</li>



<li><strong>&#8220;Postgres can&#8217;t handle our scale.&#8221;</strong> Usually said before anyone has run <code>EXPLAIN (ANALYZE, BUFFERS)</code> on the slow query. A missing index and an N+1 in the ORM explain most claims of outgrowing relational.</li>



<li><strong>&#8220;We&#8217;ll add the indexes later.&#8221;</strong> True in Postgres, expensive in DynamoDB. A new global secondary index backfills across the whole table and then taxes every subsequent write.</li>



<li><strong>&#8220;Serverless database means no operations.&#8221;</strong> You trade instance operations for capacity and key-design operations. Throttling, hot partitions and stream consumer lag all need dashboards, exactly like a relational instance does.</li>



<li><strong>&#8220;We can just use both.&#8221;</strong> Sometimes right, often a way to avoid deciding. Two stores means two consistency models, two restore procedures, and a sync path that becomes the least reliable component you own.</li>
</ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<p class="wp-block-paragraph">My default for a new application with an unfinished product definition is managed Postgres. Not because it&#8217;s better, but because being wrong about a relational schema costs an afternoon and being wrong about a key design costs a migration.</p>



<p class="wp-block-paragraph">I&#8217;d reach for DynamoDB first when the workload is a small closed set of patterns keyed off one identifier, with high write volume, tolerance for eventual consistency, and concurrency that would flatten a connection pool. Session stores, telemetry ingestion, per-user feeds, idempotency keys, feature flags. It&#8217;s excellent at all of those.</p>



<p class="wp-block-paragraph">The hybrid worth doing is narrow: relational as the system of record, DynamoDB for one high-volume pattern that joins nothing. The hybrid to avoid splits related entities across both stores and reconciles them in application code.</p>



<p class="wp-block-paragraph">Whichever you land on, instrument it on day one. Throttled requests and consumed capacity on the DynamoDB side, connection counts and slow queries on the relational side, on a CloudWatch or Grafana Cloud dashboard someone actually opens. If you&#8217;re restricting item-level access with IAM condition keys such as <code>dynamodb:LeadingKeys</code>, run the policy through my <a href="https://john-nessime.com/tools/aws-iam-policy-checker/">IAM policy checker</a> before it ships.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is DynamoDB always faster than RDS?</h3>



<p class="wp-block-paragraph">For a primary-key lookup at large scale its latency is more consistent, because that operation&#8217;s cost doesn&#8217;t grow with table size. For a query it wasn&#8217;t modeled for it can be far slower, because the fallback is a scan. Speed depends on the match between query and key design, not on the service.</p>



<h3 class="wp-block-heading">Can I run analytics on DynamoDB?</h3>



<p class="wp-block-paragraph">Not directly, and you shouldn&#8217;t try. Export the table to S3 or stream changes out, then query with Athena or load a warehouse. Treat operational and analytical stores as separate systems.</p>



<h3 class="wp-block-heading">What about Aurora Serverless as the middle ground?</h3>



<p class="wp-block-paragraph">It solves idle cost, not connections or write scaling. If you&#8217;re considering DynamoDB because of Lambda concurrency or single-primary write limits, Aurora Serverless doesn&#8217;t address that. If it&#8217;s because you&#8217;re paying for an idle dev database, it does.</p>



<h3 class="wp-block-heading">How do I migrate from DynamoDB to RDS if I get it wrong?</h3>



<p class="wp-block-paragraph">Export to S3, reshape denormalized items into normalized tables, load, and dual-write during cutover. The data movement is routine. The hard part is that your application code assumed a key-value access shape everywhere, so the rewrite lands in the service layer, not the pipeline.</p>



<h3 class="wp-block-heading">Is DynamoDB cheaper than RDS for a small application?</h3>



<p class="wp-block-paragraph">Usually yes when traffic is low and bursty, because on-demand charges per request against an idle table while an instance bills continuously. The advantage narrows as traffic steadies and reverses once throughput justifies provisioned capacity or a reserved instance.</p>



<h3 class="wp-block-heading">Do I need RDS Proxy if I use Lambda with Postgres?</h3>



<p class="wp-block-paragraph">At meaningful concurrency, yes. Below that, a direct connection reused across invocations is fine, and the proxy adds cost plus a pinning failure mode. Add it when connection counts approach your limit, not preemptively in development.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">The one thing worth remembering about RDS vs DynamoDB: you&#8217;re not choosing a performance profile, you&#8217;re choosing when to pay the modeling cost. DynamoDB takes it up front and pays you back in predictability. A relational database lets you defer it and charges interest later in query complexity.</p>



<p class="wp-block-paragraph">Write the access pattern inventory. Be honest about the admin screens and the reporting nobody has asked for yet. If the list is short, stable and keyed on one identifier, DynamoDB will serve you for years. If it isn&#8217;t, take the relational database and spend the saved effort on indexes and connection handling.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second opinion on the data store decision?</h2>



<p class="wp-block-paragraph">Cheap to review now, expensive to revisit later. Things I help with:</p>



<ul class="wp-block-list">
<li>Building the access pattern inventory with your team and pressure-testing it against the queries your roadmap implies</li>



<li>Reviewing a DynamoDB key and index design before it ships, including partition key distribution and index write amplification</li>



<li>Diagnosing throttling and hot partitions on an existing table, and whether the fix is a key change, a capacity change or a cache</li>



<li>Sizing RDS or Aurora for a serverless compute tier, including connection handling, RDS Proxy configuration and pinning</li>



<li>Modeling the cost of both options against your real request volume instead of a rule of thumb</li>



<li>Planning a migration in either direction, with a dual-write cutover and a rollback that works</li>
</ul>



<p class="wp-block-paragraph">Send me your access pattern list, a slow query plan, or your throttling metrics and I&#8217;ll tell you what I&#8217;d do with it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/rds-vs-dynamodb-access-patterns/">RDS vs DynamoDB: The Benchmark Isn&#8217;t the Decision</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Your CRM Is Not a Database: Where GoHighLevel Should Stop</title>
		<link>https://john-nessime.com/blog/data-engineering/gohighlevel-as-a-database/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 26 Sep 2026 06:00:23 +0000</pubDate>
				<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Enterprise Integration]]></category>
		<category><![CDATA[SaaS Engineering]]></category>
		<category><![CDATA[Architecture Debt]]></category>
		<category><![CDATA[CRM Audit]]></category>
		<category><![CDATA[Custom Objects]]></category>
		<category><![CDATA[Data Modeling]]></category>
		<category><![CDATA[GoHighLevel]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Marketing Automation]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Reverse ETL]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Snapshots]]></category>
		<category><![CDATA[Sub-Account Management]]></category>
		<category><![CDATA[System of Record]]></category>
		<category><![CDATA[Vendor Lock-In]]></category>
		<category><![CDATA[Webhooks]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=649</guid>

					<description><![CDATA[<p>A CRM never pushes back when you add one more field, which is exactly why the schema drifts until someone asks a question that needs a join. Four boundaries for deciding what belongs in GoHighLevel, what Custom Objects actually change, and the CRM-in-front, system-of-record-behind pattern I default to.</p>
<p>The post <a href="https://john-nessime.com/blog/data-engineering/gohighlevel-as-a-database/">Your CRM Is Not a Database: Where GoHighLevel Should Stop</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">&#8220;Can you pull every job we did last quarter, by technician, with the margin on each one?&#8221;</p>



<p class="wp-block-paragraph">The answer was no. Not &#8220;no, that&#8217;s expensive.&#8221; Just no. The jobs weren&#8217;t records. Each one was a couple of tags on a contact, three custom fields added at different times by different people, an opportunity someone had dragged into Won, and a note cut off mid-sentence. There was no job. There was a person that a job had happened to.</p>



<p class="wp-block-paragraph">Nobody made a bad decision to get there. &#8220;Can we just add a field for that?&#8221; is a five-second request and the answer is always yes, because adding a field is easy. That&#8217;s the trap. The CRM never pushes back, so the schema drifts for eighteen months and nobody notices until someone asks a question that needs a join.</p>



<p class="wp-block-paragraph">This post is about where that line sits: not whether to use the platform, but where <strong>GoHighLevel as a database</strong> stops being a shortcut and starts being architecture debt. I&#8217;ll cover the failure that stays invisible, the four boundaries that decide it, what Custom Objects actually change, and the arrangement I default to when the CRM isn&#8217;t the right home.</p>



<h2 class="wp-block-heading">The failure mode nobody sees coming</h2>



<p class="wp-block-paragraph">Databases push back. Declare a column as a date and it refuses to hold &#8220;next Tuesday.&#8221; Declare a foreign key and it refuses to orphan the row. Half the value of a schema isn&#8217;t storage, it&#8217;s the set of things it won&#8217;t let you do.</p>



<p class="wp-block-paragraph">A custom field holds whatever you type. That&#8217;s a feature when a rep jots down a competitor&#8217;s name. It&#8217;s a slow disaster when the field is called <code>contract_value</code> and it contains &#8220;1,250.00&#8221;, &#8220;$1250&#8221;, &#8220;1250 (pending)&#8221; and &#8220;ask Dave&#8221; across four hundred records.</p>



<p class="wp-block-paragraph">Here&#8217;s why it stays invisible: nothing breaks. Workflows keep firing. Dashboards keep rendering, because they count records rather than summing values. The failure surfaces the day someone needs an aggregate, and by then you have four hundred rows of ambiguity and no way to tell which reading was right. You can&#8217;t retroactively decide whether &#8220;1250 (pending)&#8221; was invoiced.</p>



<p class="wp-block-paragraph">History is the other half. A CRM field is a current value. Overwrite it and the old one is gone. Anything touching money, entitlements or consent eventually needs to prove what the state was and when, and the platform discarded that silently, months earlier.</p>



<h2 class="wp-block-heading">What the platform is genuinely good at</h2>



<p class="wp-block-paragraph">An argument against a tool isn&#8217;t credible unless it starts with the tool&#8217;s real case. GoHighLevel is very good at a specific set of things, most of them expensive to build yourself.</p>



<ul class="wp-block-list">
<li><strong>Conversation state across channels.</strong> One thread per person spanning SMS, email and calls, with delivery status attached. Rebuilding that is months of work and a compliance surface you don&#8217;t want.</li>

<li><strong>Sequenced outreach with human interrupts.</strong> Wait steps, conditions, and a reply pulling someone out of a sequence. Easy to describe, tedious to get right.</li>

<li><strong>Tenant isolation by default.</strong> Each sub-account is its own location, and nothing crosses unless you deliberately copy it.</li>

<li><strong>Non-engineers can change it.</strong> A workflow a client can edit on a Tuesday afternoon beats an elegant pipeline only you can touch.</li>
</ul>



<p class="wp-block-paragraph">Every one of those is about people and communication. That&#8217;s the pattern: when the thing you&#8217;re storing is a person, or something that happened to a person, the CRM is the right home. When it has a life independent of any person, you&#8217;re outside the design.</p>



<h2 class="wp-block-heading">Four boundaries that decide it for you</h2>



<p class="wp-block-paragraph">&#8220;Is this a database?&#8221; isn&#8217;t a useful question. These four are decidable, and any one of them crossing is usually enough.</p>



<h3 class="wp-block-heading">1. The question you need to ask requires a join</h3>



<p class="wp-block-paragraph">Not &#8220;show me contacts tagged X.&#8221; I mean revenue by technician by quarter, or every account with more than three open items where the last payment was over sixty days ago: grouping, filtering across related records, and arithmetic on typed values, all at once. Smart lists answer a narrow, predefined set of questions well, but they aren&#8217;t a query engine, and the gap between the two is where most people discover they&#8217;ve built a database by accident.</p>



<h3 class="wp-block-heading">2. More than one system writes the same field</h3>



<p class="wp-block-paragraph">This is the one that quietly costs the most. A stock level or invoice status lives in a CRM field, and both an automation and an external sync can write it. There&#8217;s no transaction and no compare-and-set. Last write wins, and &#8220;last&#8221; is whichever request happened to land second. You see it as an occasional wrong value you can&#8217;t reproduce, which is the worst kind of bug to be handed.</p>



<p class="wp-block-paragraph">The fix isn&#8217;t better retry logic. The field has the wrong owner. Exactly one system should write any given piece of state, and if that system isn&#8217;t the CRM, the CRM holds a copy for display rather than the value itself.</p>



<h3 class="wp-block-heading">3. Someone will need the history</h3>



<p class="wp-block-paragraph">Anything where you may later have to prove a prior state needs append-only storage. The workaround people reach for is a note per change. It doesn&#8217;t hold: notes aren&#8217;t structured, aren&#8217;t queryable, and the export path treats them as an afterthought, which leads straight to the next boundary.</p>



<h3 class="wp-block-heading">4. The exit shape doesn&#8217;t match what you put in</h3>



<p class="wp-block-paragraph">People check this last and should check it first. Contact export lands as CSV, and only the most recent note travels with it, truncated to 255 characters. A year of notes exports as one, clipped. Custom object data isn&#8217;t part of a direct export at all: the documented route is a snapshot from one sub-account to another, which is a copy inside the platform, not a file you keep. Export permissions are role-gated, so a client who believes they own their data may not be able to pull it themselves.</p>



<p class="wp-block-paragraph">None of that makes it a bad choice. It makes it a bad <em>sole</em> home for anything you&#8217;d be in trouble without.</p>



<h2 class="wp-block-heading">What Custom Objects change, and what they don&#8217;t</h2>



<p class="wp-block-paragraph">Custom Objects are a real improvement and worth using. Properties, policies, vehicles, courses: things that aren&#8217;t people, modelled as first-class records with their own fields and associations back to contacts, usable in workflows. For years the only option was squashing everything into a contact, and that was worse. They don&#8217;t turn the CRM into a database, though, and the reasons are specific rather than philosophical.</p>



<ul class="wp-block-list">
<li><strong>Ten objects per sub-account, documented as a hard cap.</strong> That&#8217;s your entire entity budget. Fifteen nouns worth modelling means five go back into custom fields.</li>

<li><strong>Uniqueness is limited and one-way.</strong> Up to ten unique fields per object, and only single line text, multi line text, number and phone types can be unique. It&#8217;s enforced across the sub-account and every entry point including the API, which is genuinely useful. But downgrade a unique field to non-unique and you cannot make it unique again.</li>

<li><strong>Write access is admin-only.</strong> Creating, updating and deleting object records is restricted to location admins; regular users are read-only. Check this before assuming the front desk will maintain records.</li>

<li><strong>Records count against storage quota.</strong> Object records and attached files consume the same account storage as everything else, so high-volume machine-generated data is the wrong shape.</li>

<li><strong>The exit is a snapshot, not an export.</strong> Objects, fields and associations move between sub-accounts via snapshots, subject to the same cap in the target account.</li>
</ul>



<p class="wp-block-paragraph">Read those as design constraints rather than complaints. They tell you the intended scale: dozens of entity types, no. Millions of rows, no. A handful of business nouns with clean identity and moderate volume, yes, and it works well there.</p>



<h2 class="wp-block-heading">The API budget is a design constraint too</h2>



<p class="wp-block-paragraph">If you&#8217;re treating the platform as a datastore behind an integration, the rate limits shape what&#8217;s possible. On the v2 API a marketplace app gets a burst allowance of 100 requests per 10 seconds per resource, and 200,000 requests per day per resource, where a resource means a location or company. Version 1 is end of support, so anything new belongs on v2.</p>



<p class="wp-block-paragraph">Every v2 request carries a version header alongside the bearer token. It&#8217;s easy to forget and produces a confusing failure when missing:</p>



<pre class="wp-block-code"><code>curl -H "Authorization: Bearer $TOKEN" 
     -H "Version: 2021-04-15" 
     -H "Accept: application/json" 
     "$BASE_URL/&lt;resource&gt;?locationId=$LOCATION_ID"</code></pre>



<p class="wp-block-paragraph">More useful than the numbers is that responses carry their own accounting, so you never have to guess where you are in the window: <code>X-RateLimit-Remaining</code> and <code>X-RateLimit-Max</code> cover the burst interval, <code>X-RateLimit-Daily-Remaining</code> and <code>X-RateLimit-Limit-Daily</code> cover the day.</p>



<p class="wp-block-paragraph">Pace against the remaining count rather than reacting to failures, and watch the daily counter separately: burst limits recover in seconds, a daily allowance doesn&#8217;t, and a backfill that exhausts it takes the rest of the day&#8217;s normal traffic with it. The consequence is that polling is expensive and webhooks are cheap. Push where you can; poll only to reconcile.</p>



<h2 class="wp-block-heading">The shape I reach for: CRM in front, system of record behind</h2>



<p class="wp-block-paragraph">When something crosses one of those boundaries, this is the arrangement I default to. It&#8217;s deliberately boring.</p>



<ol class="wp-block-list">
<li><strong>Pick one owner per piece of state and write it down.</strong> Contact details and conversation history belong to the CRM. Money, inventory and anything with an audit requirement belong to the external store. No field gets two owners.</li>

<li><strong>Give every record a stable external ID on both sides.</strong> Do this on day one. Retrofitting identity onto records matched by email is genuinely painful.</li>

<li><strong>Push changes out with webhooks, not polling.</strong> Webhook to a queue, queue to a worker, and make the worker idempotent, because you will receive duplicates and you&#8217;ll need to replay after an outage.</li>

<li><strong>Reconcile on a schedule anyway.</strong> Webhooks get dropped. A nightly sweep comparing counts and updated timestamps, re-pulling only the drift, catches what the push path missed without burning the daily allowance.</li>

<li><strong>Push a flattened summary back for display.</strong> The CRM doesn&#8217;t need the ledger, just the two or three values a human wants on the contact record. Treat those fields as read-only and never let a workflow write them.</li>
</ol>



<p class="wp-block-paragraph">The external side doesn&#8217;t need to be impressive. For a small engagement it&#8217;s usually managed Postgres, or plain Postgres on a modest VPS from somewhere like Contabo or InterServer, sat behind Cloudflare so nothing is directly exposed. The glue is often n8n or Make rather than anything custom. What matters is that constraints exist:</p>



<pre class="wp-block-code"><code>CREATE TABLE job (
  id             bigserial PRIMARY KEY,
  crm_contact_id text        NOT NULL,
  technician_id  bigint      NOT NULL REFERENCES technician(id),
  completed_on   date        NOT NULL,
  value_cents    bigint      NOT NULL CHECK (value_cents &gt;= 0),
  created_at     timestamptz NOT NULL DEFAULT now()
);</code></pre>



<p class="wp-block-paragraph">Nothing clever there. But <code>value_cents</code> can&#8217;t hold &#8220;ask Dave&#8221;, <code>technician_id</code> can&#8217;t point at a technician who doesn&#8217;t exist, and the quarterly margin question becomes one query instead of a conversation about why it can&#8217;t be answered.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<ul class="wp-block-list">
<li><strong>&#8220;We&#8217;re too small for a second system.&#8221;</strong> Sometimes right, and worth taking seriously. But the cost of adding a store later isn&#8217;t the store, it&#8217;s the reconstruction, and some of that history is unrecoverable.</li>

<li><strong>&#8220;We&#8217;ll clean the data up later.&#8221;</strong> Cleanup needs to know what the value should have been. Free-text fields with no history don&#8217;t retain that. This is the one I&#8217;d push back on hardest.</li>

<li><strong>&#8220;Custom Objects solved this.&#8221;</strong> They solved modelling non-person entities, which was the biggest gap. They didn&#8217;t add joins, transactions, versioning or an export path.</li>

<li><strong>&#8220;The client wants everything in one place.&#8221;</strong> They want one place to <em>look</em>, which is a UI requirement, not a storage requirement. A summary pushed back to the contact record satisfies it.</li>

<li><strong>&#8220;We can export whenever we want.&#8221;</strong> Test it before you rely on it. Export the notes. Export a custom object. Do it as the client&#8217;s own user, not as an admin.</li>
</ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<ol class="wp-block-list">
<li><strong>Name the thing.</strong> If it&#8217;s a noun that isn&#8217;t a person, it wants to be a record, not a field on a contact.</li>

<li><strong>Write out the hardest question anyone will ask of it.</strong> The actual sentence. If it needs grouping, arithmetic, or two entity types at once, the CRM isn&#8217;t the home.</li>

<li><strong>Count the writers.</strong> More than one system writing the same field means the CRM holds a copy, not the value.</li>

<li><strong>Decide whether history matters.</strong> Assume it does for anything involving money or consent.</li>

<li><strong>Do the export before you commit.</strong> Not the docs, the actual file. Whatever doesn&#8217;t come out cleanly is data you don&#8217;t really own.</li>

<li><strong>Check the entity budget.</strong> Ten objects per sub-account. If you&#8217;re modelling toward that ceiling, decide now which nouns get a slot.</li>
</ol>



<p class="wp-block-paragraph">Steps two and five take about twenty minutes together and prevent most of what I&#8217;ve described.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I use GoHighLevel as a database for my business data?</h3>



<p class="wp-block-paragraph">For people, conversations, bookings and pipeline state, yes, and it&#8217;s a good one. For anything needing joins, transactions, versioned history or guaranteed export, no. Using <strong>GoHighLevel as a database</strong> for that second category works right up until someone asks a question the platform can&#8217;t express, and by then the data is already ambiguous.</p>



<h3 class="wp-block-heading">How many Custom Objects can a sub-account have?</h3>



<p class="wp-block-paragraph">Ten per location, documented as a hard cap. Each object supports up to ten unique fields, and only single line text, multi line text, number and phone types can be marked unique. Treat those slots as a fixed budget when you plan the model.</p>



<h3 class="wp-block-heading">Can I export Custom Object data out of the platform?</h3>



<p class="wp-block-paragraph">Not through a direct export the way contacts go to CSV. The documented path is a snapshot between sub-accounts, which carries fields and associations but stays inside the platform. The API does support CRUD on objects and records, so a scripted pull is the practical route to a copy you keep.</p>



<h3 class="wp-block-heading">Why do my exported notes look truncated?</h3>



<p class="wp-block-paragraph">Contact export includes only the most recent note, cut to 255 characters. That&#8217;s expected behaviour, not a bug. If notes hold anything you&#8217;d need later, pull them through the API and store them somewhere that keeps all of them.</p>



<h3 class="wp-block-heading">What usually trips the API rate limits?</h3>



<p class="wp-block-paragraph">Backfills, and polling loops that scale with contact count. The limits are 100 requests per 10 seconds and 200,000 per day, both per resource. Read the rate limit headers on each response and pace against them rather than retrying into a wall.</p>



<h3 class="wp-block-heading">Does a sub-account count as a real tenancy boundary?</h3>



<p class="wp-block-paragraph">For operational separation, yes: each is its own location and nothing crosses unless you copy it. The consequence is that cross-client reporting has to happen outside the platform, because no query spans locations. If portfolio-level reporting is a requirement, plan for an external store from the start.</p>



<h3 class="wp-block-heading">Should I migrate off the CRM entirely?</h3>



<p class="wp-block-paragraph">Usually not. A full migration means rebuilding the conversation and outreach layer, which is the part it does best. Moving one category of data out, with a clear owner and a summary pushed back for display, is smaller and reversible.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The CRM will never tell you that you&#8217;ve gone too far. There&#8217;s no constraint to violate, no error to raise, no migration to fail. That&#8217;s exactly what makes the boundary worth drawing on purpose, before the field gets added rather than after four hundred records have been through it.</p>



<p class="wp-block-paragraph">The test I&#8217;d keep: write down the hardest question anyone will ever ask of this data, then check whether the platform can express it. If it can&#8217;t, that&#8217;s not a reporting gap a better dashboard will close. It&#8217;s the boundary telling you where <strong>GoHighLevel as a database</strong> should stop and a real system of record should start. The identity mapping, the webhooks, the reconciliation sweep: all of that is plumbing once the decision is made.</p>



<h2 class="wp-block-heading">Working on this with me</h2>



<p class="wp-block-paragraph">Most of what I do here is unglamorous and ends with the reporting question getting answered:</p>



<ul class="wp-block-list">
<li>Auditing an existing setup and mapping which custom fields have quietly become load-bearing, with an owner assigned to each.</li>

<li>Drawing the boundary: what stays in the CRM, what moves to Custom Objects, what needs an external system of record.</li>

<li>Building the sync layer: webhooks into a queue, idempotent workers, stable ID mapping both ways, and a reconciliation job that catches dropped events without burning the API allowance.</li>

<li>Standing up the store: schema design with constraints that hold, on managed Postgres or a small VPS, with backups you&#8217;ve tested restoring rather than assumed.</li>

<li>Rescuing free-text fields into typed columns, including the judgement calls about what&#8217;s recoverable.</li>

<li>Handover packaging: a real export, documented, so a client can leave with their data whether or not they keep working with you.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;re weighing this up, send me something concrete: a list of your custom field names, a sample export, or the query you can&#8217;t currently answer. That&#8217;s usually enough to tell you where the line sits.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/data-engineering/gohighlevel-as-a-database/">Your CRM Is Not a Database: Where GoHighLevel Should Stop</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Parquet vs JSON vs CSV: Where the Money Actually Goes</title>
		<link>https://john-nessime.com/blog/technical-guides/parquet-vs-json-vs-csv-cost-performance-2/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 31 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[BigQuery]]></category>
		<category><![CDATA[Columnar Storage]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Partitioning]]></category>
		<category><![CDATA[DuckDB]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Parquet]]></category>
		<category><![CDATA[Predicate Pushdown]]></category>
		<category><![CDATA[Schema Design]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=342</guid>

					<description><![CDATA[<p>A practical comparison of Parquet, JSON and CSV on cost and query performance, covering how each billing model changes the answer, the mechanics behind columnar savings, the failure modes that cancel them out, and a six-step way to model the migration before you commit to it.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/parquet-vs-json-vs-csv-cost-performance-2/">Parquet vs JSON vs CSV: Where the Money Actually Goes</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A team converts its event logs from JSON to Parquet over a weekend. The Glue job runs clean, the files land in S3, the dashboards keep working. Monday morning someone opens the Athena bill expecting it to have fallen off a cliff, and it has moved by about a tenth.</p>



<p class="wp-block-paragraph">That is the failure mode worth understanding, and it stays invisible until you go looking for it. The conversion was real. The files did get much smaller. But the dashboard queries were still selecting every column from an unpartitioned table, so the engine still had to open every column chunk in every file. Columnar storage only pays out when the query gives the reader something to skip.</p>



<p class="wp-block-paragraph">This is a working comparison of <strong>Parquet vs JSON vs CSV</strong> on the two things that end up on an invoice: bytes read, and time spent reading them. Each format gets a profile with the cases where it wins and the cases where it does not, followed by the places where the promised savings quietly evaporate and a procedure for modelling the change on your own data before you spend an engineering week on it.</p>



<h2 class="wp-block-heading">Your billing model decides how much the format matters</h2>



<p class="wp-block-paragraph">Before comparing formats, work out what you are actually charged for. The same conversion produces very different savings depending on which of these you are on.</p>



<ul class="wp-block-list"><li><strong>Bytes scanned from object storage.</strong> Amazon Athena is the clearest case. You are billed for the bytes a query reads out of S3, rounded up, with a minimum charge per query. Here a format change converts almost directly into money.</li><li><strong>Logical bytes processed.</strong> BigQuery&#8217;s on-demand model bills on the uncompressed logical size of the columns a query references in a native table, not on the compressed size sitting on disk. Compressing harder does not lower that bill. Touching fewer columns does.</li><li><strong>Compute time.</strong> Snowflake, Databricks, EMR, and anything you run on your own hardware charge you for how long a machine is awake. The format shows up as wall clock time, which is a real cost but a much softer one.</li></ul>



<p class="wp-block-paragraph">The BigQuery detail catches people out. Google documents that on-demand cost is calculated from logical, uncompressed bytes for native tables, but that when you query <em>external</em> data stored as Parquet or ORC, the bytes charged are limited to the columns BigQuery actually reads. The same file format therefore sits on opposite sides of the billing line depending on whether the data was loaded or is being referenced in place. Both Athena and BigQuery also apply a 10 MB minimum per query, and BigQuery applies it per table referenced, so a swarm of small queries against small tables costs far more than the data volume suggests.</p>



<h2 class="wp-block-heading">CSV: cheap to produce, expensive to interrogate</h2>



<p class="wp-block-paragraph">CSV has no schema, no statistics, and no internal structure. Every value is text. The number 1000000000 takes ten bytes instead of the four it would take as a 32-bit integer, and because the layout is row-wise, there is nothing for a column-aware compressor to exploit.</p>



<h3 class="wp-block-heading">Where CSV wins</h3>



<ul class="wp-block-list"><li>A human needs to open it. Nothing else comes close.</li><li>Export from a legacy system that offers no other option.</li><li>Small data. Below a few megabytes, the metadata and encoding overhead of a columnar file is not repaid.</li><li>Streaming appends. You can write a line at a time and stop whenever you like.</li></ul>



<h3 class="wp-block-heading">Where CSV does not</h3>



<p class="wp-block-paragraph">Two problems, and the second is worse than the first.</p>



<p class="wp-block-paragraph">The first is that the reader has to parse every byte, even the bytes it intends to throw away. The DuckDB team published a comparison on TPC-H data at scale factor 20, run on a laptop, reporting the median of five runs, and it is the most honest set of numbers I have seen on this. Loading the <code>lineitem</code> table took roughly 11.8 seconds from CSV against 5.2 seconds from Snappy-compressed Parquet, with the CSV file about five times larger on disk. That is a 2x gap on load, not the order of magnitude most people assume. Modern CSV readers are genuinely fast.</p>



<p class="wp-block-paragraph">The gap opens when you query the files directly rather than loading them. On the same setup, TPC-H Q1 ran in about 6.7 seconds against CSV and 0.9 seconds against Parquet, and the join-heavy Q21 took roughly 20 seconds against 2.1. That difference is not parsing throughput. It is that the Parquet reader can skip row groups using statistics and skip unreferenced columns entirely, while the CSV reader cannot skip anything, because nothing in the file tells it what it would be skipping.</p>



<p class="wp-block-paragraph">The second problem is dialect. There is an RFC for CSV and it is widely ignored. Quoting, escaping, embedded newlines, date formats, and whether <code>NA</code> means null or Namibia are conventions rather than rules, so every consumer re-guesses and sooner or later one guesses differently from the others. That failure does not raise an error. It produces wrong numbers in a report nobody questions.</p>



<h2 class="wp-block-heading">JSON: you pay for the schema on every single row</h2>



<p class="wp-block-paragraph">Newline-delimited JSON is what most event pipelines emit, because a producer can add a field without coordinating with anyone downstream. That flexibility is worth something real. It is also why JSON is usually the most expensive format on a bytes-scanned bill: every record repeats every key name, so a field called <code>customer_subscription_status</code> carries those thirty-odd bytes on every row, forever.</p>



<h3 class="wp-block-heading">Where JSON wins</h3>



<ul class="wp-block-list"><li>Landing zone for data whose shape you do not control. Accept it, keep the raw copy, argue about the schema later.</li><li>Genuinely nested or sparse records, where flattening into columns would give you a table that is mostly nulls.</li><li>Debugging. Being able to decompress one file and read a record with your eyes is worth a lot at three in the morning.</li><li>API and webhook payloads, where it is simply what arrives.</li></ul>



<h3 class="wp-block-heading">Where JSON does not</h3>



<p class="wp-block-paragraph">As a query target at any serious volume. You are paying scan charges on repeated key names, on quote characters, and on the fact that every number is stored as text. Compressing the files helps storage and helps the scan bill on Athena, but it cannot give the engine anything to skip, and gzip brings a separate problem covered below.</p>



<p class="wp-block-paragraph">The nested-data argument for keeping JSON has also weakened. Parquet now has a ratified Variant type for semi-structured data, with support arriving across Delta Lake, Iceberg, and Spark, plus a shredding scheme that pulls frequently accessed fields out into real columns while leaving the rest flexible. If you are keeping JSON purely because your records have unpredictable shape, that reason is worth revisiting rather than treating as settled.</p>



<h2 class="wp-block-heading">Parquet: what the file is actually doing</h2>



<p class="wp-block-paragraph">Understanding the layout is what lets you predict whether a migration will pay for itself, so it is worth thirty seconds.</p>



<p class="wp-block-paragraph">A Parquet file is divided into row groups. Inside each row group, every column is stored as its own contiguous column chunk, and each chunk is split into pages. The footer at the end of the file carries the schema and, per column chunk, statistics such as minimum, maximum, and null count. That structure is what makes the two headline optimisations possible:</p>



<ul class="wp-block-list"><li><strong>Column pruning.</strong> The reader consults the footer, works out the byte ranges of the three columns you asked for, and never issues a read for the other forty-seven.</li><li><strong>Predicate pushdown.</strong> If a row group&#8217;s maximum value for <code>event_date</code> falls below your filter&#8217;s lower bound, the entire row group is skipped without being read.</li></ul>



<p class="wp-block-paragraph">Two later additions refine this. The page index stores per-page minimum and maximum values in a structure near the footer, so a reader doing a selective scan can locate matching pages without walking every page header, which previously meant pulling most of the column off disk anyway. Bloom filters cover the case statistics cannot help with at all: a high-cardinality column such as a user ID, where every row group&#8217;s min and max span the whole value range and pruning by range is useless. A bloom filter can say that a value is definitely not present in a row group. DuckDB even exposes <code>parquet_bloom_probe</code> so you can check which row groups a given value would eliminate.</p>



<p class="wp-block-paragraph">Encoding does the rest of the work. Because a column chunk holds values of a single type, dictionary and run-length encoding apply before any general-purpose compressor starts. Low-cardinality columns such as country codes or status flags compress extremely well. UUIDs and free text do not, and writers often disable dictionary encoding for them. This is why compression ratios quoted in blog posts are close to meaningless for your data: the ratio is a property of your cardinality, not of the format.</p>



<h3 class="wp-block-heading">Where Parquet does not win</h3>



<ul class="wp-block-list"><li><strong>Point lookups.</strong> Fetching one complete row by ID means touching every column chunk. A row format reads a single contiguous block instead.</li><li><strong>Row-level updates.</strong> Parquet files are immutable. Changing one row means rewriting the file, which is precisely the problem Iceberg and Delta Lake exist to solve on top of it.</li><li><strong>Streaming writes.</strong> The schema lives in the footer, so a writer has to finish the file before it is readable. Small-batch writers produce small files, which carries its own tax.</li><li><strong>Tiny datasets.</strong> Below a few megabytes, footer and row group overhead is not worth paying.</li><li><strong>Human inspection.</strong> It is binary. You need tooling, and someone will eventually ask you for a CSV export anyway.</li></ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Where the savings quietly disappear</h2>



<p class="wp-block-paragraph">Every one of these has cost someone a migration that looked excellent in the design document.</p>



<ul class="wp-block-list"><li><strong>Small files.</strong> Each file costs a metadata read, an object storage request, and reader setup. Thousands of 2 MB Parquet files will lose to a handful of large CSVs. The commonly cited target is roughly 128 MB to 1 GB per file, and you need a compaction step to get there if your writer emits micro-batches.</li><li><strong>One enormous row group.</strong> Parallelism in most engines is per row group. A file containing a single row group is processed by a single thread no matter how many cores you have. Check this rather than assuming the writer got it right.</li><li><strong>Select-everything queries in the reporting layer.</strong> Column pruning cannot help a query that asks for all columns. This is the one from the opening paragraph, and it is usually a BI tool doing it rather than a person.</li><li><strong>No partitioning.</strong> Format reduces bytes per row. Partitioning reduces how many rows are considered at all. The two multiply, and partitioning is normally the larger lever. Converting without partitioning leaves most of the money on the table.</li><li><strong>Gzipped text.</strong> AWS documents that Parquet and ORC are always splittable, because they compress sections independently and carry metadata pointing at those sections, while most text compression formats are not. Bzip2 and LZO are splittable; gzip is not. One 5 GB gzipped CSV cannot be divided among workers, so it is read start to finish by one reader. The compression made storage cheaper and the query slower.</li><li><strong>Uppercase file extensions.</strong> A small operational trap worth knowing: Athena determines the compression type of CSV and JSON data from the file extension, does not recognise uppercase extensions such as <code>.GZ</code>, and treats a file with no extension as uncompressed plain text.</li></ul>



<h2 class="wp-block-heading">How to model the change before you migrate</h2>



<p class="wp-block-paragraph">Do not migrate on faith. This takes an afternoon and answers the question properly.</p>



<ol class="wp-block-list"><li><strong>Find out what you scan today.</strong> Every Athena query execution carries a <code>DataScannedInBytes</code> statistic. Pull it for a representative window and rank queries by bytes scanned. Almost always a handful of them are most of the bill, and those are the only ones worth optimising.</li><li><strong>Convert one day of data, not all of it.</strong> A single partition is enough to measure with.</li><li><strong>Look inside the resulting file</strong> before trusting it. Row group count, row group size, and compression codec decide whether the file behaves.</li><li><strong>Re-run the same queries</strong> against the converted partition and compare bytes scanned. Measured, not estimated.</li><li><strong>Price the write side.</strong> Conversion is not free. Glue jobs, EMR time, and Lambda invocations all cost money, and if a dataset is queried twice a month the conversion may never pay back.</li><li><strong>Check who else reads the files.</strong> One downstream consumer that only speaks CSV turns a clean migration into a dual-write pipeline.</li></ol>



<p class="wp-block-paragraph">For step three, DuckDB is the fastest way to inspect a file without standing up a cluster. It runs happily on a laptop or a small VPS from a provider like Contabo or InterServer, which is usually all the compute this kind of analysis needs:</p>



<pre class="wp-block-code"><code>-- one row per column chunk: row groups, sizes, codec
SELECT row_group_id, row_group_num_rows, row_group_bytes,
       path_in_schema, compression
FROM parquet_metadata('events.parquet')
LIMIT 20;

-- convert a sample and control the row group size
COPY (SELECT * FROM 'events.json')
TO 'events.parquet'
(FORMAT parquet, COMPRESSION zstd, ROW_GROUP_SIZE 1000000);</code></pre>



<p class="wp-block-paragraph">The <code>parquet_metadata</code> function reads the footer and returns a row per column chunk, so you can see straight away whether your writer produced one row group or two hundred, and which codec it used. If you would rather stay in Python, <code>pyarrow.parquet.ParquetFile</code> exposes the same footer through its <code>metadata</code> attribute, including per-column statistics and whether a bloom filter offset is present.</p>



<p class="wp-block-paragraph">If the data already lives in S3 and you want the engine you are already paying for to do the conversion, Athena&#8217;s CTAS handles it in one statement. The AWS documentation lists GZIP and SNAPPY as the compression options for Parquet output here, with GZIP as the default:</p>



<pre class="wp-block-code"><code>CREATE TABLE events_parquet
WITH (
  format = 'PARQUET',
  write_compression = 'SNAPPY',
  external_location = 's3://your-bucket/events-parquet/',
  partitioned_by = ARRAY['event_date']
) AS
SELECT event_id, user_id, event_type, amount, event_date
FROM events_json;</code></pre>



<p class="wp-block-paragraph">Three things to know before running it. Partition columns have to come last in the SELECT list. CTAS refuses to write into a location that already contains data, so a re-run needs the prefix cleared first. And Athena has a write limit of 100 partitions per CTAS statement, which means a large backfill has to be chunked rather than fired off in one go.</p>



<h2 class="wp-block-heading">How I would decide</h2>



<ul class="wp-block-list"><li><strong>Keep the raw landing zone in whatever arrives.</strong> Usually JSON. Do not convert on ingest and discard the original; you will want it the first time a schema assumption turns out to be wrong.</li><li><strong>Convert once, at the point where data becomes queryable.</strong> Partitioned Parquet for anything a dashboard or an analyst touches repeatedly.</li><li><strong>Partition before you optimise the format.</strong> If there is budget for exactly one change, partition on the column your queries actually filter on.</li><li><strong>Choose the codec by read frequency.</strong> Zstd where scanned bytes are the bill and the data is read often, Snappy where decompression speed matters more, gzip only when a consumer demands it.</li><li><strong>Leave small CSVs alone.</strong> Reference tables, seed data, config. Converting a 4 MB lookup file to Parquet is busywork.</li><li><strong>Watch the scan bill, not the storage bill.</strong> Storage is cents. Scans are dollars. Athena workgroups can cap bytes scanned per query, and that limit is worth setting before someone discovers what a full table scan costs.</li></ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is Parquet always cheaper than CSV?</h3>



<p class="wp-block-paragraph">On a bytes-scanned billing model, for analytical queries that touch a subset of columns, almost always. It is not cheaper for point lookups, for very small datasets, for row-level updates, or when the query selects every column and filters on nothing. It also costs compute to produce, which matters when the data is rarely read.</p>



<h3 class="wp-block-heading">How much smaller is Parquet than CSV in practice?</h3>



<p class="wp-block-paragraph">It depends entirely on cardinality, and anyone quoting a single ratio is guessing about your data. In the DuckDB TPC-H comparison the Parquet file came out around five times smaller than the CSV. Columns with few distinct values do far better than that; UUID and free-text columns do far worse. Measure on one partition of your own data instead of trusting a headline number.</p>



<h3 class="wp-block-heading">Does compressing my JSON or CSV give me the same savings?</h3>



<p class="wp-block-paragraph">Partially. On Athena you are billed for bytes scanned before decompression, so compression does reduce the bill. What it cannot do is let the engine skip columns or row groups, and if you reach for gzip you lose splittability, which can make queries slower even as they get cheaper. Compression and columnar layout solve different halves of the problem.</p>



<h3 class="wp-block-heading">Should I use Parquet or ORC?</h3>



<p class="wp-block-paragraph">For most teams that difference is much smaller than the difference between either of them and CSV. Parquet has wider support across engines and languages, which is usually the deciding factor. ORC has a long history in Hive-centric stacks and remains a reasonable choice if that is where you already live. Both are splittable and both support predicate pushdown.</p>



<h3 class="wp-block-heading">Why did my costs not drop after converting to Parquet?</h3>



<p class="wp-block-paragraph">The usual suspects, in the order I would check them: the table is not partitioned, the queries select all columns, the files are too small, or the writer produced one enormous row group. Pull <code>DataScannedInBytes</code> for the same query before and after. If the number barely moved, the engine was not able to skip anything, and the problem is the query or the layout rather than the format.</p>



<h3 class="wp-block-heading">Is JSON still worth keeping now that Parquet has a Variant type?</h3>



<p class="wp-block-paragraph">As a raw landing format and a transport format, yes. As the thing analysts query directly, the case is weaker than it used to be. Variant support is still spreading across engines, so confirm your query engine handles it before designing around it rather than assuming it is available everywhere.</p>



<h3 class="wp-block-heading">What file size should I aim for?</h3>



<p class="wp-block-paragraph">Roughly 128 MB to 1 GB per file is the widely used guidance, with row groups sized so a file holds several of them rather than one. Below that range you pay per-file overhead on every query; above it you can starve parallelism if the row group layout is wrong.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The <strong>Parquet vs JSON vs CSV</strong> question is not really about the formats. It is about how much of your data the engine is permitted to ignore. Parquet hands an engine the most opportunities to skip work, JSON gives it almost none, and CSV sits between the two while making every reader re-guess the schema.</p>



<p class="wp-block-paragraph">An opportunity to skip work is not the same as skipping it, though. If the query asks for every column, if the table is not partitioned, if the files are too small or the row groups too large, you have paid the conversion cost and bought a smaller file with the same bill attached. Measure bytes scanned before and after on a single partition. That one number tells you the truth in an afternoon, and it beats any comparison table you will read, including this one.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your data lake costs?</h2>



<p class="wp-block-paragraph">Most format work starts as a cost problem and ends as a layout problem. Things I can help with:</p>



<ul class="wp-block-list"><li>Auditing which queries actually drive your Athena or BigQuery bill, and whether a format change would move them at all</li><li>Designing the partition scheme and file sizing before the conversion, so the migration is worth running</li><li>Building the CSV or JSON to partitioned Parquet conversion as a Glue, Spark, or DuckDB job, including compaction for small-file pipelines</li><li>Diagnosing conversions that did not pay off: row group layout, dictionary encoding, splittability, gzip traps</li><li>Setting scan limits, workgroup controls, and alerting so one bad query cannot produce a surprise invoice</li><li>Retrofitting Iceberg or Delta Lake where the real requirement is row-level updates rather than a different file format</li></ul>



<p class="wp-block-paragraph">If you want a concrete opinion rather than a general one, send me a query&#8217;s execution statistics along with the table schema and a listing of one partition, and I will tell you where your bytes are going.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/parquet-vs-json-vs-csv-cost-performance-2/">Parquet vs JSON vs CSV: Where the Money Actually Goes</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>One Customer, Four Systems: Unifying Shopify, Stripe, CRM and Support Data</title>
		<link>https://john-nessime.com/blog/technical-guides/unify-shopify-stripe-crm-support-data/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 29 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Customer 360]]></category>
		<category><![CDATA[Customer Support Analytics]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[Data Warehouse]]></category>
		<category><![CDATA[dbt]]></category>
		<category><![CDATA[Ecommerce Analytics]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Event-Driven Architecture]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Identity Resolution]]></category>
		<category><![CDATA[PII Redaction]]></category>
		<category><![CDATA[Reverse ETL]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[Stripe]]></category>
		<category><![CDATA[Webhooks]]></category>
		<category><![CDATA[Zendesk]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=350</guid>

					<description><![CDATA[<p>Three CRM records, one person, and nothing in the logs to explain it. A practical walkthrough of what actually breaks when you unify Shopify, Stripe, CRM and support data: guest-checkout identity, duplicate and out-of-order webhooks, revenue that never ties to payouts, deletion requests that miss half your copies, and reverse ETL write loops. Organized by failure family, with schema and handler patterns you can apply directly.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/unify-shopify-stripe-crm-support-data/">One Customer, Four Systems: Unifying Shopify, Stripe, CRM and Support Data</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The project usually starts with a screenshot. Someone in support pastes three CRM records into a channel, all carrying the same person&#8217;s name, and asks which one is real. One has the order history. One has the subscription. The third has four tickets attached and no orders at all.</p>



<p class="wp-block-paragraph">Nobody broke anything. Shopify, Stripe, the CRM and the helpdesk each created a customer record using whatever identifier they had at the moment they needed one, and none of those identifiers agreed. The systems are all correct in isolation. The person is the thing that got lost.</p>



<p class="wp-block-paragraph">This post covers what actually breaks when you unify Shopify, Stripe, CRM and support data, and how to build the join so it survives guest checkouts, duplicate webhooks, refunds, currency conversion and deletion requests. It&#8217;s organized by failure family rather than by tool, because the tools get acquired and renamed every couple of years and the failures never change.</p>



<h2 class="wp-block-heading">Failure family one: the join key is not the email address</h2>



<p class="wp-block-paragraph">Every one of these systems will happily give you an email address, which is exactly why so many pipelines join on it and quietly produce garbage.</p>



<p class="wp-block-paragraph">Start with where identity actually lives in each system:</p>



<ul class="wp-block-list">
<li><strong>Shopify</strong> has a customer ID, but an order does not have to be attached to one. Guest checkout produces an order with an email on it and no durable customer record behind it.</li>

<li><strong>Stripe</strong> has a customer ID, but only if a Customer object was created. Checkout Sessions that don&#8217;t create one are associated with guest customers instead, which Stripe groups in the Dashboard based on the same card, email or phone. That grouping is read-only and it is not an ID you can join on.</li>

<li><strong>The CRM</strong> keys on email in almost every default configuration, and cheerfully creates a second contact when the email differs by one character.</li>

<li><strong>The helpdesk</strong> keys on the requester&#8217;s email, which is whatever the person typed into a contact form or sent the ticket from. It is very often not the address on the order.</li>
</ul>



<p class="wp-block-paragraph">Here&#8217;s the part that stays invisible until someone finally checks: nothing errors. Every dashboard still returns a number. Lifetime value gets silently split across three profiles, so your LTV looks lower and your repeat-purchase rate looks worse than it is. Support metrics look <em>better</em> than they are, because a customer coming back with the same unresolved problem shows up as a brand new contact. You can run like that for a year without a single failed job.</p>



<h3 class="wp-block-heading">Build an identifier table, not a fuzzy match</h3>



<p class="wp-block-paragraph">The fix is boring and it works: a surrogate person key, and a separate table of every identifier ever observed for that person, with the source that reported it.</p>



<pre class="wp-block-code"><code>-- One row per real human, generated by you, owned by you.
CREATE TABLE person (
  person_key      uuid PRIMARY KEY,
  created_at      timestamptz NOT NULL DEFAULT now(),
  merged_into     uuid NULL REFERENCES person(person_key)
);

-- One row per identifier per source. This is the whole trick.
CREATE TABLE person_identifier (
  person_key      uuid NOT NULL REFERENCES person(person_key),
  source          text NOT NULL,   -- shopify | stripe | crm | helpdesk
  id_type         text NOT NULL,   -- email | phone | shopify_customer_id | stripe_customer_id
  id_value        text NOT NULL,   -- already normalized
  first_seen_at   timestamptz NOT NULL,
  last_seen_at    timestamptz NOT NULL,
  PRIMARY KEY (source, id_type, id_value)
);</code></pre>



<p class="wp-block-paragraph">Normalization happens before the row is written, not in the join. Lowercase and trim emails, store phone numbers in E.164, strip whitespace from external IDs. Resist the urge to get clever with Gmail&#8217;s dot and plus handling. Stripping <code>+tags</code> is a business policy decision with real consequences, not a technical detail, and the same rule applied to a domain that treats plus addresses as distinct mailboxes will merge two genuinely different people.</p>



<p class="wp-block-paragraph">Deterministic matching first, always. Exact matches on shared identifiers, transitively closed. Probabilistic matching on name plus address plus device is worth reaching for only when you have exhausted the deterministic keys, and it has an asymmetric failure mode: a household sharing one card gets collapsed into one person, and unmerging afterwards is expensive because every downstream system has already copied the merged key into its own records. Merges are cheap to make and painful to undo. Bias toward leaving two records separate.</p>



<p class="wp-block-paragraph">One thing worth doing early on the Stripe side: when you create a Customer, write your own identifier into <code>metadata</code>. Stripe allows fifty key-value pairs per object, with keys up to forty characters and values up to five hundred, and it only returns metadata for requests made with a secret key. That&#8217;s enough room for a person key and an order reference, which turns a future fuzzy join into an exact one. Never put anything sensitive there.</p>



<h2 class="wp-block-heading">Failure family two: webhooks arrive twice, out of order, or not at all</h2>



<p class="wp-block-paragraph">Both platforms are explicit about this, and both get treated as if they weren&#8217;t.</p>



<p class="wp-block-paragraph">Stripe delivers at least once, does not guarantee ordering, and retries failed deliveries with exponential backoff for up to three days. The event <code>id</code> (the one starting <code>evt_</code>) stays identical across every retry, which makes it your deduplication key. Shopify sends <code>X-Shopify-Webhook-Id</code> for the same purpose, signs the raw body into <code>X-Shopify-Hmac-Sha256</code>, and states plainly that ordering isn&#8217;t guaranteed within a topic or across topics for the same resource. Shopify&#8217;s own documentation goes further and tells you not to rely on webhooks as your only source of data.</p>



<p class="wp-block-paragraph">For a customer-data pipeline specifically, out-of-order delivery is nastier than duplication. Consider a <code>customers/update</code> that changes someone&#8217;s email arriving before the <code>orders/create</code> that was placed under the old address. Process those in arrival order and you attach the order to a person who doesn&#8217;t exist yet, or worse, to the wrong one. Order your writes by the event&#8217;s own timestamp, using <code>X-Shopify-Triggered-At</code> or the <code>updated_at</code> in the payload, and make profile updates last-writer-wins on that timestamp rather than on insertion time.</p>



<h3 class="wp-block-heading">Do almost nothing in the handler</h3>



<p class="wp-block-paragraph">Verify the signature, claim the event ID, write the raw body to durable storage, return 2xx. That&#8217;s it. Identity resolution, CRM lookups and enrichment calls all belong in a worker reading from that store. A handler that tries to resolve identity inline will eventually be slow enough to time out, which triggers a retry, which arrives while the first attempt is still running.</p>



<p class="wp-block-paragraph">Make the claim atomic rather than check-then-act, because two retries genuinely can land concurrently:</p>



<pre class="wp-block-code"><code>-- Atomic claim. If this inserts zero rows, someone already has it.
INSERT INTO webhook_event (source, event_id, topic, received_at, payload)
VALUES ($1, $2, $3, now(), $4)
ON CONFLICT (source, event_id) DO NOTHING
RETURNING event_id;</code></pre>



<p class="wp-block-paragraph">Return a 2xx for a duplicate. A duplicate is a success from the sender&#8217;s point of view, and returning an error just schedules more of them.</p>



<p class="wp-block-paragraph">Signature verification has one recurring trap on the Shopify side: it runs against the <em>raw</em> body. Any JSON body parser that runs first will change the bytes and every signature will fail. Capture the raw body before parsing, and compare in constant time.</p>



<pre class="wp-block-code"><code>import base64, hashlib, hmac

def verify_shopify(raw_body: bytes, header_value: str, client_secret: str) -&gt; bool:
    digest = hmac.new(
        client_secret.encode("utf-8"),
        raw_body,
        hashlib.sha256,
    ).digest()
    expected = base64.b64encode(digest).decode("utf-8")
    return hmac.compare_digest(expected, header_value)</code></pre>



<p class="wp-block-paragraph">Worth knowing before you spend an afternoon on it: after rotating an app&#8217;s client secret, Shopify notes it can take up to an hour before signatures are generated with the new one. If verification starts failing right after a rotation, that&#8217;s usually why.</p>



<p class="wp-block-paragraph">And because delivery isn&#8217;t guaranteed, webhooks alone are never enough. Run a reconciliation job that pulls orders and customers modified since the last watermark and fills the gaps. This is the job that saves you during an outage on your side, and it&#8217;s the one people skip. If you&#8217;d rather not build the buffering and replay layer, managed webhook infrastructure like Hookdeck exists for exactly this, and self-hosting a small queue and worker on a modest VPS from somewhere like Contabo or InterServer is perfectly reasonable too. The point is that something has to catch what the webhook missed.</p>



<h2 class="wp-block-heading">Failure family three: &#8220;revenue&#8221; means four different things</h2>



<p class="wp-block-paragraph">This is the one that turns into a meeting. Finance says the number is wrong. Analytics says the number is right. Both are correct, because they&#8217;re measuring different things and nobody wrote down which.</p>



<ul class="wp-block-list">
<li>Shopify reports gross sales at the moment the order is placed, in the store&#8217;s currency and the store&#8217;s timezone.</li>

<li>Stripe pays out in batches, net of processing fees, refunds and disputes, on its own schedule, in the settlement currency of the account.</li>

<li>A refund issued in Shopify and a refund issued in Stripe are two different events that may or may not be the same underlying money movement.</li>

<li>A dispute lands days or weeks after the order and reverses revenue in a period that has already been reported.</li>
</ul>



<p class="wp-block-paragraph">There&#8217;s a structural trap here too. If the store runs Shopify Payments, there is no Stripe account for you to query. Shopify Payments is built on Stripe&#8217;s processing infrastructure, but it&#8217;s Shopify&#8217;s gateway, and payouts, fees and disputes live in the Shopify admin rather than behind a Stripe API key. Plenty of businesses end up with Shopify Payments handling storefront orders and a separate Stripe account handling subscriptions or invoicing, which means two payment systems, two payout schedules and one customer who has no idea any of this exists.</p>



<p class="wp-block-paragraph">Do not build a single <code>revenue</code> column. Build separate fact tables at their natural grain (orders, order line items, payments, refunds, disputes, payouts) plus a bridging table that links a payout back to the transactions inside it. Then let each team define its own metric on top. Finance gets net settled cash. Marketing gets gross booked revenue. Neither has to be wrong for the other to be right.</p>



<p class="wp-block-paragraph">Store every timestamp in UTC and keep the original timezone as a separate column. Store every amount in minor units as an integer alongside its currency code, and store the FX rate that was actually applied rather than recalculating it later from a rates table. Recalculated FX is the single most common source of &#8220;the number moved and nobody changed anything.&#8221;</p>



<p class="wp-block-paragraph">On ingestion, Stripe Data Pipeline can push Stripe data directly into Snowflake, Redshift or Databricks as a data share, which removes a connector from the picture if you&#8217;re already on one of those. Otherwise a managed ELT tool like Fivetran or a self-hosted Airbyte will handle Shopify, Stripe, the CRM and the helpdesk with off-the-shelf connectors. Both routes are fine. The connector is genuinely the easy part.</p>



<h2 class="wp-block-heading">Failure family four: deletion is the requirement nobody models</h2>



<p class="wp-block-paragraph">If you&#8217;re building this as a Shopify app, three compliance webhooks are mandatory: <code>customers/data_request</code>, <code>customers/redact</code> and <code>shop/redact</code>. You acknowledge with a 200 and complete the action within thirty days. <code>shop/redact</code> fires forty-eight hours after uninstall and covers the shop&#8217;s data. These are enforced at app review, so you&#8217;ll find out about them either way.</p>



<p class="wp-block-paragraph">The interesting problem isn&#8217;t receiving the webhook. It&#8217;s that by the time it arrives, that person&#8217;s email exists in your raw landing zone, your staging tables, your identity table, your customer view, a BI extract, the CRM, the helpdesk, and a Slack message from a debugging session six months ago. A delete that only touches the golden record is not a delete.</p>



<p class="wp-block-paragraph">The design decision that makes this tractable is to hold personal data in exactly one place. Every downstream table carries <code>person_key</code> and nothing else. Facts, aggregates, models, extracts and syncs all reference the key. When a redaction request lands, you redact one row set in the identifier table and the rest of the warehouse is already anonymous. Getting this right at schema-design time costs an afternoon; retrofitting it costs a quarter.</p>



<p class="wp-block-paragraph">Practical pieces that make it real:</p>



<ul class="wp-block-list">
<li>Tag every table and column that can contain an identifier, in your catalog or in dbt metadata, and treat the tag list as the deletion scope.</li>

<li>Partition the raw landing zone so you can rewrite a partition rather than scan an entire bucket looking for one email.</li>

<li>Keep a deletion queue with a status per downstream system, including the CRM and the helpdesk, which have their own APIs and their own idea of what deletion means.</li>

<li>Write down what you retain and why. Financial records generally have to survive a deletion request for tax and dispute reasons. Say so rather than promising a hard delete you can&#8217;t perform.</li>
</ul>



<h2 class="wp-block-heading">Failure family five: pushing the unified view back out</h2>



<p class="wp-block-paragraph">A customer view nobody can act on is a report. The value shows up when a support agent opens a ticket and can see subscription status and lifetime order count without switching tabs.</p>



<p class="wp-block-paragraph">That&#8217;s reverse ETL, and the market is well served: Hightouch, Census (now part of Fivetran), Polytomic, RudderStack, and iPaaS-style tools like Workato if you need branching logic rather than field syncs. The warehouse-native argument is straightforward and mostly correct: your models already live there, and you avoid maintaining a second copy of the customer profile inside a vendor&#8217;s rigid schema. The counter-argument is also real. Reverse ETL is batch by nature, so if you need sub-minute reaction to an event, a packaged CDP or a direct event pipeline will serve you better, and you&#8217;ll be running several vendors plus warehouse compute plus the people to maintain the models.</p>



<p class="wp-block-paragraph">Three things go wrong here more or less every time.</p>



<p class="wp-block-paragraph"><strong>The write loop.</strong> You compute a churn-risk field and sync it into the CRM. Your ELT connector reads the CRM back into the warehouse. Now the computed field is an input to the model that computes it. This is genuinely hard to spot because nothing errors, the value just drifts. Namespace every synced field with a consistent prefix and explicitly exclude that namespace from ingestion.</p>



<p class="wp-block-paragraph"><strong>The resync stampede.</strong> Incremental syncs push diffs and stay small. Change a model definition and the next run wants to update every record, which will meet a helpdesk API rate limit at speed. Cap the batch size, and treat a full resync as a planned operation rather than something a merge to main can trigger.</p>



<p class="wp-block-paragraph"><strong>Syncing logic instead of keys.</strong> Push <code>person_key</code> into a custom field on the CRM contact and the helpdesk user. It makes the key searchable by agents, it gives the next pipeline a deterministic join, and it stops three teams from each rebuilding their own version of your matching rules.</p>



<h2 class="wp-block-heading">A pipeline shape that holds up</h2>



<p class="wp-block-paragraph">Put together, the order of operations matters more than the tool choice:</p>



<ol class="wp-block-list">
<li>Land raw. Every webhook body and every API page goes to immutable storage first, keyed by source and event ID, before anything interprets it.</li>

<li>Deduplicate on the provider&#8217;s event ID, atomically, with a retention window longer than the provider&#8217;s retry window.</li>

<li>Normalize identifiers into the identifier table as rows arrive, with the source recorded.</li>

<li>Resolve identity deterministically into <code>person_key</code>, as a scheduled job, never inside a request handler.</li>

<li>Build fact tables per system at their own grain, keyed on <code>person_key</code>, with no raw PII.</li>

<li>Model the customer view on top, in dbt or whatever your team already uses. The Fivetran <code>dbt_customer360</code> package is a reasonable starting point if you want to see how someone else structured the identity step.</li>

<li>Activate a small, named, namespaced set of fields back into the CRM and helpdesk.</li>

<li>Run reconciliation and freshness checks on a schedule, and alert on them like you would any other production job.</li>
</ol>



<h2 class="wp-block-heading">Troubleshooting: symptom to root cause</h2>



<p class="wp-block-paragraph">These are the ones that come up repeatedly, and the check that usually settles them.</p>



<ul class="wp-block-list">
<li><strong>Warehouse customer count is much higher than Shopify&#8217;s.</strong> Guest orders are minting a new person on every purchase. Check how many <code>person</code> rows have exactly one identifier and one order.</li>

<li><strong>Warehouse count is much lower.</strong> An over-eager merge rule. Look for person keys with an implausible number of distinct emails or shipping addresses attached.</li>

<li><strong>BI revenue doesn&#8217;t match Stripe payouts.</strong> Grain mismatch, not a data loss. Reconcile a single payout against its constituent transactions before touching anything else.</li>

<li><strong>A CRM field keeps flipping between two values.</strong> Write loop. Check whether the field&#8217;s namespace is excluded from your ELT source configuration.</li>

<li><strong>Recent orders missing, older ones fine.</strong> Webhook delivery gap. Confirm the reconciliation job ran, and check its watermark rather than trusting its exit code.</li>

<li><strong>Every Shopify signature suddenly fails.</strong> Either a body parser is running before verification, or the client secret was rotated within the last hour.</li>

<li><strong>Tickets attached to the wrong customer after a merge.</strong> The merge happened in the warehouse but the synced key in the helpdesk still points at the old person. Merges need to propagate outward, not just inward.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Joining on email in the model layer instead of resolving identity once, upstream.</li>

<li>Treating webhooks as the system of record instead of as a low-latency hint, with a reconciliation pull behind them.</li>

<li>Doing identity resolution, enrichment or CRM writes inside the webhook handler.</li>

<li>Ordering profile updates by arrival time rather than by the event&#8217;s own timestamp.</li>

<li>One <code>revenue</code> column that finance and marketing are both told to use.</li>

<li>Storing amounts as floats, or recalculating historical FX from a rates table.</li>

<li>Copying PII into every downstream table because it was convenient at the time.</li>

<li>Reaching for probabilistic matching before the deterministic keys have been exhausted.</li>

<li>Assuming a Shopify store on Shopify Payments has a Stripe account you can query.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>Own the surrogate key. Never let a vendor&#8217;s internal ID become the primary identity of a person in your systems.</li>

<li>Write your <code>person_key</code> into Stripe <code>metadata</code> and Shopify customer metafields at creation time, so future joins are exact rather than inferred.</li>

<li>Keep raw payloads immutable and replayable. Every model bug you&#8217;ll ever have is cheaper to fix if you can reprocess from raw.</li>

<li>Version your matching rules and record which version produced each merge, so you can audit and reverse a bad rule.</li>

<li>Alert on freshness and on row-count deltas per source, not just on job success. A connector returning zero rows exits cleanly.</li>

<li>Reconcile one payout end to end by hand before you trust the automated version. It&#8217;s the fastest way to find a grain error.</li>

<li>Document the metric definitions next to the models. Most &#8220;the data is wrong&#8221; tickets are definition disputes.</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Do I need a CDP to unify Shopify, Stripe, CRM and support data?</h3>



<p class="wp-block-paragraph">Not necessarily. If you already run a warehouse and have someone comfortable with dbt, ELT plus modelling plus reverse ETL covers the same ground with more control over the data model. A packaged CDP earns its cost when you need real-time behavioural segmentation, when nobody on the team can maintain models, or when marketing needs to ship changes without waiting on data engineering.</p>



<h3 class="wp-block-heading">How do I handle guest checkout customers?</h3>



<p class="wp-block-paragraph">Create a person from the order&#8217;s email anyway, and mark the record as identity-weak. When that email later appears attached to a real Shopify customer or Stripe Customer object, the deterministic match links them and the order history follows. The mistake is discarding guest orders because they have no customer ID, which is exactly the revenue you most want attributed.</p>



<h3 class="wp-block-heading">Should I sync everything into the CRM or keep it in the warehouse?</h3>



<p class="wp-block-paragraph">Keep the full picture in the warehouse and sync a deliberately small set of fields outward. The test is whether an agent or a rep will act on the field during a conversation. Lifetime value, subscription status, open ticket count and churn risk pass. A hundred-column profile does not, and it turns every model change into a rate-limit incident.</p>



<h3 class="wp-block-heading">Why don&#8217;t my Shopify sales match my Stripe payouts?</h3>



<p class="wp-block-paragraph">Because they&#8217;re measuring different things at different times. Shopify shows gross sales at order time; Stripe pays out batched amounts net of fees, refunds and disputes, and the timing differs by days. Add currency conversion and a timezone mismatch and the numbers will never tie without an explicit bridging table linking payouts to the transactions inside them.</p>



<h3 class="wp-block-heading">Can I use email as the primary key if my store requires account creation?</h3>



<p class="wp-block-paragraph">You can get away with it longer, but it&#8217;s still a bad primary key. People change email addresses, use one address for orders and another for support, and share addresses within a household. A surrogate key with an identifier table costs very little up front and means an email change is a new row rather than a broken join and a duplicate customer.</p>



<h3 class="wp-block-heading">How do I test the identity resolution before trusting it?</h3>



<p class="wp-block-paragraph">Run it in shadow mode. Compute person keys without activating them, then have support pick twenty customers they know are duplicates and twenty they know are distinct, and check what the resolver did. That exercise finds normalization bugs and over-eager merge rules faster than any aggregate metric, because the people who answer the tickets already know where the duplicates are.</p>



<h3 class="wp-block-heading">What&#8217;s the minimum viable version of this?</h3>



<p class="wp-block-paragraph">Raw landing with deduplication, an identifier table with deterministic matching, one fact table per source, and a single synced field into the helpdesk. That&#8217;s a couple of weeks of work and it removes most of the pain. Everything above it is refinement.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Conclusion: unify Shopify, Stripe, CRM and support data around a key you own</h2>



<p class="wp-block-paragraph">If you take one thing from this: the hard part of unifying Shopify, Stripe, CRM and support data is not moving the data. Connectors are commodity. The hard part is deciding what a customer <em>is</em>, writing that decision down as a key you own, and then refusing to let any downstream system quietly invent its own version of it.</p>



<p class="wp-block-paragraph">Do the identity work first, land everything raw so you can reprocess when you get it wrong, and keep personal data in one place so deletion stays a single operation. The dashboards will follow. Skip that order and you get exactly the screenshot this post opened with, eighteen months later and with a lot more rows behind it.</p>



<h2 class="wp-block-heading">Need help wiring this together?</h2>



<p class="wp-block-paragraph">I work on exactly this kind of plumbing: the unglamorous layer between an ecommerce platform, a payment processor and the tools your team actually opens all day. Things I can help with here:</p>



<ul class="wp-block-list">
<li>Designing the identity model (surrogate keys, identifier tables, deterministic matching rules) and auditing an existing one for over-merging.</li>

<li>Building idempotent, signature-verified webhook intake for Shopify and Stripe, with a dead-letter path and a reconciliation job behind it.</li>

<li>Setting up ELT into Snowflake, BigQuery, Redshift or Postgres, whether with Fivetran, self-hosted Airbyte, or Stripe Data Pipeline where it fits.</li>

<li>Modelling orders, payments, refunds, disputes and payouts so finance and marketing can each get a number they trust.</li>

<li>Reverse ETL back into the CRM and helpdesk without write loops, rate-limit incidents or a hundred junk fields.</li>

<li>Making deletion and data-subject requests actually work across the warehouse, the raw landing zone and every downstream copy.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;re stuck on a specific piece, send me the concrete thing: a webhook payload, a schema dump, the query that disagrees with your Stripe payout report. It&#8217;s much easier to say something useful about a real artifact than about a diagram.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/unify-shopify-stripe-crm-support-data/">One Customer, Four Systems: Unifying Shopify, Stripe, CRM and Support Data</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>As-Planned vs As-Built Analysis: Building a Platform That Survives Cross-Examination</title>
		<link>https://john-nessime.com/blog/technical-guides/as-planned-vs-as-built-analysis-platform/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 24 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Case Studies]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[As-Built Schedule]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[Capital Projects]]></category>
		<category><![CDATA[Construction Technology]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Lineage]]></category>
		<category><![CDATA[Delay Analysis]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Extension of Time]]></category>
		<category><![CDATA[Forensic Schedule Analysis]]></category>
		<category><![CDATA[Legal Tech]]></category>
		<category><![CDATA[OCR]]></category>
		<category><![CDATA[PostgreSQL]]></category>
		<category><![CDATA[Primavera P6]]></category>
		<category><![CDATA[Project Controls]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[XER Files]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=274</guid>

					<description><![CDATA[<p>Most as-planned vs as-built analysis compares the baseline to the last P6 update and calls the result an as-built. It isn't one. This is how to build the data layer underneath a delay analysis: versioned XER ingestion, activity identity across renumbering, a first-appearance table that proves when every actual date entered the record, calendar-safe float, and record linking that proposes candidates instead of asserting cause.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/as-planned-vs-as-built-analysis-platform/">As-Planned vs As-Built Analysis: Building a Platform That Survives Cross-Examination</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question that kills a delay analysis is never about methodology. It is five words from the other side&#8217;s expert: &#8220;which update did that come from?&#8221;</p>



<p class="wp-block-paragraph">Picture the experts&#8217; meeting. Your exhibit shows activity C-2140, structural steel erection to level four, finishing well behind where the baseline put it. He has the same XER files you do. He opens update 19 and points out that C-2140 carries no actual finish. Same in update 20. In update 21 an actual finish appears, and the date on it sits three weeks in the past, inside the window you already closed out.</p>



<p class="wp-block-paragraph">So which file is the record? And if one activity was back-filled a month after the fact, what does that say about the other four thousand?</p>



<p class="wp-block-paragraph">That is the failure an <strong>as-planned vs as-built analysis</strong> has to survive, and no amount of care inside the Excel workbook fixes it. It is a data problem before it is a scheduling problem. This post covers the platform underneath the analysis: how to ingest a long series of Primavera P6 updates, how to prove where every actual date came from, and how to produce output a tribunal can audit instead of having to trust.</p>



<h2 class="wp-block-heading">The as-built is not a file, and treating it as one is the invisible failure</h2>



<p class="wp-block-paragraph">Almost every quick as-planned vs as-built comparison does the same thing: take the approved baseline, take the last update in the sequence, join on activity ID, subtract the dates, sort by variance. It produces a clean-looking table in an afternoon.</p>



<p class="wp-block-paragraph">It is also the weakest possible construction of an as-built, for three reasons that never show up in the output.</p>



<ul class="wp-block-list">
<li><strong>A P6 update is a forecast document.</strong> It was produced to tell the employer when the job would finish, not to record what happened. Actual dates are a by-product of that exercise, entered by a planner working to a monthly deadline.</li>

<li><strong>The last update has been edited the most.</strong> By the end of a long job it has usually been re-baselined, partially renumbered, had calendars swapped, had activities merged or deleted. Every one of those edits is invisible in a two-file comparison.</li>

<li><strong>Actual dates get back-filled.</strong> A date typed in month 21 describing month 19 is not contemporaneous evidence in the same sense as a date typed in month 19. It might still be right. It is a different quality of record, and the difference is exactly what gets probed in cross-examination.</li>
</ul>



<p class="wp-block-paragraph">AACE International&#8217;s Recommended Practice 29R-03 places the as-planned versus as-built family in its observational, static group: you compare a planned network against an as-built without inserting or removing delay events. The SCL Delay and Disruption Protocol, second edition, is blunter about what makes any of it work, pushing hard on agreeing a record-keeping regime up front and never overwriting a programme version. Both point at the same requirement. You need the whole sequence of updates, intact, and you need to know which one told you what.</p>



<p class="wp-block-paragraph">The platform&#8217;s whole job is to make that sequence queryable.</p>



<h2 class="wp-block-heading">A worked example: what thirty-four updates actually look like</h2>



<p class="wp-block-paragraph">Take a job shaped like this. Three years of monthly updates, call it thirty-four XER exports. Around four thousand activities in the current programme. Three baseline revisions, two agreed and one submitted but never accepted. Somewhere near month fourteen the contractor moved most trades from a five-day to a six-day calendar to recover. The numbers are illustrative rather than from any particular project, but the shape is ordinary.</p>



<p class="wp-block-paragraph">Run the naive comparison and you get a variance table. Run the sequence properly and four separate questions fall out, none of which the variance table can answer:</p>



<ol class="wp-block-list">
<li>Which activities existed in the baseline, disappeared mid-job, and came back under a different activity ID?</li>

<li>For each actual date in the final update, in which update did that value first appear, and how far behind the event was it?</li>

<li>When float moved, was it because work moved, or because someone changed a calendar or a duration?</li>

<li>What contemporaneous record sits underneath each actual date, and does it agree?</li>
</ol>



<p class="wp-block-paragraph">Those four questions are the platform. Everything below is how to answer each one without hand-checking four thousand rows thirty-four times.</p>



<h2 class="wp-block-heading">Failure family one: activity identity across versions</h2>



<p class="wp-block-paragraph">An XER file is tab-delimited text. It opens in a text editor. The structure is a header line beginning <code>ERMHDR</code>, then a run of tables, each introduced by <code>%T</code> with the table name, a <code>%F</code> line carrying the field names, and <code>%R</code> lines carrying rows, closed by <code>%E</code>. The tables you care about first are <code>PROJECT</code>, <code>TASK</code>, <code>TASKPRED</code> and <code>CALENDAR</code>.</p>



<pre class="wp-block-code"><code>ERMHDR	19.12	2026-01-15	Project	admin	Admin User	dbxDatabaseNoName	Project Management	USD
%T	TASK
%F	task_id	proj_id	wbs_id	clndr_id	task_code	task_name	status_code	act_start_date	act_end_date	target_start_date	target_end_date	total_float_hr_cnt
%R	1001	100	200	1	C-2140	Structural Steel Erection L4	TK_Complete	...
%E</code></pre>



<p class="wp-block-paragraph">Here is the trap. <code>task_id</code> is the primary key of the P6 database the file was exported from. It is not a stable identifier for an activity across time. If the project was ever copied between databases, restored, or exported from a different P6 instance, the same physical activity can carry a different <code>task_id</code>. Worse, two XERs from different sources can reuse the same <code>task_id</code> for completely unrelated activities.</p>



<p class="wp-block-paragraph"><code>task_code</code> is the Activity ID a human sees, and it is stable right up until someone renumbers. Neither field alone gives you identity. So do not pick one. Build an identity resolution layer and keep the evidence for every decision it makes:</p>



<ul class="wp-block-list">
<li>Use <code>(proj_id, task_code)</code> as the working join key, scoped to a single source database.</li>

<li>Store <code>task_id</code>, <code>task_name</code> and <code>wbs_id</code> on every snapshot row so a rename or a WBS move is detectable after the fact.</li>

<li>Emit an explicit <em>unresolved</em> record when an activity vanishes or appears mid-sequence. Do not guess.</li>
</ul>



<p class="wp-block-paragraph">That last point is the one people get wrong. When C-2140 disappears in update 22 and an activity with the same name but the ID C-2140A appears in update 23, there are three candidate explanations: deletion and replacement, renumbering, or a split. The platform cannot know which. A tool that silently picks one is worse than a tool that flags it, because the analyst never learns there was a question. Surface it, hand it to a human, record the decision and the reason.</p>



<h2 class="wp-block-heading">Failure family two: date provenance, and the table that makes the whole thing worth building</h2>



<p class="wp-block-paragraph">If you build one thing, build this. For every activity and every date field, record the first update in which that value appeared, and every subsequent change to it.</p>



<p class="wp-block-paragraph">Assume a snapshot table with one row per activity per update, appended and never modified. A window function does the work:</p>



<pre class="wp-block-code"><code>-- First appearance of each distinct actual finish value,
-- and how far the reported date lags the update that reported it.
WITH changes AS (
  SELECT
    task_code,
    version_no,
    data_date,
    act_end_date,
    LAG(act_end_date) OVER (
      PARTITION BY task_code ORDER BY version_no
    ) AS prev_act_end
  FROM activity_snapshot
)
SELECT
  task_code,
  version_no          AS first_reported_in_update,
  act_end_date        AS reported_actual_finish,
  data_date           AS update_data_date,
  data_date::date - act_end_date::date AS reporting_lag_days
FROM changes
WHERE act_end_date IS NOT NULL
  AND (prev_act_end IS NULL OR prev_act_end &lt;&gt; act_end_date)
ORDER BY reporting_lag_days DESC;</code></pre>



<p class="wp-block-paragraph">Three things arrive at once. You can answer the question that opened this post, for any activity, in a second: C-2140&#8217;s actual finish first appeared in update 21, carrying a date twenty-three days behind that update&#8217;s data date.</p>



<p class="wp-block-paragraph">The distribution of that lag column is then a finding in its own right. If most activities report within days and one subcontractor&#8217;s work consistently reports a month late, that is not a rounding artefact, it is a records issue you can evidence. And any actual date that changes after it was first stated deserves a hard look. Actuals are supposed to lock. A value that moves has either been corrected, which should have a paper trail, or overwritten, which should not have happened.</p>



<p class="wp-block-paragraph">P6 ships a Schedule Comparison tool, the one that used to be called Claim Digger and now lives inside Visualizer. It does a field-by-field diff and dumps a large HTML table, which is genuinely useful for checking one submitted update against the last. That is what it was built for. It was not built to reason across thirty-four files at once, and pushing its output into Excel does not change that. Deltek&#8217;s Acumen Fuse and Steelray&#8217;s Delay Analyzer go further, with what the vendors describe as half-step analysis, separating progress from scope revisions across successive versions. If your instruction allows the licence and the timetable allows the learning curve, look at them seriously before writing any code. Neither is priced publicly, both target enterprise project controls teams, and neither does the records-linking work in the last section for you.</p>



<h2 class="wp-block-heading">Failure family three: calendars, and float that quietly means nothing</h2>



<p class="wp-block-paragraph">Durations and float in an XER are stored in hours. <code>total_float_hr_cnt</code> holds hours, not days. A value of 40 is five days only if the activity runs an eight-hour calendar. Divide everything by eight and a job with mixed shift patterns produces float numbers that are confidently wrong.</p>



<p class="wp-block-paragraph">The calendar itself lives in the <code>CALENDAR</code> table, in a <code>clndr_data</code> field holding a proprietary parenthesised blob describing work weeks, exceptions and holidays. It is the hardest part of the format to parse correctly, and getting it wrong produces durations and dates that look plausible and are not.</p>



<p class="wp-block-paragraph">Two rules follow, and both are about the read path, not the maths.</p>



<ul class="wp-block-list">
<li>Store float and durations in hours exactly as exported. Convert at presentation time, using that activity&#8217;s own calendar, and print the calendar name next to the number. If you cannot resolve the calendar, print hours and say so.</li>

<li>Track calendar assignment as a versioned attribute, the same as any date. A change from a five-day to a six-day calendar changes every float figure downstream of it, with no change to logic and no change to progress. An activity can go from critical to non-critical because of an administrative edit, and a variance table will attribute that shift to the works.</li>
</ul>



<p class="wp-block-paragraph">This is also why an as-built critical path is not simply the chain with zero float in the last update. Retrospective longest path and as-planned versus as-built windows analysis exist as separate methods precisely because the float figures in a progressed programme are a function of how the programme was maintained.</p>



<h2 class="wp-block-heading">Failure family four: separating movement from editing</h2>



<p class="wp-block-paragraph">Every delta between two consecutive updates falls into one of a small number of categories, and collapsing them into a single &#8220;variance&#8221; number is where analyses lose their defensibility. Classify each change explicitly:</p>



<ul class="wp-block-list">
<li><strong>Progress:</strong> an actual start or finish appeared, or remaining duration reduced consistent with work done.</li>

<li><strong>Duration change:</strong> original or remaining duration edited without corresponding progress.</li>

<li><strong>Logic change:</strong> a relationship added, removed, retyped, or its lag altered.</li>

<li><strong>Calendar change:</strong> activity reassigned, or the calendar definition itself edited.</li>

<li><strong>Constraint change:</strong> a date constraint applied, moved or removed.</li>

<li><strong>Scope change:</strong> activity added or deleted.</li>
</ul>



<p class="wp-block-paragraph">Six buckets, one row per change, with the before value, the after value and the version pair. Once that table exists, a lot of the argument stops being an argument. &#8220;The completion date moved out eighteen days in window 12&#8221; becomes &#8220;of which fourteen days sit with progress on the critical chain and four with two logic edits applied in the same submission.&#8221; The second version is the one that gets tested and holds.</p>



<p class="wp-block-paragraph">Logic changes deserve their own view. Pull <code>TASKPRED</code> for every version, key each relationship on the predecessor and successor activity codes plus the relationship type, and diff the sets. Relationships that appear late in a job, on activities that are already in progress, are worth reading one by one.</p>



<h2 class="wp-block-heading">Failure family five: joining the schedule to the records</h2>



<p class="wp-block-paragraph">The schedule tells you when. It never tells you why. Cause comes from daily reports, site instructions, RFIs, correspondence, minutes, inspection records, weather data, and delivery notes. That material arrives as scanned PDFs and email exports, and it does not carry activity IDs. Nobody has ever written &#8220;C-2140&#8221; on a site diary.</p>



<p class="wp-block-paragraph">So the join key is not the activity ID. It is a fuzzy composite of date, location or area, and trade or discipline. The practical approach:</p>



<ol class="wp-block-list">
<li>Extract structured fields from each record: document date, report date, area, trade, headcount if present, and the narrative text. Text extraction plus a layout-aware OCR pass handles most of it. Handwriting on older site diaries will not fully automate, and you should budget for that.</li>

<li>Normalise area and trade against the schedule&#8217;s own coding. Activity codes and WBS paths are usually the best available crosswalk, since they encode area and discipline already.</li>

<li>Generate <em>candidate</em> links: for each activity&#8217;s as-built window, every record whose date falls inside it and whose area or trade matches.</li>

<li>Have a human confirm, reject or annotate each candidate. Store who decided and when.</li>
</ol>



<p class="wp-block-paragraph">Step four is not optional and it is not a nicety. A machine-proposed link is a search result. Causation is an opinion, and the expert has to own it. Build the tool so it accelerates the search and refuses to state the conclusion, and you get the productivity without handing opposing counsel an argument about black boxes.</p>



<h2 class="wp-block-heading">The data model, in about six tables</h2>



<p class="wp-block-paragraph">The grain that makes everything else easy: one immutable row per activity per schedule version. Nothing is ever updated in place.</p>



<pre class="wp-block-code"><code>CREATE TABLE schedule_version (
  version_id     bigserial PRIMARY KEY,
  version_no     integer     NOT NULL,   -- ordinal in the sequence
  source_file    text        NOT NULL,
  file_sha256    char(64)    NOT NULL,   -- proves the file was not altered
  p6_export_ts   timestamptz,            -- from the ERMHDR line
  data_date      timestamptz NOT NULL,
  ingested_at    timestamptz NOT NULL DEFAULT now(),
  notes          text
);

CREATE TABLE activity_snapshot (
  version_id           bigint  NOT NULL REFERENCES schedule_version,
  task_code            text    NOT NULL,
  task_id              bigint  NOT NULL,
  task_name            text,
  wbs_path             text,
  clndr_id             bigint,
  status_code          text,
  target_start_date    timestamptz,
  target_end_date      timestamptz,
  act_start_date       timestamptz,
  act_end_date         timestamptz,
  total_float_hr_cnt   numeric,
  PRIMARY KEY (version_id, task_code)
);</code></pre>



<p class="wp-block-paragraph">Add <code>relationship_snapshot</code> keyed on <code>(version_id, pred_task_code, succ_task_code, rel_type)</code>, a <code>calendar_snapshot</code> holding the raw <code>clndr_data</code> alongside your parsed working pattern, a <code>record_document</code> table for the contemporaneous material, and a <code>record_link</code> table carrying the human decision on each candidate link.</p>



<p class="wp-block-paragraph">Thirty-four updates at four thousand activities is under a hundred and fifty thousand rows. This is not a big-data problem and it should not be built like one. PostgreSQL on a single machine, with the parsing done in Python, will run every query in this post fast enough that you stop thinking about it.</p>



<p class="wp-block-paragraph">On the parsing side, an XER is simple enough to read with the standard library, and there are open-source parsers such as PyP6Xer if you would rather not write the tokeniser yourself. Whichever route you take, keep the raw file. Hash it on ingest, store the hash in <code>schedule_version</code>, and never write to the original. Chain of custody over the source files is the cheapest credibility you will ever buy.</p>



<h2 class="wp-block-heading">Where to run it, and the confidentiality problem nobody budgets for</h2>



<p class="wp-block-paragraph">A modest VPS is plenty. Contabo and InterServer both sell machines with more than enough memory and disk for a job this size, and the specification matters far less than the question you should ask first: does your confidentiality undertaking actually permit the disclosure material to sit on that machine, in that jurisdiction? Often the answer is no, and the analysis has to run on a workstation inside the client&#8217;s own environment. Find that out before you build anything.</p>



<p class="wp-block-paragraph">Three practical points that follow from the material being privileged rather than ordinary project data:</p>



<ul class="wp-block-list">
<li><strong>Full-disk encryption on anything that holds the files</strong>, including the laptop you take to the hearing. This is table stakes and it is still routinely skipped.</li>

<li><strong>A VPN for untrusted networks.</strong> You will open the data room from a hotel or an airport at some point. NordVPN and Surfshark both handle that case. It protects the transport; it does nothing about where the data comes to rest, so do not let it substitute for the jurisdiction question above.</li>

<li><strong>A disposal plan.</strong> Undertakings usually require destruction or return at the close of the engagement, and deleting a file does not destroy it. Overwrite-based tools such as O&amp;O SafeErase cover the workstation case. If the volume was encrypted from the start, destroying the key is faster and cleaner than overwriting the data, and easier to certify.</li>
</ul>



<p class="wp-block-paragraph">If cloud storage is permitted, write-once object storage with a retention lock suits the ingested originals well. It gives you an immutability story that does not rest on your own discipline.</p>



<h2 class="wp-block-heading">Troubleshooting the ingest</h2>



<ul class="wp-block-list">
<li><strong>The parser dies on a decode error.</strong> XER files are not reliably UTF-8, particularly when they came from a database with a non-English locale. Try a single-byte fallback encoding before assuming the file is corrupt, and log which encoding succeeded against the version record.</li>

<li><strong>Row and field counts do not match.</strong> Activity names can contain characters that confuse a naive split, and some exports wrap fields inconsistently. Parse defensively on field count per <code>%F</code> line rather than assuming.</li>

<li><strong>Duplicate <code>task_code</code> in one file.</strong> One XER can carry several projects plus the shared global data they depend on. Always scope by <code>proj_id</code> and confirm you have the right project before anything else.</li>

<li><strong>Activity counts drop between updates for no reason.</strong> Check whether the export was filtered or taken at a WBS level rather than the full project. A partial export looks exactly like a deletion event in your diff.</li>

<li><strong>Dates shift by a few hours.</strong> P6 stores times, not just dates, and calendar start hours vary. Normalise to the activity calendar&#8217;s working day before comparing, or every finish looks a shift early.</li>

<li><strong>Float figures look wrong across the board.</strong> Check the hours-versus-days conversion and the calendar assignment before checking anything else. It is almost always one of those two.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating the final update as the as-built without stating that assumption anywhere in the report.</li>

<li>Joining on <code>task_id</code> because it is the primary key, and inheriting whatever database history came with the file.</li>

<li>Letting the platform resolve an ambiguous activity match silently, so the analyst never sees the ambiguity existed.</li>

<li>Converting float to days with a fixed divisor, then reporting a single variance figure that mixes progress with programme edits.</li>

<li>Automating the record-to-activity link all the way to a stated cause, which turns an evidence tool into a target.</li>

<li>Modifying source XERs in place, then having no way to demonstrate they are as received.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>Hash every source file on ingest and record the hash next to the version. It takes one line and it answers a question you will otherwise answer badly.</li>

<li>Make every table append-only. Corrections become new rows with a reason, never overwrites.</li>

<li>Build the first-appearance view early. It reframes the analysis from &#8220;what do the files say&#8221; to &#8220;when did the files start saying it&#8221;, which is the more defensible question.</li>

<li>Make every exhibit reproducible from one query, so a challenged figure traces back to a file and a row rather than a spreadsheet cell.</li>

<li>Keep a written record of every judgement call the platform surfaced and a human resolved. That log is often more useful in the hearing than the analysis itself.</li>

<li>Validate against the commercial tools where you can. If Acumen Fuse or Schedule Comparison disagrees with your diff on a given version pair, one of you is wrong and you want to know which before the report goes out.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is as-planned vs as-built analysis still accepted in construction disputes?</h3>



<p class="wp-block-paragraph">Yes, as one of a recognised family of methods. AACE Recommended Practice 29R-03 includes it in its taxonomy as an observational method, and the SCL Protocol&#8217;s second edition deliberately stopped naming a single preferred technique for retrospective analysis, setting out selection factors instead. Acceptance in a given dispute turns on the records available, the contract and the forum, not on the method&#8217;s reputation in the abstract.</p>



<h3 class="wp-block-heading">How many schedule updates do I need before building a platform is worth it?</h3>



<p class="wp-block-paragraph">Roughly: below about eight updates, a careful analyst with a workbook is faster. Above twenty, manual comparison stops being reliable, not because people are careless but because the number of pairwise comparisons grows past what anyone can hold. The crossover in practice sits somewhere in the low teens, and it moves earlier if there are multiple baselines or a renumbering event.</p>



<h3 class="wp-block-heading">Can I do as-planned vs as-built analysis without a Primavera P6 licence?</h3>



<p class="wp-block-paragraph">For the data work, yes. An XER is plain text and parses without P6 installed, and browser-based and open-source viewers exist. You will still want licensed access somewhere in the engagement for anything that requires recalculating the network, because reproducing P6&#8217;s scheduling engine faithfully is not a project you want inside a disputes timetable.</p>



<h3 class="wp-block-heading">What is the difference between as-planned vs as-built and a windows analysis?</h3>



<p class="wp-block-paragraph">The simple form of as-planned versus as-built compares planned dates against actual dates observationally, without necessarily using network logic. A windows analysis divides the project into periods, usually monthly, and examines the contemporaneous critical path and the critical delay in each period before investigating the cause. Windows demands more of the records and generally carries more weight where the programme was maintained properly. This platform supports both, because both need the same version history underneath.</p>



<h3 class="wp-block-heading">What do I do when the contractor&#8217;s updates were not maintained properly?</h3>



<p class="wp-block-paragraph">State it, quantify it, and let it drive method selection. That is exactly what the first-appearance table is for: reporting lag, actuals that moved after being stated, out-of-sequence progress and unexplained logic edits are all measurable. Recommended practice treats recreating a schedule from records as a last resort, used when contemporaneous data is unavailable, and recreated schedules carry less evidential weight. Showing the deficiency with numbers is stronger than asserting it.</p>



<h3 class="wp-block-heading">Should I build this or buy commercial delay analysis software?</h3>



<p class="wp-block-paragraph">Buy for schedule diagnostics and version comparison. That problem is well solved and the established tools have years of edge cases behind them. Build for the parts specific to your instruction: the record linking, the exhibit generation in your house format, and the audit trail. The hybrid is usually right, and the deciding factor is more often the licence and the timescale than the technical fit.</p>



<h3 class="wp-block-heading">Does the platform need to reproduce P6&#8217;s critical path calculation?</h3>



<p class="wp-block-paragraph">No, and it should not try. Read the float and date values P6 already calculated and stored in each export, and treat them as evidence of what the programme said at that moment. Anywhere you genuinely need a recalculated network, do it in the scheduling tool so the result is reproducible by anyone with the same file.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">A credible as-planned vs as-built analysis is not a better comparison between two files. It is a complete, immutable, queryable history of what every schedule update said and when it started saying it. Build the version history and the first-appearance table first, and the exhibits fall out of it. Build the exhibits first and you will spend the hearing defending a spreadsheet whose provenance you cannot reconstruct.</p>



<p class="wp-block-paragraph">The scheduling judgement stays with the expert. The platform&#8217;s job is narrower and duller: make sure that when someone asks where a date came from, the answer takes one query and not one week.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need the data side of this built?</h2>



<p class="wp-block-paragraph">I build the engineering layer underneath delay and quantum work, so the expert spends their time on opinion rather than on data wrangling. That usually looks like:</p>



<ul class="wp-block-list">
<li>Ingesting a full sequence of P6 XER or XML updates into a versioned, append-only PostgreSQL model with hashing and chain of custody on every source file.</li>

<li>Building the first-appearance and reporting-lag views that show when each actual date entered the record, and flagging actuals that moved after being stated.</li>

<li>Classifying every between-update change into progress, duration, logic, calendar, constraint or scope, with before and after values on each row.</li>

<li>Extracting structured fields from daily reports, RFIs and correspondence with OCR and document processing, then generating candidate activity links for an analyst to confirm or reject.</li>

<li>Producing exhibit-grade output in Power BI or as static deliverables, where every figure traces back to a query, a file and a row.</li>

<li>Standing it up on infrastructure that fits the confidentiality regime you work under, whether that is a locked-down VPS or a machine inside the client&#8217;s own environment.</li>
</ul>



<p class="wp-block-paragraph">If you have a stack of XERs and a deadline, send me two consecutive updates and whatever the record-keeping looked like, and I will tell you what the sequence can and cannot support.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/as-planned-vs-as-built-analysis-platform/">As-Planned vs As-Built Analysis: Building a Platform That Survives Cross-Examination</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Extracting Clauses, Parties, Dates and Obligations with Amazon Bedrock</title>
		<link>https://john-nessime.com/blog/technical-guides/extract-contract-data-amazon-bedrock/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 18 Aug 2026 11:17:11 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Bedrock Data Automation]]></category>
		<category><![CDATA[Confidence Scoring]]></category>
		<category><![CDATA[Contract Analysis]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Human In The Loop]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Legal Tech]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Step Functions]]></category>
		<category><![CDATA[Structured Outputs]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=392</guid>

					<description><![CDATA[<p>Valid JSON is not correct data. A practical guide to extracting clauses, parties, dates and obligations from contracts with Amazon Bedrock: the two build paths, schema design, the three failure families that produce confident wrong answers, and the validation layer that catches them.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/extract-contract-data-amazon-bedrock/">Extracting Clauses, Parties, Dates and Obligations with Amazon Bedrock</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The extraction job finishes clean. Every contract in the batch produces valid JSON, every field is populated, the schema validator passes, and the results load into the obligation tracker without a single error. Three weeks later someone in legal asks why the tracker says the renewal notice window is 30 days when the contract says 90. That is the moment you learn the real problem with contract data extraction: the failure mode is not malformed output. It is well-formed output that is wrong.</p>



<p class="wp-block-paragraph">This post is a practical guide to how I&#8217;d extract contract data with Amazon Bedrock: the clauses, parties, dates, and obligations that feed contract lifecycle tools, obligation trackers, and compliance dashboards. It covers the two build paths Bedrock gives you, how to design the extraction schema, the three failure families that produce confident wrong answers, and the validation layer that catches them before they reach a system someone trusts.</p>



<h2 class="wp-block-heading">Two ways to extract contract data with Amazon Bedrock</h2>



<p class="wp-block-paragraph">Bedrock offers two genuinely different routes to the same destination, and picking the wrong one costs you either flexibility or months of pipeline plumbing you didn&#8217;t need to build.</p>



<h3 class="wp-block-heading">Path one: call a model directly</h3>



<p class="wp-block-paragraph">You send the contract to a foundation model, typically Anthropic Claude, through the Converse or InvokeModel API, with a prompt and a schema describing the fields you want back. Converse accepts PDF documents as a content block, so contract bytes go straight from S3 without a separate OCR step. One detail that catches people: full visual understanding of a PDF through Converse with Claude requires citations to be enabled. Without them the API falls back to plain text extraction, and anything living in a table or a scanned signature page silently disappears from the model&#8217;s view.</p>



<p class="wp-block-paragraph">The piece that changed this path from fragile to dependable is structured outputs. Instead of asking the model nicely for JSON and writing defensive parsing code for the day it prepends an apology, you attach a JSON Schema to the request and Bedrock enforces it during token generation through constrained decoding. The response cannot fail to parse. One mechanic worth knowing: the first request with a new schema triggers a grammar compilation step, and compiled grammars are cached for a limited time, so schema churn has a latency cost.</p>



<h3 class="wp-block-heading">Path two: Bedrock Data Automation</h3>



<p class="wp-block-paragraph">Bedrock Data Automation (BDA) is the managed intelligent document processing service. You define a blueprint, which is a list of fields with types and natural language instructions for how to find and normalize each one, attach it to a project, and invoke the async processing API against files in S3. BDA returns two things a raw model call does not give you out of the box: a confidence score per field and visual grounding, meaning bounding boxes pointing back to where on the page each value came from. A project can hold multiple document blueprints, so a mixed intake of NDAs, master service agreements, and amendments routes each document to the right extraction logic automatically.</p>



<p class="wp-block-paragraph">My honest read on the choice: BDA wins when you want the boring parts managed and you need per-field confidence and grounding for a human review loop, which for contracts you almost always do. The direct model path wins when you need reasoning the blueprint format can&#8217;t express, like interpreting a clause against the definitions section. Plenty of real pipelines use both: BDA for the structured pass, a direct Claude call for the clauses that need interpretation. Amazon Textract still earns its place for high-volume standardized forms, but contracts are exactly the variable, long-tail documents layout-based extraction was never great at.</p>



<h2 class="wp-block-heading">Design the schema before you touch the API</h2>



<p class="wp-block-paragraph">The schema is not a formality. With structured outputs, the field names, descriptions, and ordering you write become context the model generates against, so the schema is effectively part of the prompt. A vague schema produces structurally valid garbage: JSON that parses perfectly and means nothing.</p>



<p class="wp-block-paragraph">Three design rules that pay for themselves. First, one concept per field: &#8220;renewal terms&#8221; as a single string is unusable downstream; split it into whether renewal exists, whether it is automatic, the duration, and the notice period to prevent it. Second, write field descriptions like instructions to a junior reviewer, including what the field is not. Third, make every field nullable and tell the model to return null when the contract is silent. A model forced to fill a required field will fill it with something.</p>



<p class="wp-block-paragraph">Here is the shape I&#8217;d start with for a general commercial agreement, trimmed to the core:</p>



<pre class="wp-block-code"><code>{
  "type": "object",
  "properties": {
    "parties": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "legal_name": { "type": ["string", "null"],
            "description": "Full registered entity name from the preamble or signature block, not the defined shorthand" },
          "defined_term": { "type": ["string", "null"],
            "description": "The shorthand assigned in the preamble, e.g. Supplier, Client" },
          "role": { "type": ["string", "null"] }
        }
      }
    },
    "effective_date": { "type": ["string", "null"],
      "description": "Date the agreement takes effect. Null if only a signature date exists." },
    "governing_law": { "type": ["string", "null"] },
    "obligations": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "obligor": { "type": ["string", "null"] },
          "description": { "type": ["string", "null"] },
          "deadline_text": { "type": ["string", "null"],
            "description": "The deadline exactly as written, e.g. within 30 days of the Effective Date" },
          "source_clause": { "type": ["string", "null"],
            "description": "Section number where this obligation appears" }
        }
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Notice <code>deadline_text</code> keeps the deadline as written instead of asking the model to compute a calendar date. That is deliberate, and the reason is the second failure family below.</p>



<h2 class="wp-block-heading">Failure family one: parties that aren&#8217;t who they say they are</h2>



<p class="wp-block-paragraph">Contracts name their parties once, in the preamble, then never again. From that point on everything refers to &#8220;the Supplier&#8221; or &#8220;the Receiving Party.&#8221; A naive extraction returns the defined term as the party name, which is useless for matching against a vendor master or a CRM. Worse, the signature block sometimes names a different legal entity than the preamble, usually an affiliate signing on behalf of a subsidiary, and the model will happily pick whichever it saw last.</p>



<p class="wp-block-paragraph">The fix is structural, not clever prompting. Extract the legal name and the defined term as separate fields, as in the schema above, with instructions to take legal names from the preamble and cross-check the signature block. Then validate outside the model: if the two entities differ, flag the document for review rather than picking one. That mismatch is occasionally a genuine drafting error in the contract itself, which makes it exactly the kind of thing you want surfaced, not smoothed over.</p>



<h2 class="wp-block-heading">Failure family two: dates that mean different things</h2>



<p class="wp-block-paragraph">A contract carries several dates that look interchangeable and are not: execution, effective, commencement, expiry, and the deadlines derived from notice periods. The extraction failure that costs money is almost never a misread date string. It is a correctly read date assigned to the wrong concept, or a relative deadline flattened into a wrong absolute one.</p>



<p class="wp-block-paragraph">Relative dates are the invisible one. &#8220;Either party may terminate by written notice no later than ninety days before the end of the then-current term&#8221; contains no date at all. If your schema demands a date type, the model computes one, and its arithmetic rests on assumptions about the term start that may be wrong, especially when an amendment has reset the term. That computed date looks exactly as trustworthy in your tracker as a real one. This is why I extract deadline language verbatim and compute calendar dates in code, where the arithmetic is testable and the inputs are auditable.</p>



<p class="wp-block-paragraph">Cheap validation catches most of the rest. The effective date must not fall after the expiry. A notice period must be a plausible number of days. If the agreement auto-renews, a renewal duration should exist. Fields that fail these checks get flagged. None of this needs a model; it is twenty lines of code standing between extraction and the systems people trust.</p>



<h2 class="wp-block-heading">Failure family three: obligations, cross-references, and the amendment problem</h2>



<p class="wp-block-paragraph">Obligations are the hardest target because they are relational. &#8220;The Supplier shall deliver the reports described in Schedule 3 within the period set out in Section 4.2, except as provided in Section 9&#8221; is one obligation spread across three locations. Chunk the contract carelessly and the model sees the reference with no Section 4.2 in sight. It will either drop the deadline or invent a plausible one, and the second outcome is far worse because nothing downstream looks broken.</p>



<p class="wp-block-paragraph">Contracts resist chunking more than almost any other document type, because defined terms and cross-references make every section depend on distant ones. The practical answers, in order of preference:</p>



<ol class="wp-block-list">
<li>Don&#8217;t chunk. Modern model context windows fit most commercial agreements whole, and whole-document extraction is the single biggest accuracy win available. Reach for chunking only when a document genuinely doesn&#8217;t fit.</li>



<li>If you must split, prepend the definitions section and the table of contents to every chunk, and split on section boundaries rather than fixed token counts so no clause straddles a cut.</li>



<li>Capture the source clause reference for every obligation, as in the schema above, so a reviewer can jump straight to the language instead of re-reading the contract.</li>
</ol>



<p class="wp-block-paragraph">Then there are amendments. An amendment is a short document that quietly rewrites the base agreement, and if you extract each file independently your dataset will assert the original terms with full confidence long after they stopped being true. Treat the contract family, not the file, as the unit of extraction: process the base agreement, process each amendment, and resolve them in order so later documents override earlier ones. BDA&#8217;s multi-blueprint routing helps here, since amendments have a recognizably different shape and can carry their own blueprint focused on what changed.</p>



<h2 class="wp-block-heading">The validation layer that makes it trustworthy</h2>



<p class="wp-block-paragraph">Schema compliance tells you the output is well-formed. It says nothing about whether it is true. The trust comes from three checks layered after extraction.</p>



<ul class="wp-block-list">
<li><strong>Cross-field consistency.</strong> Dates in order, notice periods plausible, every obligation carrying an obligor that matches an extracted party. Pure code, no model.</li>



<li><strong>Grounding.</strong> Every extracted value should point back to a location in the document. BDA gives you this natively through bounding boxes; on the direct model path, requiring a source clause reference per field is the lightweight version. A value that cannot be traced to the page is a value you cannot audit.</li>



<li><strong>Confidence-based routing.</strong> Use BDA&#8217;s per-field confidence scores to route low-confidence extractions to human review and let high-confidence ones flow through. The goal is not eliminating review; it is spending reviewer minutes only where the model is unsure. Party names and governing law typically extract very reliably, while obligations from unusual templates are where automation honestly ends and review begins.</li>
</ul>



<p class="wp-block-paragraph">Operationally this looks like S3 event notifications triggering a Step Functions workflow: extract, validate, then branch to auto-accept or a review queue. Instrument it from day one. Track the auto-accept rate, per-field confidence distributions, and validation failure counts; a slow drift in confidence is usually the first sign a new contract template has entered the intake. I ship these to Grafana Cloud with the rest of the pipeline metrics, and since direct model calls bill by tokens while BDA bills per unit of content processed, a cost dashboard in something like Vantage or CloudZero shows quickly whether whole-document extraction is worth its token count for your mix. It usually is.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<p class="wp-block-paragraph"><strong>Fields come back null for content that is clearly in the document.</strong> On the Converse path with PDFs, check whether you are getting text-only extraction; content in tables, stamps, or scanned pages needs the visual analysis mode. For scans, verify image quality before blaming the model.</p>



<p class="wp-block-paragraph"><strong>The same contract extracts differently on reruns.</strong> Set temperature to zero for extraction work, then look for instructions the model can resolve more than one way. Ambiguity in a schema description is the usual culprit.</p>



<p class="wp-block-paragraph"><strong>First requests with a new schema are slow.</strong> That is grammar compilation for structured outputs. Stabilize your schemas and warm them before batch runs rather than generating schemas dynamically per request.</p>



<p class="wp-block-paragraph"><strong>BDA accuracy is mediocre on your templates.</strong> Blueprint instructions are tunable, and BDA can refine them automatically from a small set of example documents with known correct values. Feed it your hardest real examples, not your cleanest ones.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating schema-valid JSON as verified data. Structure and truth are separate properties, and the second one needs its own checks.</li>



<li>Extracting the defined term as the party name and matching it against vendor records.</li>



<li>Letting the model compute calendar dates from relative deadline language instead of extracting the language and computing in code.</li>



<li>Chunking by token count through a document whose clauses reference each other across fifty pages.</li>



<li>Processing amendments as standalone documents and silently keeping stale base-agreement terms.</li>



<li>Making every schema field required, which converts &#8220;the contract is silent&#8221; into a fabricated answer.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Extract whole documents whenever the context window allows. Every chunking strategy is a workaround, not a feature.</li>



<li>Keep a small labeled evaluation set of your own contracts and score every prompt or blueprint change against it before shipping.</li>



<li>Require a source reference for every extracted value, whether that is a BDA bounding box or a clause number from the model.</li>



<li>Route on confidence and validation results, and measure your auto-accept rate as a first-class metric.</li>



<li>Version your schemas and store the schema version alongside every extraction, so you can tell which records predate a fix.</li>



<li>Keep humans in the loop for high-stakes fields. The realistic goal is automating the bulk and focusing legal review, not replacing it.</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Should I use Bedrock Data Automation or call Claude directly for contract extraction?</h3>



<p class="wp-block-paragraph">BDA if you want managed parsing, per-field confidence scores, and visual grounding for a review workflow. Direct model calls if you need interpretive reasoning over clauses or conversational access to the document. Many production pipelines combine both.</p>



<h3 class="wp-block-heading">Do I still need Amazon Textract?</h3>



<p class="wp-block-paragraph">Usually not as the primary extractor for contracts, since layout-based extraction struggles with variable legal drafting. It remains a good fit for high-volume standardized forms, and some pipelines route documents between Textract and generative extraction by type.</p>



<h3 class="wp-block-heading">How accurate is contract data extraction in practice?</h3>



<p class="wp-block-paragraph">It varies sharply by field type. Parties, governing law, and explicit dates extract reliably. Obligations, renewal mechanics, and anything expressed through cross-references are meaningfully harder, which is why confidence routing and human review exist. Distrust any vendor quoting one accuracy number for &#8220;contracts.&#8221;</p>



<h3 class="wp-block-heading">How do I extract obligations specifically, not just metadata?</h3>



<p class="wp-block-paragraph">Model each obligation as its own object with obligor, description, deadline language, and source clause. Extract against the whole document so cross-references resolve, and validate that every obligor matches an extracted party.</p>



<h3 class="wp-block-heading">Can Bedrock handle scanned contracts?</h3>



<p class="wp-block-paragraph">Yes. BDA processes scanned documents as part of its parsing stage, and Claude&#8217;s visual document understanding reads scanned pages when invoked with visual analysis. Scan quality still matters; poor scans degrade every approach.</p>



<h3 class="wp-block-heading">Is it safe to put contracts through Bedrock?</h3>



<p class="wp-block-paragraph">Bedrock runs within your AWS account boundary, and AWS states that customer content is not used to train the underlying models. For sensitive agreements the usual controls apply: VPC endpoints for private connectivity, KMS encryption on the buckets, and IAM scoped to the pipeline roles. Confirm the specifics against your own compliance requirements.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">If you remember one thing, make it this: when you extract contract data with Amazon Bedrock, valid output is the starting line, not the finish. Structured outputs and BDA blueprints have genuinely solved the formatting problem, which means the remaining risk is concentrated in meaning: the wrong entity, the wrong date concept, the obligation whose deadline lives two sections away. Design the schema around those failures, validate outside the model, ground every value to the page, and route on confidence. That is the difference between a demo and a pipeline a legal team will actually trust.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help building a contract extraction pipeline on AWS?</h2>



<p class="wp-block-paragraph">I design and build document intelligence pipelines on AWS as a freelance DevOps engineer. If you&#8217;re extracting data from contracts or other legal documents, here&#8217;s what I can help with:</p>



<ul class="wp-block-list">
<li>Designing extraction schemas and BDA blueprints for your actual contract types, including amendments and multi-document families</li>



<li>Building the full serverless pipeline: S3 intake, Step Functions orchestration, extraction, validation, and review routing</li>



<li>Adding the validation and grounding layer that catches confident wrong answers before they reach your CLM or tracker</li>



<li>Setting up evaluation sets and accuracy measurement so prompt and blueprint changes are tested, not guessed</li>



<li>Locking down the security side: VPC endpoints, KMS encryption, and IAM scoping for sensitive documents</li>



<li>Cost tuning across model choice, whole-document versus chunked extraction, and BDA versus direct model calls</li>
</ul>



<p class="wp-block-paragraph">Send me a sample contract (redacted is fine), your target field list, or the output of a pipeline that isn&#8217;t behaving, and I&#8217;ll tell you what I&#8217;d do with it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/extract-contract-data-amazon-bedrock/">Extracting Clauses, Parties, Dates and Obligations with Amazon Bedrock</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
