<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>REST API | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/rest-api/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/rest-api/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Thu, 29 Oct 2026 06:58:01 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>REST API | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/rest-api/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Your CRM Is Not a Database: Where GoHighLevel Should Stop</title>
		<link>https://john-nessime.com/blog/data-engineering/gohighlevel-as-a-database/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 26 Sep 2026 06:00:23 +0000</pubDate>
				<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Enterprise Integration]]></category>
		<category><![CDATA[SaaS Engineering]]></category>
		<category><![CDATA[Architecture Debt]]></category>
		<category><![CDATA[CRM Audit]]></category>
		<category><![CDATA[Custom Objects]]></category>
		<category><![CDATA[Data Modeling]]></category>
		<category><![CDATA[GoHighLevel]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Marketing Automation]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Reverse ETL]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Snapshots]]></category>
		<category><![CDATA[Sub-Account Management]]></category>
		<category><![CDATA[System of Record]]></category>
		<category><![CDATA[Vendor Lock-In]]></category>
		<category><![CDATA[Webhooks]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=649</guid>

					<description><![CDATA[<p>A CRM never pushes back when you add one more field, which is exactly why the schema drifts until someone asks a question that needs a join. Four boundaries for deciding what belongs in GoHighLevel, what Custom Objects actually change, and the CRM-in-front, system-of-record-behind pattern I default to.</p>
<p>The post <a href="https://john-nessime.com/blog/data-engineering/gohighlevel-as-a-database/">Your CRM Is Not a Database: Where GoHighLevel Should Stop</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">&#8220;Can you pull every job we did last quarter, by technician, with the margin on each one?&#8221;</p>



<p class="wp-block-paragraph">The answer was no. Not &#8220;no, that&#8217;s expensive.&#8221; Just no. The jobs weren&#8217;t records. Each one was a couple of tags on a contact, three custom fields added at different times by different people, an opportunity someone had dragged into Won, and a note cut off mid-sentence. There was no job. There was a person that a job had happened to.</p>



<p class="wp-block-paragraph">Nobody made a bad decision to get there. &#8220;Can we just add a field for that?&#8221; is a five-second request and the answer is always yes, because adding a field is easy. That&#8217;s the trap. The CRM never pushes back, so the schema drifts for eighteen months and nobody notices until someone asks a question that needs a join.</p>



<p class="wp-block-paragraph">This post is about where that line sits: not whether to use the platform, but where <strong>GoHighLevel as a database</strong> stops being a shortcut and starts being architecture debt. I&#8217;ll cover the failure that stays invisible, the four boundaries that decide it, what Custom Objects actually change, and the arrangement I default to when the CRM isn&#8217;t the right home.</p>



<h2 class="wp-block-heading">The failure mode nobody sees coming</h2>



<p class="wp-block-paragraph">Databases push back. Declare a column as a date and it refuses to hold &#8220;next Tuesday.&#8221; Declare a foreign key and it refuses to orphan the row. Half the value of a schema isn&#8217;t storage, it&#8217;s the set of things it won&#8217;t let you do.</p>



<p class="wp-block-paragraph">A custom field holds whatever you type. That&#8217;s a feature when a rep jots down a competitor&#8217;s name. It&#8217;s a slow disaster when the field is called <code>contract_value</code> and it contains &#8220;1,250.00&#8221;, &#8220;$1250&#8221;, &#8220;1250 (pending)&#8221; and &#8220;ask Dave&#8221; across four hundred records.</p>



<p class="wp-block-paragraph">Here&#8217;s why it stays invisible: nothing breaks. Workflows keep firing. Dashboards keep rendering, because they count records rather than summing values. The failure surfaces the day someone needs an aggregate, and by then you have four hundred rows of ambiguity and no way to tell which reading was right. You can&#8217;t retroactively decide whether &#8220;1250 (pending)&#8221; was invoiced.</p>



<p class="wp-block-paragraph">History is the other half. A CRM field is a current value. Overwrite it and the old one is gone. Anything touching money, entitlements or consent eventually needs to prove what the state was and when, and the platform discarded that silently, months earlier.</p>



<h2 class="wp-block-heading">What the platform is genuinely good at</h2>



<p class="wp-block-paragraph">An argument against a tool isn&#8217;t credible unless it starts with the tool&#8217;s real case. GoHighLevel is very good at a specific set of things, most of them expensive to build yourself.</p>



<ul class="wp-block-list">
<li><strong>Conversation state across channels.</strong> One thread per person spanning SMS, email and calls, with delivery status attached. Rebuilding that is months of work and a compliance surface you don&#8217;t want.</li>

<li><strong>Sequenced outreach with human interrupts.</strong> Wait steps, conditions, and a reply pulling someone out of a sequence. Easy to describe, tedious to get right.</li>

<li><strong>Tenant isolation by default.</strong> Each sub-account is its own location, and nothing crosses unless you deliberately copy it.</li>

<li><strong>Non-engineers can change it.</strong> A workflow a client can edit on a Tuesday afternoon beats an elegant pipeline only you can touch.</li>
</ul>



<p class="wp-block-paragraph">Every one of those is about people and communication. That&#8217;s the pattern: when the thing you&#8217;re storing is a person, or something that happened to a person, the CRM is the right home. When it has a life independent of any person, you&#8217;re outside the design.</p>



<h2 class="wp-block-heading">Four boundaries that decide it for you</h2>



<p class="wp-block-paragraph">&#8220;Is this a database?&#8221; isn&#8217;t a useful question. These four are decidable, and any one of them crossing is usually enough.</p>



<h3 class="wp-block-heading">1. The question you need to ask requires a join</h3>



<p class="wp-block-paragraph">Not &#8220;show me contacts tagged X.&#8221; I mean revenue by technician by quarter, or every account with more than three open items where the last payment was over sixty days ago: grouping, filtering across related records, and arithmetic on typed values, all at once. Smart lists answer a narrow, predefined set of questions well, but they aren&#8217;t a query engine, and the gap between the two is where most people discover they&#8217;ve built a database by accident.</p>



<h3 class="wp-block-heading">2. More than one system writes the same field</h3>



<p class="wp-block-paragraph">This is the one that quietly costs the most. A stock level or invoice status lives in a CRM field, and both an automation and an external sync can write it. There&#8217;s no transaction and no compare-and-set. Last write wins, and &#8220;last&#8221; is whichever request happened to land second. You see it as an occasional wrong value you can&#8217;t reproduce, which is the worst kind of bug to be handed.</p>



<p class="wp-block-paragraph">The fix isn&#8217;t better retry logic. The field has the wrong owner. Exactly one system should write any given piece of state, and if that system isn&#8217;t the CRM, the CRM holds a copy for display rather than the value itself.</p>



<h3 class="wp-block-heading">3. Someone will need the history</h3>



<p class="wp-block-paragraph">Anything where you may later have to prove a prior state needs append-only storage. The workaround people reach for is a note per change. It doesn&#8217;t hold: notes aren&#8217;t structured, aren&#8217;t queryable, and the export path treats them as an afterthought, which leads straight to the next boundary.</p>



<h3 class="wp-block-heading">4. The exit shape doesn&#8217;t match what you put in</h3>



<p class="wp-block-paragraph">People check this last and should check it first. Contact export lands as CSV, and only the most recent note travels with it, truncated to 255 characters. A year of notes exports as one, clipped. Custom object data isn&#8217;t part of a direct export at all: the documented route is a snapshot from one sub-account to another, which is a copy inside the platform, not a file you keep. Export permissions are role-gated, so a client who believes they own their data may not be able to pull it themselves.</p>



<p class="wp-block-paragraph">None of that makes it a bad choice. It makes it a bad <em>sole</em> home for anything you&#8217;d be in trouble without.</p>



<h2 class="wp-block-heading">What Custom Objects change, and what they don&#8217;t</h2>



<p class="wp-block-paragraph">Custom Objects are a real improvement and worth using. Properties, policies, vehicles, courses: things that aren&#8217;t people, modelled as first-class records with their own fields and associations back to contacts, usable in workflows. For years the only option was squashing everything into a contact, and that was worse. They don&#8217;t turn the CRM into a database, though, and the reasons are specific rather than philosophical.</p>



<ul class="wp-block-list">
<li><strong>Ten objects per sub-account, documented as a hard cap.</strong> That&#8217;s your entire entity budget. Fifteen nouns worth modelling means five go back into custom fields.</li>

<li><strong>Uniqueness is limited and one-way.</strong> Up to ten unique fields per object, and only single line text, multi line text, number and phone types can be unique. It&#8217;s enforced across the sub-account and every entry point including the API, which is genuinely useful. But downgrade a unique field to non-unique and you cannot make it unique again.</li>

<li><strong>Write access is admin-only.</strong> Creating, updating and deleting object records is restricted to location admins; regular users are read-only. Check this before assuming the front desk will maintain records.</li>

<li><strong>Records count against storage quota.</strong> Object records and attached files consume the same account storage as everything else, so high-volume machine-generated data is the wrong shape.</li>

<li><strong>The exit is a snapshot, not an export.</strong> Objects, fields and associations move between sub-accounts via snapshots, subject to the same cap in the target account.</li>
</ul>



<p class="wp-block-paragraph">Read those as design constraints rather than complaints. They tell you the intended scale: dozens of entity types, no. Millions of rows, no. A handful of business nouns with clean identity and moderate volume, yes, and it works well there.</p>



<h2 class="wp-block-heading">The API budget is a design constraint too</h2>



<p class="wp-block-paragraph">If you&#8217;re treating the platform as a datastore behind an integration, the rate limits shape what&#8217;s possible. On the v2 API a marketplace app gets a burst allowance of 100 requests per 10 seconds per resource, and 200,000 requests per day per resource, where a resource means a location or company. Version 1 is end of support, so anything new belongs on v2.</p>



<p class="wp-block-paragraph">Every v2 request carries a version header alongside the bearer token. It&#8217;s easy to forget and produces a confusing failure when missing:</p>



<pre class="wp-block-code"><code>curl -H "Authorization: Bearer $TOKEN" 
     -H "Version: 2021-04-15" 
     -H "Accept: application/json" 
     "$BASE_URL/&lt;resource&gt;?locationId=$LOCATION_ID"</code></pre>



<p class="wp-block-paragraph">More useful than the numbers is that responses carry their own accounting, so you never have to guess where you are in the window: <code>X-RateLimit-Remaining</code> and <code>X-RateLimit-Max</code> cover the burst interval, <code>X-RateLimit-Daily-Remaining</code> and <code>X-RateLimit-Limit-Daily</code> cover the day.</p>



<p class="wp-block-paragraph">Pace against the remaining count rather than reacting to failures, and watch the daily counter separately: burst limits recover in seconds, a daily allowance doesn&#8217;t, and a backfill that exhausts it takes the rest of the day&#8217;s normal traffic with it. The consequence is that polling is expensive and webhooks are cheap. Push where you can; poll only to reconcile.</p>



<h2 class="wp-block-heading">The shape I reach for: CRM in front, system of record behind</h2>



<p class="wp-block-paragraph">When something crosses one of those boundaries, this is the arrangement I default to. It&#8217;s deliberately boring.</p>



<ol class="wp-block-list">
<li><strong>Pick one owner per piece of state and write it down.</strong> Contact details and conversation history belong to the CRM. Money, inventory and anything with an audit requirement belong to the external store. No field gets two owners.</li>

<li><strong>Give every record a stable external ID on both sides.</strong> Do this on day one. Retrofitting identity onto records matched by email is genuinely painful.</li>

<li><strong>Push changes out with webhooks, not polling.</strong> Webhook to a queue, queue to a worker, and make the worker idempotent, because you will receive duplicates and you&#8217;ll need to replay after an outage.</li>

<li><strong>Reconcile on a schedule anyway.</strong> Webhooks get dropped. A nightly sweep comparing counts and updated timestamps, re-pulling only the drift, catches what the push path missed without burning the daily allowance.</li>

<li><strong>Push a flattened summary back for display.</strong> The CRM doesn&#8217;t need the ledger, just the two or three values a human wants on the contact record. Treat those fields as read-only and never let a workflow write them.</li>
</ol>



<p class="wp-block-paragraph">The external side doesn&#8217;t need to be impressive. For a small engagement it&#8217;s usually managed Postgres, or plain Postgres on a modest VPS from somewhere like Contabo or InterServer, sat behind Cloudflare so nothing is directly exposed. The glue is often n8n or Make rather than anything custom. What matters is that constraints exist:</p>



<pre class="wp-block-code"><code>CREATE TABLE job (
  id             bigserial PRIMARY KEY,
  crm_contact_id text        NOT NULL,
  technician_id  bigint      NOT NULL REFERENCES technician(id),
  completed_on   date        NOT NULL,
  value_cents    bigint      NOT NULL CHECK (value_cents &gt;= 0),
  created_at     timestamptz NOT NULL DEFAULT now()
);</code></pre>



<p class="wp-block-paragraph">Nothing clever there. But <code>value_cents</code> can&#8217;t hold &#8220;ask Dave&#8221;, <code>technician_id</code> can&#8217;t point at a technician who doesn&#8217;t exist, and the quarterly margin question becomes one query instead of a conversation about why it can&#8217;t be answered.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<ul class="wp-block-list">
<li><strong>&#8220;We&#8217;re too small for a second system.&#8221;</strong> Sometimes right, and worth taking seriously. But the cost of adding a store later isn&#8217;t the store, it&#8217;s the reconstruction, and some of that history is unrecoverable.</li>

<li><strong>&#8220;We&#8217;ll clean the data up later.&#8221;</strong> Cleanup needs to know what the value should have been. Free-text fields with no history don&#8217;t retain that. This is the one I&#8217;d push back on hardest.</li>

<li><strong>&#8220;Custom Objects solved this.&#8221;</strong> They solved modelling non-person entities, which was the biggest gap. They didn&#8217;t add joins, transactions, versioning or an export path.</li>

<li><strong>&#8220;The client wants everything in one place.&#8221;</strong> They want one place to <em>look</em>, which is a UI requirement, not a storage requirement. A summary pushed back to the contact record satisfies it.</li>

<li><strong>&#8220;We can export whenever we want.&#8221;</strong> Test it before you rely on it. Export the notes. Export a custom object. Do it as the client&#8217;s own user, not as an admin.</li>
</ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<ol class="wp-block-list">
<li><strong>Name the thing.</strong> If it&#8217;s a noun that isn&#8217;t a person, it wants to be a record, not a field on a contact.</li>

<li><strong>Write out the hardest question anyone will ask of it.</strong> The actual sentence. If it needs grouping, arithmetic, or two entity types at once, the CRM isn&#8217;t the home.</li>

<li><strong>Count the writers.</strong> More than one system writing the same field means the CRM holds a copy, not the value.</li>

<li><strong>Decide whether history matters.</strong> Assume it does for anything involving money or consent.</li>

<li><strong>Do the export before you commit.</strong> Not the docs, the actual file. Whatever doesn&#8217;t come out cleanly is data you don&#8217;t really own.</li>

<li><strong>Check the entity budget.</strong> Ten objects per sub-account. If you&#8217;re modelling toward that ceiling, decide now which nouns get a slot.</li>
</ol>



<p class="wp-block-paragraph">Steps two and five take about twenty minutes together and prevent most of what I&#8217;ve described.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I use GoHighLevel as a database for my business data?</h3>



<p class="wp-block-paragraph">For people, conversations, bookings and pipeline state, yes, and it&#8217;s a good one. For anything needing joins, transactions, versioned history or guaranteed export, no. Using <strong>GoHighLevel as a database</strong> for that second category works right up until someone asks a question the platform can&#8217;t express, and by then the data is already ambiguous.</p>



<h3 class="wp-block-heading">How many Custom Objects can a sub-account have?</h3>



<p class="wp-block-paragraph">Ten per location, documented as a hard cap. Each object supports up to ten unique fields, and only single line text, multi line text, number and phone types can be marked unique. Treat those slots as a fixed budget when you plan the model.</p>



<h3 class="wp-block-heading">Can I export Custom Object data out of the platform?</h3>



<p class="wp-block-paragraph">Not through a direct export the way contacts go to CSV. The documented path is a snapshot between sub-accounts, which carries fields and associations but stays inside the platform. The API does support CRUD on objects and records, so a scripted pull is the practical route to a copy you keep.</p>



<h3 class="wp-block-heading">Why do my exported notes look truncated?</h3>



<p class="wp-block-paragraph">Contact export includes only the most recent note, cut to 255 characters. That&#8217;s expected behaviour, not a bug. If notes hold anything you&#8217;d need later, pull them through the API and store them somewhere that keeps all of them.</p>



<h3 class="wp-block-heading">What usually trips the API rate limits?</h3>



<p class="wp-block-paragraph">Backfills, and polling loops that scale with contact count. The limits are 100 requests per 10 seconds and 200,000 per day, both per resource. Read the rate limit headers on each response and pace against them rather than retrying into a wall.</p>



<h3 class="wp-block-heading">Does a sub-account count as a real tenancy boundary?</h3>



<p class="wp-block-paragraph">For operational separation, yes: each is its own location and nothing crosses unless you copy it. The consequence is that cross-client reporting has to happen outside the platform, because no query spans locations. If portfolio-level reporting is a requirement, plan for an external store from the start.</p>



<h3 class="wp-block-heading">Should I migrate off the CRM entirely?</h3>



<p class="wp-block-paragraph">Usually not. A full migration means rebuilding the conversation and outreach layer, which is the part it does best. Moving one category of data out, with a clear owner and a summary pushed back for display, is smaller and reversible.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The CRM will never tell you that you&#8217;ve gone too far. There&#8217;s no constraint to violate, no error to raise, no migration to fail. That&#8217;s exactly what makes the boundary worth drawing on purpose, before the field gets added rather than after four hundred records have been through it.</p>



<p class="wp-block-paragraph">The test I&#8217;d keep: write down the hardest question anyone will ever ask of this data, then check whether the platform can express it. If it can&#8217;t, that&#8217;s not a reporting gap a better dashboard will close. It&#8217;s the boundary telling you where <strong>GoHighLevel as a database</strong> should stop and a real system of record should start. The identity mapping, the webhooks, the reconciliation sweep: all of that is plumbing once the decision is made.</p>



<h2 class="wp-block-heading">Working on this with me</h2>



<p class="wp-block-paragraph">Most of what I do here is unglamorous and ends with the reporting question getting answered:</p>



<ul class="wp-block-list">
<li>Auditing an existing setup and mapping which custom fields have quietly become load-bearing, with an owner assigned to each.</li>

<li>Drawing the boundary: what stays in the CRM, what moves to Custom Objects, what needs an external system of record.</li>

<li>Building the sync layer: webhooks into a queue, idempotent workers, stable ID mapping both ways, and a reconciliation job that catches dropped events without burning the API allowance.</li>

<li>Standing up the store: schema design with constraints that hold, on managed Postgres or a small VPS, with backups you&#8217;ve tested restoring rather than assumed.</li>

<li>Rescuing free-text fields into typed columns, including the judgement calls about what&#8217;s recoverable.</li>

<li>Handover packaging: a real export, documented, so a client can leave with their data whether or not they keep working with you.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;re weighing this up, send me something concrete: a list of your custom field names, a sample export, or the query you can&#8217;t currently answer. That&#8217;s usually enough to tell you where the line sits.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/data-engineering/gohighlevel-as-a-database/">Your CRM Is Not a Database: Where GoHighLevel Should Stop</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Six Signs GoHighLevel Needs a Developer, Not Another Workflow</title>
		<link>https://john-nessime.com/blog/workflow-automation/gohighlevel-custom-development-vs-workflows/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 17 Sep 2026 06:00:00 +0000</pubDate>
				<category><![CDATA[Enterprise Integration]]></category>
		<category><![CDATA[SaaS Engineering]]></category>
		<category><![CDATA[Workflow Automation]]></category>
		<category><![CDATA[Architecture Debt]]></category>
		<category><![CDATA[CRM Audit]]></category>
		<category><![CDATA[Error Handling]]></category>
		<category><![CDATA[Execution Logs]]></category>
		<category><![CDATA[Exponential Backoff]]></category>
		<category><![CDATA[GoHighLevel]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[iPaaS]]></category>
		<category><![CDATA[Marketing Automation]]></category>
		<category><![CDATA[n8n]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Over-Engineering]]></category>
		<category><![CDATA[Private Integration]]></category>
		<category><![CDATA[Project Scoping]]></category>
		<category><![CDATA[Race Conditions]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[Reconciliation]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Sub-Account Management]]></category>
		<category><![CDATA[Webhooks]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=647</guid>

					<description><![CDATA[<p>Adding another workflow is usually the cheap answer, until it isn't. Six signals that a GoHighLevel account has outgrown the builder, the honest case for staying in it, and the three levels of code to try before committing to a marketplace app.</p>
<p>The post <a href="https://john-nessime.com/blog/workflow-automation/gohighlevel-custom-development-vs-workflows/">Six Signs GoHighLevel Needs a Developer, Not Another Workflow</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The account had forty-one workflows. Two were named &#8220;Lead Follow-Up v3&#8221; and &#8220;Lead Follow-Up v3 (USE THIS ONE)&#8221;. When someone asked what happens to a contact who submits the form twice in one afternoon, nobody could answer. Not the agency owner, not the person who built it.</p>



<p class="wp-block-paragraph">That is the moment. Not a crash, not an error banner. A plain question about the system that nobody can answer from the canvas.</p>



<p class="wp-block-paragraph">GoHighLevel is good at what it was built for and most accounts never need anything else. But there is a point where the next workflow makes the account worse instead of better, and past it, <strong>GoHighLevel custom development</strong> costs less than another six months of patching. Below: the failure mode hiding under all the others, six signals you have crossed that line, the honest case for staying in the builder, and three levels of code to try before committing to a full app.</p>



<h2 class="wp-block-heading">The failure hiding under all the others</h2>



<p class="wp-block-paragraph">The workflow builder has no failure semantics. Everything below is a consequence of that.</p>



<p class="wp-block-paragraph">When a step works, the contact moves on. When a step fails, the contact usually also moves on. No transaction, no rollback, no retry budget you control, no way to say &#8220;this contact is in a bad state, stop and tell someone&#8221;. A Custom Webhook that gets a 500 back does not park the contact for review. It becomes a line in an execution log nobody reads, and the contact continues as though the other system had answered.</p>



<p class="wp-block-paragraph">Fine when the step sends a text message. Not fine when the step is the only thing keeping two systems in agreement. So the useful test is not &#8220;is this hard to build in the builder&#8221;, because plenty of hard things build fine. The test is: <em>what happens when this step fails, and would I find out?</em></p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Signal one: you are storing state in tags</h2>



<p class="wp-block-paragraph">Tags start as labels. Then someone needs to know whether a lead was called, so there is a <code>called</code> tag. Then whether they were called twice, so <code>called-2</code>. Then a workflow that removes <code>called</code> when a new form arrives, and another that puts it back if an appointment was booked first.</p>



<p class="wp-block-paragraph">Those are no longer labels. That is a state machine with no diagram, no validation, and no guarantee that only one workflow writes to it at a time. Two workflows fired by the same event will both read the old value and both write, and whichever finishes last wins. That is a race condition, and it hides during testing because you test one contact at a time. The tell is a tag name with a number in it, or a tag whose meaning needs the word &#8220;unless&#8221; to explain.</p>



<h2 class="wp-block-heading">Signal two: the data has to be right, not eventually right</h2>



<p class="wp-block-paragraph">Marketing automation tolerates drift. One extra nurture email hurts nobody. The moment GoHighLevel holds something that must reconcile with another system, that tolerance is gone. Order status from a store, invoice state from an accounting package, availability shared with a clinic system.</p>



<p class="wp-block-paragraph">Those need idempotency, meaning a repeated message produces the same result as a single one, and reconciliation, meaning something periodically compares both sides and reports the difference. The builder gives you neither: no natural home for an idempotency key, no scheduled job that reads both systems and complains. You can fake reconciliation with a workflow on a timer, but that is a batch job written in a UI designed for follow-up sequences, and the first time it half-runs you will not hear about it.</p>



<h2 class="wp-block-heading">Signal three: per-execution billing became a per-lead tax</h2>



<p class="wp-block-paragraph">The actions that make workflows powerful are premium actions, billed per execution against the agency wallet. Custom Webhook, Inbound Webhook, Custom Code and the workflow AI actions all sit in that category. HighLevel gives each sub-account a small free allowance when premium actions are first switched on, publishes volume tiers above that, and lets you rebill sub-accounts individually instead of absorbing the charge. Take the current rate and tiers from HighLevel&#8217;s billing documentation rather than any blog post, including this one.</p>



<p class="wp-block-paragraph">The mechanism is the point. You pay per execution, not per useful outcome. A workflow calling out three times to enrich, score and push a lead is three executions per lead. Multiply by volume and sub-accounts, and the plumbing line can pass the cost of a small service doing all three in one request.</p>



<p class="wp-block-paragraph">Two caveats. Consolidating three calls into one endpoint is usually cheaper than moving the whole thing to code. And whatever you move still costs something to run, whether that is a VPS from a provider like Contabo or InterServer, a hosted platform such as Make or n8n, or a serverless function. Compare both sides.</p>



<h2 class="wp-block-heading">Signal four: the same logic lives in twenty sub-accounts</h2>



<p class="wp-block-paragraph">Snapshots are the platform&#8217;s answer and better than people expect. You can refresh a snapshot and push selected assets to sub-accounts that previously imported it, choosing both the assets and the recipients, and the push is selective rather than destructive.</p>



<p class="wp-block-paragraph">Two limits. Push Update only reaches sub-accounts inside your own agency, so anything shared externally has to be re-imported through a new link. And HighLevel documents that when a snapshot refresh deletes steps, contacts waiting on those steps in workflows created from that snapshot are removed. Fine for a copy change, destructive for a structural one.</p>



<p class="wp-block-paragraph">Snapshots distribute structure. They do not give you versioning, a rollback path, a diff between intended and running state, or a test that fails before the change reaches a client. Once a change has to land correctly in twenty accounts on one day, you want those, and they come from putting the logic behind an API you control and calling it from a thin step in each account.</p>



<h2 class="wp-block-heading">Signal five: the integration must survive the other side being down</h2>



<p class="wp-block-paragraph">A Custom Webhook fires the moment the contact reaches the step. If your endpoint is slow, restarting or rate limited, that single firing is what you get. No queue in front, no backoff behind.</p>



<p class="wp-block-paragraph">The API side has numbers worth knowing. HighLevel documents a burst limit of 100 requests per 10 seconds and a daily limit of 200,000 requests, counted per marketplace app per resource, where a resource is a location or the company. Every response carries headers telling you where you stand, so backoff can follow what the server says instead of a guessed sleep.</p>



<pre class="wp-block-code"><code>curl -sS -D - -o /dev/null 
  -H "Authorization: Bearer $GHL_TOKEN" 
  -H "Version: $GHL_API_VERSION" 
  "$GHL_API_BASE/contacts/" 
  | grep -i '^x-ratelimit'</code></pre>



<p class="wp-block-paragraph"><code>-D -</code> writes response headers to stdout while <code>-o /dev/null</code> discards the body, so you read the rate limit state without dumping contact data into your shell history. Set the base URL and the required <code>Version</code> value from the current API reference. Watch <code>X-RateLimit-Max</code> and <code>X-RateLimit-Remaining</code> for the burst window, <code>X-RateLimit-Interval-Milliseconds</code> for its length, and <code>X-RateLimit-Limit-Daily</code> with <code>X-RateLimit-Daily-Remaining</code> for the day.</p>



<p class="wp-block-paragraph">The daily budget sounds enormous until a migration. Reading a contact, upserting it, then writing custom fields is three calls each, so forty thousand contacts is 120,000 calls before live traffic touches the account. Batch the reads, skip writes where nothing changed, run backfills overnight. Watch for loops too: writes made through the API can themselves fire workflow triggers, so a receiver that writes back has built a cycle.</p>



<h2 class="wp-block-heading">Signal six: someone needs a screen, not a form</h2>



<p class="wp-block-paragraph">Forms collect. They do not display. Once a client asks to see a filtered view of their own data, approve something in two clicks, or work a queue that is not a pipeline, you are looking at an interface.</p>



<p class="wp-block-paragraph">The platform supports this. Marketplace apps can add Custom Pages, which render inside HighLevel in an embedded iframe loading a URL you host, appearing on the app details page or in the left navigation. HighLevel publishes an app template to start from, plus an SSO mechanism so your page can identify the user instead of asking them to log in again. That is still a real application with hosting, auth and deployment: the biggest step on this list, the one I would defer longest, and the only honest answer when the requirement is a screen.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Where the workflow builder genuinely wins</h2>



<p class="wp-block-paragraph">A one-sided argument is not worth reading, so here is the case against writing code.</p>



<ul class="wp-block-list">
<li><strong>The person who owns the outcome can change it.</strong> An account manager adjusts send timing on a Friday with no deploy and no developer awake.</li>



<li><strong>Messaging and scheduling are solved.</strong> SMS, email, calendars, reminders and the delivery infrastructure behind them are the platform&#8217;s core. Rebuilding any of it is a bad trade.</li>



<li><strong>Timezone handling is built in.</strong> Time-based steps run in the account timezone or the contact&#8217;s own, falling back to the account when a contact has none.</li>



<li><strong>No pipeline to maintain.</strong> No repository, no secrets rotation, no dependency upgrades, no server needing patched at an awkward hour.</li>



<li><strong>Execution history is right there.</strong> For a linear sequence, reading what happened to a contact beats correlating logs across two systems.</li>
</ul>



<p class="wp-block-paragraph">If the automation is a follow-up sequence with branching, the builder is correct and code is the expensive mistake.</p>



<h2 class="wp-block-heading">Three levels before you build an app</h2>



<p class="wp-block-paragraph">&#8220;Needs a developer&#8221; is not one decision. It is three, and most accounts only need the first.</p>



<h3 class="wp-block-heading">Level one: the Custom Code action</h3>



<p class="wp-block-paragraph">The builder has a Custom Code premium action that runs JavaScript inside the workflow. Values from earlier steps are mapped in as named input properties and arrive in an <code>InputData</code> object. The action returns a JavaScript object or an array of objects for later steps to reference, and <code>console.log</code> output is captured so you can see what ran.</p>



<pre class="wp-block-code"><code>// Normalise a phone number before it reaches a branch step.
// rawPhone and country are input properties mapped from earlier steps.
const digits = String(InputData.rawPhone || "").replace(/D/g, "");
const cc = InputData.country === "US" ? "1" : "";

if (!digits) {
  console.log("no phone on contact, routing to manual branch");
  return { e164: "", usable: false };
}

return {
  e164: "+" + (digits.startsWith(cc) ? digits : cc + digits),
  usable: digits.length &gt;= 10
};</code></pre>



<p class="wp-block-paragraph">This kills a lot of branch sprawl. Ten If/Else steps existing only to reshape a value collapse into one action returning a clean value and a boolean. Still inside the platform, still billed per execution, still no retries. But logic smeared across a canvas now reads in one place.</p>



<h3 class="wp-block-heading">Level two: a private integration and a script you own</h3>



<p class="wp-block-paragraph">For one account you do not need a marketplace app. Create a Private Integration inside the sub-account, select only the scopes the job needs, and use the token as a bearer credential. No OAuth handshake, no refresh cycle, no listing.</p>



<p class="wp-block-paragraph">This is where reconciliation jobs, nightly audits, migrations and bulk cleanups belong. Run it as a cron job on a small VPS, behind a Cloudflare Tunnel if it needs an inbound endpoint, with the token in a secrets store rather than a config file. Treat that token as a credential with a rotation plan, not a string you paste once and forget.</p>



<h3 class="wp-block-heading">Level three: a marketplace app</h3>



<p class="wp-block-paragraph">You need this when the thing spans many accounts, has a user interface, or is a product you intend to sell. It means OAuth 2.0 on the authorization code grant, token storage and refresh per installed location, webhook handling, and somewhere to run it. Note the version boundary: the v1 API is end of life and unmaintained, and new builds target the OAuth-based v2. If a tutorial hands you a permanent location API key, it is describing the old world.</p>



<p class="wp-block-paragraph">Marketplace apps can also register their own workflow triggers and actions, which appear in the builder for sub-accounts that installed the app. That is the pattern I reach for first: the hard logic lives in your service, and the person who owns the campaign still drags a step onto a canvas. It requires the workflows read scope, and those custom actions are themselves premium actions billed per execution.</p>



<h2 class="wp-block-heading">Arguments that do not survive contact</h2>



<ul class="wp-block-list">
<li><strong>&#8220;We will just add one more workflow.&#8221;</strong> The cost is not building it. It is that every future workflow now has to be reasoned about alongside it. Ten workflows listening for one tag is not ten problems, it is the interactions between them.</li>



<li><strong>&#8220;No-code means no developer.&#8221;</strong> It means no deployment. Someone still holds the whole model in their head, and when they leave, an undocumented canvas is harder to inherit than a repository with a README.</li>



<li><strong>&#8220;Zapier or Make will handle it.&#8221;</strong> Often true and worth trying first. What they do not give you is one place where business logic lives. Split rules across three tools and debugging means three log formats.</li>



<li><strong>&#8220;It works, so it is fine.&#8221;</strong> Every silent failure worked right up until someone counted. Absence of errors in a system with no error reporting is not evidence.</li>



<li><strong>&#8220;Custom code is lock-in.&#8221;</strong> Backwards. Logic on a canvas is bound to one vendor&#8217;s UI and exports only as a snapshot. Logic in a repository moves.</li>
</ul>



<h2 class="wp-block-heading">How I would decide</h2>



<ol class="wp-block-list">
<li><strong>Write down what happens when each external call fails.</strong> If the answer anywhere is &#8220;the contact continues as if it succeeded&#8221; and that matters, you have found your requirement.</li>



<li><strong>Count the workflows writing to the same tag or field.</strong> More than two means you have a state machine. Draw it before touching anything.</li>



<li><strong>Count premium executions per lead, then multiply by monthly volume.</strong> Compare against a few hours a month maintaining a small service. Sometimes the builder still wins. Do the arithmetic instead of assuming.</li>



<li><strong>Ask how a change reaches every account.</strong> If it involves opening sub-accounts one at a time, that is the real cost, and it grows with every client.</li>



<li><strong>Try the smallest level that fits, and leave the trigger in the builder.</strong> Custom Code action, then private integration script, then marketplace app. Whatever moves out, keep the entry point as a workflow step so the marketing side can still change timing and audience without you.</li>
</ol>



<p class="wp-block-paragraph">Whatever you build, monitor it. An uptime check on your receiver and an alert when the reconciliation job reports a non-zero difference will catch more real problems than any amount of careful workflow naming.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need a marketplace app to use the GoHighLevel API?</h3>



<p class="wp-block-paragraph">No. For a single sub-account, create a Private Integration inside that account, grant only the scopes you need, and use the token as a bearer credential. Marketplace apps and OAuth are for anything spanning accounts or distributed to other people.</p>



<h3 class="wp-block-heading">How much automation should stay in workflows?</h3>



<p class="wp-block-paragraph">The parts a non-developer should be able to change: timing, audience, message copy, branching on obvious contact attributes. Move out the parts where being wrong is expensive and being wrong is silent.</p>



<h3 class="wp-block-heading">Why does my workflow show as successful when the integration clearly failed?</h3>



<p class="wp-block-paragraph">A step firing and a step achieving something are different events, and the builder records the first. A webhook returning an error still counts as having run. If the outcome matters, verify it explicitly rather than trusting the log.</p>



<h3 class="wp-block-heading">Should I avoid premium workflow actions on cost grounds?</h3>



<p class="wp-block-paragraph">Not on principle. They cost per execution because they are the expensive verbs. The problem is fragmentation: three separate calls where one would do triples the bill for the same outcome. Consolidate before considering a move off the platform.</p>



<h3 class="wp-block-heading">What does a GoHighLevel custom development project usually start with?</h3>



<p class="wp-block-paragraph">An audit, not a build. Map which workflows write to which fields, where external calls happen, and what the failure path is for each. Often that document alone removes the need for the project.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Ask what happens when a step fails. That question separates automation belonging in the builder from automation needing code, more reliably than counting steps, counting workflows, or judging how clever the branching has become.</p>



<p class="wp-block-paragraph">If failure is visible and cheap, stay in the builder and enjoy that your client can edit it themselves. If failure is silent and expensive, GoHighLevel custom development stops being an upgrade and becomes maintenance you already deferred. Then start small: a Custom Code action beats a marketplace app you will not want to maintain.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Working on this with me</h2>



<p class="wp-block-paragraph">I work on the engineering side of GoHighLevel accounts, usually the parts failing quietly rather than the parts that look broken.</p>



<ul class="wp-block-list">
<li>Auditing an account for silent failures: which workflows write to which tags and fields, and what happens on every external call that returns an error.</li>



<li>Replacing branch sprawl with Custom Code actions, so logic reads as one block instead of fifteen steps on a canvas.</li>



<li>Webhook receivers and sync services that retry, deduplicate and reconcile, with the rate limit headers driving backoff instead of a fixed sleep.</li>



<li>Migrations and backfills planned against the daily API budget, batched and run in the account&#8217;s timezone so live traffic keeps working.</li>



<li>Private integration setup with least-privilege scopes, token storage and a rotation plan that is written down.</li>



<li>Marketplace app work when genuinely warranted: OAuth handling, custom workflow actions, and Custom Pages hosted somewhere you control.</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion, send something concrete: a screenshot of the workflow you are least sure about, an execution log entry that makes no sense, or the response body your webhook endpoint is returning. That is a faster conversation than a scoping call.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/workflow-automation/gohighlevel-custom-development-vs-workflows/">Six Signs GoHighLevel Needs a Developer, Not Another Workflow</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Primavera P6 vs Unifier vs Aconex vs Textura: Which Oracle Product Do You Actually Need?</title>
		<link>https://john-nessime.com/blog/devops/primavera-p6-vs-unifier-vs-aconex/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Fri, 28 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[BIM]]></category>
		<category><![CDATA[Capital Projects]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Common Data Environment]]></category>
		<category><![CDATA[Construction Technology]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Enterprise Software]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Oracle]]></category>
		<category><![CDATA[Oracle Aconex]]></category>
		<category><![CDATA[Oracle Textura]]></category>
		<category><![CDATA[Primavera Gateway]]></category>
		<category><![CDATA[Primavera P6]]></category>
		<category><![CDATA[Primavera Unifier]]></category>
		<category><![CDATA[Project Controls]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Self Hosting]]></category>
		<category><![CDATA[Software Selection]]></category>
		<category><![CDATA[System of Record]]></category>
		<category><![CDATA[Vendor Lock-In]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=252</guid>

					<description><![CDATA[<p>All four Oracle Construction and Engineering products overlap on purpose, and three of them hold cost. The damage comes from owning two and never deciding which one is authoritative. A practical comparison by whose record each product holds, plus a decision procedure and the integration constraints nobody scopes.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/primavera-p6-vs-unifier-vs-aconex/">Primavera P6 vs Unifier vs Aconex vs Textura: Which Oracle Product Do You Actually Need?</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The forecast in the monthly report doesn&#8217;t match the forecast in the cost system. Someone asks which one is right, and the honest answer is that nobody knows, because there are three places holding a number and nobody ever wrote down which one wins.</p>



<p class="wp-block-paragraph">That conversation is usually the first sign that an organisation bought two Oracle products without deciding what each one was for. It is a common outcome, and it is not because the products are bad. It&#8217;s because they are sold as a suite, they overlap on purpose, and the sales conversation is about capability rather than authority.</p>



<p class="wp-block-paragraph">This post is a working comparison of the four Oracle Construction and Engineering products people most often confuse: <strong>Primavera P6 vs Unifier vs Aconex</strong>, plus Textura. One profile each, what it genuinely owns, where it wins, where it doesn&#8217;t, and then the part nobody covers: how to decide which system holds the authoritative number when three of them offer a cost module.</p>



<h2 class="wp-block-heading">The question that actually sorts them</h2>



<p class="wp-block-paragraph">Feature lists will not separate these products. All four manage documents. Three of them track cost. Two of them hold a schedule. If you compare capability columns you will conclude that you need all four, which is exactly the conclusion the suite is designed to produce.</p>



<p class="wp-block-paragraph">The question that does separate them is: <em>whose record is this, and who else needs to write to it?</em></p>



<p class="wp-block-paragraph">Two of these products are single-organisation systems. You run them, you own the data, and outside parties get in only if you give them a seat and they accept your governance. One of them is deliberately multi-organisation, where every company on the project holds its own workspace and decides what it shares. And one is a transactional network that touches money leaving your bank account, which makes it a finance system wearing a construction badge.</p>



<p class="wp-block-paragraph">Get that axis right and the rest of the decision mostly falls out.</p>



<h2 class="wp-block-heading">Primavera P6: the schedule, and only the schedule</h2>



<p class="wp-block-paragraph">P6 is a critical path method scheduling engine with resource and cost loading on top. It exists to answer questions about time: what drives the finish date, what happens if this activity slips, what does the resource histogram look like if we accelerate.</p>



<p class="wp-block-paragraph">Two things confuse people here. First, P6 comes in two shapes. P6 Professional is the desktop client that planners actually live in. P6 EPPM is the enterprise web layer with the shared database, portfolio views and web services. Most organisations run both against the same database, and most planners will fight you to keep the desktop client.</p>



<p class="wp-block-paragraph">Second, Oracle Primavera Cloud is a separate product, not P6 in a browser. Oracle says so on its own product page, which is unusually direct for vendor copy. Primavera Cloud shares a lineage with P6 and is positioned as the cloud destination, but the data model is not identical and things planners rely on in P6 do not all have equivalents. If someone tells you a move to Primavera Cloud is a hosting change, they have not tried it.</p>



<h3 class="wp-block-heading">Where P6 wins</h3>



<ul class="wp-block-list">
<li>Contractual programme scheduling, where the schedule is a deliverable and gets audited</li>

<li>Forensic delay work, because the XER and PMXML formats are what every expert and every third-party analysis tool expects</li>

<li>Resource-loaded planning across a portfolio with a shared resource pool</li>

<li>Anywhere the client&#8217;s contract names it, which is more places than you would expect</li>
</ul>



<h3 class="wp-block-heading">Where P6 doesn&#8217;t</h3>



<ul class="wp-block-list">
<li>Cost control. P6 has cost fields and they are useful for cash flow shape, but a resource-loaded schedule is not a commitment ledger. It has no concept of a purchase order lifecycle, a variation approval or a payment certificate</li>

<li>Correspondence and document control. There is nothing here for transmittals, RFIs or drawing registers</li>

<li>Collaboration across companies. Subcontractors do not get P6 logins, and you would not want them writing to your programme if they did</li>
</ul>



<h2 class="wp-block-heading">Primavera Unifier: a business process engine that happens to know about projects</h2>



<p class="wp-block-paragraph">Unifier is the one people describe worst, usually as &#8220;the cost one&#8221;. That undersells it and also sets bad expectations.</p>



<p class="wp-block-paragraph">Unifier is a configurable workflow platform for capital programmes. You define business processes: a change order, a funding request, an invoice, a payment application, a site instruction. Each one has a form, a workflow with approval steps and routing rules, and a defined effect on the cost sheet. Oracle ships preconfigured processes and a no-code designer called uDesigner for changing them or building your own. Alongside the business processes sit the managers, including the Cost Manager, Schedule Manager and Document Manager.</p>



<p class="wp-block-paragraph">It&#8217;s sold as base products, principally Project Controls and Facilities and Asset Management, which matters because Unifier&#8217;s remit runs past handover into operating the asset.</p>



<p class="wp-block-paragraph">The honest trade-off with Unifier is this: its configurability is the reason to buy it and the reason implementations run long. A platform that can model any approval workflow will not tell you what your approval workflow should be. If your organisation has not written its cost control process down, Unifier will faithfully automate the argument you are having instead of the process you wanted.</p>



<h3 class="wp-block-heading">Where Unifier wins</h3>



<ul class="wp-block-list">
<li>Owner-side capital programme control, where budget, funding, commitments, forecast and actuals need to roll up across many projects</li>

<li>Anywhere approvals need an audit trail with defined routing, delegation and thresholds</li>

<li>Public sector and utility programmes with fund tracking, where money has strings attached and the strings must be reportable</li>

<li>Organisations that want one governed process applied to every project rather than per-project improvisation</li>
</ul>



<h3 class="wp-block-heading">Where Unifier doesn&#8217;t</h3>



<ul class="wp-block-list">
<li>Supply chain collaboration. It&#8217;s your instance, your governance, your seats. External parties can be given access, but they are guests in your system and they behave like it</li>

<li>Drawing and model control at scale. The Document Manager is real, but it is not a common data environment and it is not built for design coordination</li>

<li>Fast starts. Configuration is a project in its own right, and treating it as an install is the most common way to burn a first year</li>
</ul>



<h2 class="wp-block-heading">Aconex: the project record, owned by nobody in particular</h2>



<p class="wp-block-paragraph">Aconex is a common data environment for correspondence, documents, workflows and models across every organisation on a project. Document control, transmittals, RFIs, submittals, drawing registers, model coordination, field inspections.</p>



<p class="wp-block-paragraph">Its defining property is not a feature. It is the data ownership model. Every organisation on the project gets its own workspace and controls what it shares and when. That neutrality is the entire commercial argument, and it is the thing you cannot replicate by inviting subcontractors into your instance of something else.</p>



<p class="wp-block-paragraph">This matters more than it sounds. A designer who believes the main contractor can read their internal drafts will keep their real work somewhere else, and your &#8220;single source of truth&#8221; quietly becomes a partial one. The reason Aconex accumulates a complete project record is that nobody is afraid of it. Oracle&#8217;s own materials lean on that neutrality hard, and in this case the marketing claim and the engineering consequence are the same thing.</p>



<p class="wp-block-paragraph">On the model side, Aconex Model Coordination is design-neutral and built on open BIM standards including IFC and BCF, so it sits alongside the authoring tools rather than replacing them.</p>



<h3 class="wp-block-heading">Where Aconex wins</h3>



<ul class="wp-block-list">
<li>Multi-party projects where the correspondence record will be read by lawyers later</li>

<li>Design-build and joint venture delivery, where no single party should hold everyone&#8217;s data</li>

<li>Document and model volume, which Oracle&#8217;s datasheets describe as unbounded on data and participants</li>

<li>Handover, because the audit trail and the linked document set are the deliverable</li>
</ul>



<h3 class="wp-block-heading">Where Aconex doesn&#8217;t</h3>



<ul class="wp-block-list">
<li>Portfolio-level capital planning. Aconex is organised around projects, not around a programme of investment decisions</li>

<li>Asset lifecycle after handover. The record ends where the project ends</li>

<li>Deep enterprise cost governance. Connected Cost exists and is genuinely useful for project cost, but if your organisation needs funding, capital planning and multi-year rollup, that is Unifier&#8217;s shape</li>
</ul>



<h2 class="wp-block-heading">Textura: money leaving the building</h2>



<p class="wp-block-paragraph">Textura came into Oracle by acquisition and still feels different from the rest of the suite, because it is solving a different class of problem.</p>



<p class="wp-block-paragraph">Textura Payment Management runs the construction payment cycle: initiating a draw, subcontractor pay applications, markups and approvals, owner billing, collecting lien waivers and compliance documents, enforcing compliance holds, payment authorisation and disbursement. It&#8217;s a network product. Your subcontractors transact in it, and the value comes from them being there.</p>



<p class="wp-block-paragraph">The compliance side is the part that decides whether it&#8217;s worth buying. If your risk is a subcontractor&#8217;s insurance lapsing quietly and nobody noticing until a claim, an automated payment hold tied to a missing document is a control you cannot easily build with a spreadsheet and good intentions.</p>



<h3 class="wp-block-heading">Where Textura wins</h3>



<ul class="wp-block-list">
<li>General contractors with a wide subcontractor base and a monthly draw cycle that eats accounts payable alive</li>

<li>Jurisdictions with real lien exposure, where waiver collection is a legal control rather than paperwork</li>

<li>Compliance enforcement at the point of payment, which is the only moment anyone reliably pays attention</li>

<li>Audit trails on who approved what, signed what and released what</li>
</ul>



<h3 class="wp-block-heading">Where Textura doesn&#8217;t</h3>



<ul class="wp-block-list">
<li>It is not a cost control system. It handles the payment leg, not the budget, forecast or commitment ledger</li>

<li>It does not replace your ERP. It integrates with it, and that integration is part of the implementation cost</li>

<li>Adoption is a supply chain problem, not a software problem. If your subcontractors will not onboard, you own a slower process than the one you replaced</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The overlap trap, and why the cost module is where it bites</h2>



<p class="wp-block-paragraph">Here&#8217;s the failure mode that is invisible at procurement and expensive eighteen months later.</p>



<p class="wp-block-paragraph">Three of these products can hold cost. P6 has cost-loaded activities. Unifier has a full Cost Manager. Aconex has Connected Cost. Nobody sets out to run three cost systems, but it happens gradually: the schedule gets cost-loaded because the client asked for an S-curve, the finance team runs the commitment ledger in Unifier, and the project team starts logging variations in Aconex because that&#8217;s where the correspondence already is.</p>



<p class="wp-block-paragraph">Now you have three forecasts. They disagree. Not because anyone is wrong, but because they are answering different questions with different cut-off dates and different accrual rules. And the reconciliation lands in a spreadsheet maintained by one person, which is the outcome every one of these products was bought to prevent.</p>



<p class="wp-block-paragraph">The fix is a decision, not a tool. Before anything is configured, write down one sentence per data class: which system is authoritative, and what the others are allowed to hold.</p>



<ul class="wp-block-list">
<li><strong>Schedule dates and logic:</strong> P6 is authoritative. Everything else receives dates, nothing else edits them</li>

<li><strong>Budget, commitments and forecast:</strong> pick one. If you own Unifier, it is Unifier. If you don&#8217;t, Connected Cost is a reasonable home, but say so out loud</li>

<li><strong>Correspondence and controlled documents:</strong> Aconex if it&#8217;s on the project, and if it is, do not run a parallel drawing register anywhere else</li>

<li><strong>Subcontractor payment and compliance status:</strong> Textura, with the resulting actuals flowing to whichever system owns cost</li>

<li><strong>Everything the authoritative system does not own:</strong> read-only, refreshed by integration, and clearly labelled in reports as a copy</li>
</ul>



<p class="wp-block-paragraph">That list takes an afternoon and saves a year. It&#8217;s also the single most common thing missing from Oracle Primavera implementations I get asked to look at.</p>



<h2 class="wp-block-heading">Integration is the real cost, and it has a shape</h2>



<p class="wp-block-paragraph">Once you own two of these, you own an integration. Budget for it as a workstream, not a checkbox.</p>



<p class="wp-block-paragraph">Oracle offers more than one path. <strong>Primavera Gateway</strong> is the established broker for P6 and Unifier, with built-in providers per application and a canonical intermediate format so mappings survive version changes. In the P6 to Unifier direction, Gateway maps the P6 Project ID to Unifier&#8217;s Project Number, and what flows depends on how the schedule is built: a duration-based schedule sends durations, a resource-loaded one sends durations and units. Oracle also has a lighter integration platform for exchanging data between Unifier, P6, Aconex and Primavera Cloud, so check which one your version and deployment model actually supports before designing anything.</p>



<p class="wp-block-paragraph">Aconex Connected Cost takes a different route, talking to P6 over its web services API. Two constraints from that integration are worth knowing before you draw an architecture diagram:</p>



<ul class="wp-block-list">
<li>You can configure both a P6 and a Primavera Cloud integration for the organisation, but an individual project can only link to one of them at a time</li>

<li>Self-hosted or third-party-hosted P6 servers have to be approved by Oracle and reachable over the internet before the integration can be established. Oracle-hosted P6 skips that approval step</li>
</ul>



<p class="wp-block-paragraph">That second one has teeth if you are running P6 EPPM on your own infrastructure. &#8220;Internet accessible&#8221; and &#8220;exposed&#8221; are not the same thing, and the gap between them is where your security review lives. If you&#8217;re standing up middleware or a reverse proxy in front of an on-premise P6 or Unifier deployment, that box needs to be treated as production: a dedicated host from a provider like Contabo or InterServer rather than a corner of an existing server, TLS terminated properly, access restricted to known networks, and remote administrative access over a business VPN such as NordLayer or Surfshark rather than a port forward someone opened for a consultant and forgot about.</p>



<p class="wp-block-paragraph">Back up the database host properly too, and test a restore. On-premise P6 and Unifier estates are frequently the last thing anyone thinks about until they need it, and imaging tools from vendors like O&amp;O Software earn their keep on exactly this kind of long-lived, rarely-touched server.</p>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<p class="wp-block-paragraph">Work through this in order and stop at the first answer that fits. Buying more than you need is far more common than buying too little.</p>



<ol class="wp-block-list">
<li><strong>Name the pain in one sentence.</strong> &#8220;We cannot defend our programme in a delay claim&#8221; points at P6. &#8220;We cannot tell the board what the capital programme will cost&#8221; points at Unifier. &#8220;We cannot find the approved revision&#8221; points at Aconex. &#8220;Subcontractor payments take six weeks and compliance is a filing cabinet&#8221; points at Textura</li>

<li><strong>Count the organisations that must write to the system.</strong> One, and you are choosing between P6 and Unifier. Many, and you are choosing between Aconex and Textura</li>

<li><strong>Ask whether the problem outlives the project.</strong> Assets, maintenance and multi-year capital planning are Unifier&#8217;s territory. Anything that ends at handover is not</li>

<li><strong>Check what the contract already mandates.</strong> Plenty of clients name a scheduling tool or a project CDE in the contract documents. That decision is made, so spend your effort elsewhere</li>

<li><strong>Ask the reseller how each product meters.</strong> Per named user, per project, per transaction or per value processed changes the architecture, not just the invoice. A per-user product pushes you towards fewer, more senior users, and that changes who does data entry</li>

<li><strong>Only then decide on a second product,</strong> and write the system-of-record sentence for every data class the two of them share before signing anything</li>
</ol>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<ul class="wp-block-list">
<li><strong>&#8220;Unifier can do document control, so we don&#8217;t need Aconex.&#8221;</strong> Technically true, practically false on multi-party projects. The obstacle is not features, it is that other companies will not put their real correspondence in your instance</li>

<li><strong>&#8220;Aconex has cost, so we don&#8217;t need Unifier.&#8221;</strong> Fine for project-level cost. It falls down when you need portfolio funding, capital planning and asset lifecycle</li>

<li><strong>&#8220;Primavera Cloud is just P6 in the browser.&#8221;</strong> Oracle itself states that Primavera Cloud is not P6. Treat any migration as a data model exercise with a gap analysis, not a lift and shift</li>

<li><strong>&#8220;We&#8217;ll integrate it later.&#8221;</strong> Later means after users have created data under two incompatible breakdown structures. Agree the project and cost breakdown codes on day one, even if the integration itself waits</li>

<li><strong>&#8220;The suite integrates out of the box.&#8221;</strong> Connectors exist and they are good. Mapping your codes, your workflow states and your calendars is still work, and it is work that needs someone who understands both sides</li>
</ul>



<h2 class="wp-block-heading">Practices worth adopting early</h2>



<ul class="wp-block-list">
<li>Standardise the project ID before anything else. Gateway maps P6&#8217;s Project ID to Unifier&#8217;s Project Number, and a free-text naming convention will hurt on the day you connect them</li>

<li>Keep the schedule in one place and let it flow outward. Bidirectional date editing across two systems is a support ticket generator</li>

<li>Pilot Unifier configuration on one real process end to end before configuring twenty. uDesigner makes it easy to build the wrong thing quickly</li>

<li>Onboard the supply chain before go-live on Aconex or Textura. Network products fail on adoption, not on uptime</li>

<li>Export your own data on a schedule. XER and PMXML for schedules, and regular extracts from the cloud products, so you are not negotiating from zero at renewal</li>

<li>Write the system-of-record statement into the implementation charter, where people will actually see it</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">What is the difference between Primavera P6 and Unifier?</h3>



<p class="wp-block-paragraph">P6 is a critical path scheduling tool that answers questions about time and resources. Unifier is a configurable business process and cost platform that answers questions about money, approvals and governance across a capital programme. They overlap slightly because P6 can hold cost data and Unifier has a Schedule Manager, but neither is a substitute for the other.</p>



<h3 class="wp-block-heading">Do I need Aconex if I already have Unifier?</h3>



<p class="wp-block-paragraph">If the project involves several independent organisations exchanging drawings and correspondence, usually yes. Unifier&#8217;s Document Manager works well for your own records, but external parties are guests in your instance. Aconex gives every organisation its own workspace and control over what it shares, which is what makes the project record complete rather than partial.</p>



<h3 class="wp-block-heading">Is Oracle Primavera Cloud replacing P6?</h3>



<p class="wp-block-paragraph">Oracle positions Primavera Cloud as the cloud platform and publishes migration guidance from P6, while continuing to sell and support P6 EPPM. Oracle also states plainly that Primavera Cloud is not P6. Plan on the two coexisting for now, and treat any migration as a data model exercise with a proper gap analysis rather than a hosting change.</p>



<h3 class="wp-block-heading">Can Textura replace our accounting system?</h3>



<p class="wp-block-paragraph">No. Textura handles the construction payment cycle: pay applications, approvals, lien waivers, compliance holds and disbursement. It integrates with ERP systems rather than replacing them, and that integration is a real line item in the implementation.</p>



<h3 class="wp-block-heading">Which product should a general contractor start with?</h3>



<p class="wp-block-paragraph">It depends on which pain is loudest. If schedules are being disputed, P6. If the project record is scattered across email, Aconex. If the monthly draw cycle and subcontractor compliance are consuming the finance team, Textura. Contractors reach for Unifier less often than owners do, because Unifier&#8217;s strengths are programme-level funding and governance.</p>



<h3 class="wp-block-heading">How hard is it to integrate P6 with Unifier?</h3>



<p class="wp-block-paragraph">The mechanism is well trodden through Primavera Gateway, which ships providers for both. The hard part is not the connector, it&#8217;s agreement: matching project identifiers, deciding which system owns dates, and reconciling breakdown structures that were designed independently. Expect the mapping work to take longer than the technical setup.</p>



<h3 class="wp-block-heading">What data should I be exporting for portability?</h3>



<p class="wp-block-paragraph">For schedules, take regular XER and PMXML exports, since those formats are what independent analysis tools and delay experts read. For the cloud products, set up scheduled extracts of documents, transaction records and audit trails. The point is not distrust of the vendor, it&#8217;s that a project record you cannot read without an active subscription is a liability during a dispute.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The <strong>Primavera P6 vs Unifier vs Aconex</strong> comparison, with Textura alongside them, is not really a features question. Each of these products is good at the thing it was built for, and Oracle is not trying to trick anyone by giving three of them a cost module.</p>



<p class="wp-block-paragraph">The damage comes from owning two of them and never deciding which one is authoritative. Sort the products by whose record they hold and how many organisations write to them, pick the smallest set that covers your loudest pain, and write down the system of record for every data class before a single workflow gets configured.</p>



<p class="wp-block-paragraph">Do that and the monthly report matches the cost system. Skip it and you&#8217;ll meet the spreadsheet.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help untangling this?</h2>



<p class="wp-block-paragraph">Most of the work in this area is not clicking through screens, it&#8217;s the plumbing and the decisions around it. That part I can help with:</p>



<ul class="wp-block-list">
<li>Writing the system-of-record map for your data classes before you commit to a second Oracle product, so cost and dates have one owner each</li>

<li>Designing and building integrations between P6, Unifier, Aconex, Textura and your ERP, including the field mapping and error handling nobody scopes</li>

<li>Hardening on-premise P6 EPPM or Unifier deployments that need to be reachable for a cloud integration, with proper TLS, network restrictions and audited access</li>

<li>Automating schedule and cost data extraction into a warehouse or BI layer, so reporting stops depending on someone manually exporting on a Friday</li>

<li>Building repeatable backup, restore-testing and disaster recovery for self-hosted Primavera estates</li>

<li>Second-opinion reviews on an implementation or integration design that already exists and is not behaving</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;re stuck on something concrete, send it over: an integration mapping spec, a Gateway error, a network diagram for an on-premise P6 that needs to talk to a cloud service. Easier to give you something useful when I can see the actual thing.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/primavera-p6-vs-unifier-vs-aconex/">Primavera P6 vs Unifier vs Aconex vs Textura: Which Oracle Product Do You Actually Need?</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>ALB vs API Gateway: Pick the Limits You Can Live With</title>
		<link>https://john-nessime.com/blog/devops/alb-vs-api-gateway/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 25 Aug 2026 09:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Networking]]></category>
		<category><![CDATA[API Gateway]]></category>
		<category><![CDATA[Application Load Balancer]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[JWKS]]></category>
		<category><![CDATA[JWT Validation]]></category>
		<category><![CDATA[Listener Rules]]></category>
		<category><![CDATA[Load Balancing]]></category>
		<category><![CDATA[OIDC]]></category>
		<category><![CDATA[Private Integration]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Web Application Firewall]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=550</guid>

					<description><![CDATA[<p>Choosing between an Application Load Balancer and API Gateway is not a feature comparison. It is a choice of hard limits, a billing curve and an authentication story. Here is how each one bills you, which constraints surface months later, what ALB's native JWT validation changes, and a decision procedure you can run in ten minutes.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/alb-vs-api-gateway/">ALB vs API Gateway: Pick the Limits You Can Live With</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said the reporting endpoint returns a 504 after about thirty seconds. The application log said the job finished in forty-one seconds and wrote a perfectly good response to a client that had already hung up. Nothing crashed, nothing scaled wrong, nobody deployed anything that week.</p>



<p class="wp-block-paragraph">The integration timeout had been sitting in front of that API since the day someone created it. It only became a bug when a customer&#8217;s dataset got big enough to cross it.</p>



<p class="wp-block-paragraph">That is the real shape of the ALB vs API Gateway decision. You&#8217;re not picking a feature list. You&#8217;re picking a set of hard limits, a billing curve and an authentication story, then living inside them. The failure mode that bites isn&#8217;t a bad Friday, it&#8217;s finding the constraint months later, once the thing in front of your app has become load-bearing.</p>



<p class="wp-block-paragraph">Below: how each one bills you, which limits surface late, what changed recently on the load balancer side, and a decision procedure.</p>



<h2 class="wp-block-heading">The old rule of thumb is shakier than it looks</h2>



<p class="wp-block-paragraph">The advice you find everywhere is: containers and instances behind an Application Load Balancer, serverless behind API Gateway. Wrong often enough to cost you. ALB takes Lambda functions as targets. API Gateway reaches containers through private integrations with an NLB, an ALB, or Cloud Map. Compute type barely narrows the field.</p>



<p class="wp-block-paragraph">What separates them is who&#8217;s calling and what needs enforcing before a request reaches your code. A browser session against your own web app is a different problem from a partner&#8217;s server calling a documented endpoint with a key you issued and a quota you promised. One needs routing and a session. The other needs a contract.</p>



<h2 class="wp-block-heading">Application Load Balancer: routing you own the position of</h2>



<p class="wp-block-paragraph">ALB is a Layer 7 router inside your VPC. You give it subnets and security groups, attach listeners, and write rules that match on host, path, header, query string, method or source IP, then forward, redirect, return a fixed response, or authenticate.</p>



<h3 class="wp-block-heading">Where it wins</h3>



<ul class="wp-block-list">
<li><strong>Long requests.</strong> The idle timeout is an attribute you set, defaulting to 60 seconds. Report generation and slow third-party calls become a number in Terraform instead of an architecture problem.</li>

<li><strong>Big bodies and streaming.</strong> Against instance, IP and container targets there&#8217;s no small payload ceiling to design around, and WebSocket upgrades pass through.</li>

<li><strong>Browser authentication.</strong> An HTTPS listener rule can run an <code>authenticate-oidc</code> or <code>authenticate-cognito</code> action before forwarding. The load balancer does the redirect dance, sets a session cookie, and hands your target the verified claims in a header. Your app reads a header instead of implementing OAuth.</li>
</ul>



<h3 class="wp-block-heading">Where it doesn&#8217;t</h3>



<ul class="wp-block-list">
<li><strong>No per-client rate limiting.</strong> No API keys with quotas attached. Rate limiting means AWS WAF rate-based rules in front, counting requests per source over a window. Blunter than a per-consumer plan, and a separate bill.</li>

<li><strong>No request validation or response caching.</strong> A malformed body reaches your handler and burns your compute. A cacheable GET is recomputed unless you put a CDN in front.</li>

<li><strong>Lambda targets are constrained.</strong> Request body capped at 1 MB, response JSON capped at 1 MB, WebSocket upgrades rejected with a 400. If your function returns a large document, that ceiling arrives as a 502 in production, not an error at deploy time.</li>

<li><strong>You pay for it while it sleeps.</strong> An hourly charge per load balancer whether or not a request arrives. Three environments, three floors.</li>
</ul>



<h2 class="wp-block-heading">Amazon API Gateway: the contract layer</h2>



<p class="wp-block-paragraph">API Gateway isn&#8217;t a load balancer with extra features. It&#8217;s a place to define what your API promises: which routes exist, who may call them, how often, and in what shape. That framing explains most of its trade-offs.</p>



<p class="wp-block-paragraph">Choosing it is really two decisions, because REST APIs and HTTP APIs are different products under one name.</p>



<ul class="wp-block-list">
<li><strong>REST APIs</strong> carry the full management surface: API keys, usage plans, per-client throttling, request validation, response caching, AWS WAF integration, the private endpoint type, execution logs and X-Ray tracing.</li>

<li><strong>HTTP APIs</strong> are the stripped-down, cheaper option. They add a built-in JWT authorizer, automatic deployments and Cloud Map private integrations. They offer none of the six management features listed above.</li>
</ul>



<p class="wp-block-paragraph">That second line is the one people get wrong. HTTP APIs aren&#8217;t &#8220;REST APIs but cheaper&#8221;, they&#8217;re a different product that happens to serve HTTP. Pick them to save money, then discover you need metered access, and you&#8217;re rebuilding usage plans in your own code.</p>



<h3 class="wp-block-heading">Where it wins</h3>



<ul class="wp-block-list">
<li><strong>You&#8217;re selling or governing access.</strong> API keys tied to usage plans, on REST APIs, are the only native answer to &#8220;customer B must not burn customer A&#8217;s capacity&#8221;.</li>

<li><strong>Rejecting bad requests before they cost you.</strong> Request validation against a model means a malformed payload never invokes your function. On per-invocation compute that&#8217;s a security boundary and a cost lever at once.</li>

<li><strong>Zero idle cost.</strong> No hourly floor, so a dev stage nobody calls is free. For genuinely spiky traffic this is often the whole argument, and there&#8217;s no network to design either.</li>
</ul>



<h3 class="wp-block-heading">Where it doesn&#8217;t</h3>



<ul class="wp-block-list">
<li><strong>The integration timeout.</strong> The default is 29 seconds. It can be raised, but AWS notes that going beyond 29 seconds may require reducing your Region-level throttle quota. You&#8217;re buying request duration with request rate.</li>

<li><strong>Payload ceilings.</strong> There&#8217;s a documented maximum payload size, well below what a file upload service wants. Large objects want presigned S3 URLs either way, but here the gateway forces that conversation early.</li>

<li><strong>Cost scales with request count.</strong> Per request plus data transfer. A health check hammered by a monitoring tool shows up directly on the line item.</li>

<li><strong>The account throttle is shared.</strong> The default steady-state rate and burst quota applies per Region across your APIs in that account. Raisable, but a shared ceiling by default rather than a per-API one.</li>
</ul>



<h2 class="wp-block-heading">What changed: the load balancer validates JWTs itself now</h2>



<p class="wp-block-paragraph">For years the clean argument for API Gateway on machine-to-machine APIs was token validation. If a partner&#8217;s service calls you with a bearer token, someone has to check signature, issuer and expiry, and you&#8217;d rather that not be a library duplicated in every service.</p>



<p class="wp-block-paragraph">ALB does this natively now. A rule on an HTTPS listener can carry a <code>jwt-validation</code> action that runs before any routing action. The load balancer fetches signing keys from a JWKS endpoint you configure, checks the signature, and requires <code>iss</code> and <code>exp</code>. It also validates <code>nbf</code> and <code>iat</code> when present, and you can configure up to ten additional claims. Valid tokens forward unchanged. Invalid ones never reach your VPC.</p>



<p class="wp-block-paragraph">This isn&#8217;t <code>authenticate-oidc</code>, which is built for humans in browsers and works by redirecting. JWT validation does no redirect. It inspects the token and returns a yes or no, which is what service-to-service traffic needs. Validation action first, routing action second, and order matters:</p>



<pre class="wp-block-code"><code>aws elbv2 create-rule 
  --listener-arn "$LISTENER_ARN" 
  --priority 10 
  --conditions Field=path-pattern,Values="/api/*" 
  --actions '[
    {
      "Type": "jwt-validation",
      "JwtValidationConfig": {
        "JwksEndpoint": "https://issuer.example.com/.well-known/jwks.json",
        "Issuer": "https://issuer.example.com"
      },
      "Order": 1
    },
    {
      "Type": "forward",
      "TargetGroupArn": "$TARGET_GROUP_ARN",
      "Order": 2
    }
  ]'</code></pre>



<p class="wp-block-paragraph">Two constraints first. Only RS256 is supported, and the JWKS endpoint has a maximum response size and key count. A provider publishing a large key set fails that check, and the failure mode is that requests stop reaching your targets. Fetch your provider&#8217;s JWKS document and look at it before committing.</p>



<p class="wp-block-paragraph">So &#8220;we need token validation at the edge&#8221; is no longer by itself a reason to reach for API Gateway. What the gateway still owns is everything downstream of that yes or no: keys, quotas, per-consumer throttles, shape validation, caching.</p>



<h2 class="wp-block-heading">How each one actually bills you</h2>



<p class="wp-block-paragraph">This is where the argument usually goes wrong, because people compare unit prices instead of curve shapes.</p>



<p class="wp-block-paragraph">API Gateway is a slope with no floor: per request, plus data out. Zero traffic costs nothing, ten times the traffic costs roughly ten times as much.</p>



<p class="wp-block-paragraph">ALB is a floor plus a stranger slope. An hourly rate per load balancer, plus capacity units measured across four dimensions at once:</p>



<ul class="wp-block-list">
<li>New connections per second</li>

<li>Active connections per minute</li>

<li>Processed bytes per hour</li>

<li>Rule evaluations per second, where the first ten processed rules per request are free</li>
</ul>



<p class="wp-block-paragraph">You&#8217;re charged on the highest of the four in a given hour, not the sum. That mechanic explains most surprises. Clients that open a fresh connection per call can bill you on new connections while your bandwidth graph looks flat. A listener with dozens of path rules can bill you on rule evaluations while everything else idles.</p>



<p class="wp-block-paragraph">Two details worth knowing: Lambda targets get a smaller processed-bytes allowance per capacity unit than instance, container and IP targets, so identical traffic consumes more units. Enabling mutual TLS reduces the active-connections allowance per unit.</p>



<p class="wp-block-paragraph">Before modelling anything, look at what you already consume:</p>



<pre class="wp-block-code"><code>aws cloudwatch get-metric-statistics 
  --namespace AWS/ApplicationELB 
  --metric-name ConsumedLCUs 
  --dimensions Name=LoadBalancer,Value=app/my-alb/0123456789abcdef 
  --start-time "$(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ)" 
  --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" 
  --period 3600 
  --statistics Sum Maximum</code></pre>



<p class="wp-block-paragraph">Run that against a representative week and you have the real input. For API Gateway, the equivalent is request count per stage. Across several accounts, a cost visibility tool like Vantage or CloudZero attributes this per environment faster than Cost Explorer, though disciplined tagging gets you most of the way for free.</p>



<p class="wp-block-paragraph">Rough shape: low steady traffic favours API Gateway because you skip the floor. High-volume, byte-heavy, long-lived-connection traffic favours ALB because the per-request slope disappears. Chatty traffic with tiny payloads is where the curves cross, and that one is worth calculating.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">A decision procedure for ALB vs API Gateway</h2>



<p class="wp-block-paragraph">Run these in order, stop at the first clear answer.</p>



<ol class="wp-block-list">
<li><strong>Does any request legitimately run past about thirty seconds?</strong> If yes and you can&#8217;t make it asynchronous, ALB. Raising the gateway timeout is possible but trades against your regional throttle quota.</li>

<li><strong>Do you need per-consumer quotas or metered access?</strong> If yes, API Gateway REST APIs. Nothing else here gives you keys and usage plans natively.</li>

<li><strong>Do requests or responses carry large payloads?</strong> If yes, ALB with instance or container targets. Both options constrain size, but the Lambda-target and gateway ceilings arrive sooner.</li>

<li><strong>Is traffic occasional, spiky, or mostly zero?</strong> If yes, API Gateway. An hourly floor for a few thousand requests a day is real cost with no matching benefit.</li>

<li><strong>Is the caller a browser session against your own app?</strong> If yes, ALB, with an authenticate action if you need login. The gateway&#8217;s management features are dead weight here.</li>

<li><strong>Is the caller another service with a bearer token and nothing more?</strong> Either works now. Decide on cost curve and payload shape, because JWT validation stopped being the tiebreaker.</li>
</ol>



<p class="wp-block-paragraph">Still torn? Choose the limits you&#8217;d rather explain to a customer in a year. Usually ALB for applications, API Gateway for products. Whichever you pick, alarm on the constraint rather than the symptom: integration latency approaching the timeout, or consumed capacity units trending up. Grafana Cloud or plain CloudWatch alarms both do the job. Then send a payload just over your ceiling in staging and confirm what the client actually receives.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<h3 class="wp-block-heading">&#8220;It&#8217;s serverless, so it&#8217;s cheaper&#8221;</h3>



<p class="wp-block-paragraph">Per-request billing is cheaper at low volume and more expensive at high volume. Arithmetic, not philosophy. The crossover depends on your request size and connection reuse, which is why the capacity-unit metric matters more than anyone&#8217;s price table.</p>



<h3 class="wp-block-heading">&#8220;We&#8217;ll put API Gateway in front of it later&#8221;</h3>



<p class="wp-block-paragraph">You can, through a private integration, and it&#8217;s a legitimate pattern. But &#8220;later&#8221; means a second hop, a second bill, a second set of timeouts and a second place to look during an incident. Adding it deliberately is fine. Assuming it&#8217;s free is not.</p>



<h3 class="wp-block-heading">&#8220;ALB has no rate limiting, so it&#8217;s insecure&#8221;</h3>



<p class="wp-block-paragraph">ALB has no <em>native</em> rate limiting, but WAF rate-based rules in front of it are a real control, and a source-based limit is often closer to what you wanted than a per-key quota. Cloudflare in front of the load balancer is another common answer when you already want its caching and bot controls. What ALB genuinely lacks is the per-consumer contract, which matters when consumers are customers and doesn&#8217;t when they&#8217;re your own front end.</p>



<p class="wp-block-paragraph">Check the pairing before you rely on it, though: WAF associates with ALB and with API Gateway REST APIs, but not with HTTP APIs. And the amount of request body WAF reads is configurable for some resource types and fixed for the load balancer. Anything past that window isn&#8217;t scanned.</p>



<h3 class="wp-block-heading">&#8220;The front door has to be an AWS service&#8221;</h3>



<p class="wp-block-paragraph">For one service with predictable traffic, a plain VPS running Nginx or Caddy does the same job with far less surface. Providers like Contabo or InterServer are reasonable homes for a small internal API. It won&#8217;t scale into a multi-service platform and it moves patching and certificate renewal onto you, but &#8220;which AWS service&#8221; is sometimes the wrong question.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I use API Gateway and ALB together?</h3>



<p class="wp-block-paragraph">Yes. API Gateway supports private integrations with an Application Load Balancer, so the gateway handles keys, quotas and validation while the load balancer routes inside the VPC. Sound pattern for a public product API in front of an existing internal platform. It also doubles the hops, the bills and the places a request can die, so use it when you need both jobs done.</p>



<h3 class="wp-block-heading">Which is cheaper, ALB or API Gateway?</h3>



<p class="wp-block-paragraph">Neither, universally. API Gateway has no idle cost and charges per request, so it wins at low or intermittent volume. ALB charges an hourly floor plus capacity units billed on the highest of four dimensions, so it wins once the per-request slope dominates. Pull your consumed capacity units and request counts and do the arithmetic on your own traffic.</p>



<h3 class="wp-block-heading">Can ALB do rate limiting?</h3>



<p class="wp-block-paragraph">Not natively. You get it by associating AWS WAF with the load balancer and using rate-based rules, which limit by source over a time window. If you need per-key quotas, that&#8217;s an API Gateway REST API usage plan, or logic you write yourself.</p>



<h3 class="wp-block-heading">Does ALB support JWT authentication now?</h3>



<p class="wp-block-paragraph">Yes, through a <code>jwt-validation</code> action on an HTTPS listener rule. It verifies the signature against a JWKS endpoint and checks issuer and expiry, plus optional additional claims. Note the RS256-only restriction and the JWKS size and key count limits before designing around it.</p>



<h3 class="wp-block-heading">Should I use HTTP APIs or REST APIs?</h3>



<p class="wp-block-paragraph">REST APIs if you need any of: API keys, usage plans, per-client throttling, request validation, response caching, WAF integration, or a private endpoint type. HTTP APIs when you need none of those and want the built-in JWT authorizer at a lower price. Migrating between them isn&#8217;t a config flag, so decide before you publish a URL.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">ALB vs API Gateway isn&#8217;t a question about serverless versus containers, and it stopped being a question about token validation. It&#8217;s a question about whether the thing in front of your application needs to enforce a contract with people outside your organisation.</p>



<p class="wp-block-paragraph">If it does, take the gateway and accept its timeout and payload ceilings as the price of keys, quotas and validation. If it doesn&#8217;t, take the load balancer and spend the saved effort on the network and the health checks. Either way, write down the three numbers that will eventually bite: timeout, payload ceiling, throttle quota. The 504 in that opening ticket wasn&#8217;t a bug. It was a decision nobody had written down.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second opinion on your front door?</h2>



<p class="wp-block-paragraph">Cheap to get right at the start, expensive to revisit. If you&#8217;re staring at this now, or already living with the wrong answer:</p>



<ul class="wp-block-list">
<li>Modelling ALB capacity units against API Gateway request counts on your actual traffic, so the comparison uses your numbers</li>

<li>Auditing an existing gateway or load balancer for the limits you&#8217;re closest to hitting: timeouts, payload ceilings, throttle quotas</li>

<li>Moving browser authentication off your application onto an ALB authenticate rule, or setting up JWT validation at the listener for service-to-service traffic</li>

<li>Designing the gateway plus private integration pattern when you need both layers, including where to terminate TLS and how to preserve client IPs</li>

<li>Building the Terraform or OpenTofu modules for either option, with listener rules, target groups, health checks and alarms in one place</li>

<li>Setting up alarms that fire on approaching limits rather than the 5XX that follows</li>
</ul>



<p class="wp-block-paragraph">Send me a listener rule dump, a stage configuration, or a week of consumed capacity unit metrics and I&#8217;ll tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/alb-vs-api-gateway/">ALB vs API Gateway: Pick the Limits You Can Live With</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Backfilling Historical API Data into S3 Without Silent Gaps</title>
		<link>https://john-nessime.com/blog/devops/backfill-historical-api-data-into-s3/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 24 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Backfill]]></category>
		<category><![CDATA[Boto3]]></category>
		<category><![CDATA[Conditional Requests]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Cursor Pagination]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Exponential Backoff]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Parquet]]></category>
		<category><![CDATA[Partition Projection]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[S3 Inventory]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=268</guid>

					<description><![CDATA[<p>A backfill that exits zero can still be missing a week of data, and nothing will tell you. This is a practical guide to the failure families behind silent gaps: pagination drift under a mutating source, retries that duplicate pages, prefix layouts designed for writes instead of reads, the seam where backfill meets live ingest, and the storage class rules that make mistakes expensive. Includes deterministic key derivation, S3 conditional writes, Athena partition projection, and a per-window manifest pattern that turns completeness into something you can query.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/backfill-historical-api-data-into-s3/">Backfilling Historical API Data into S3 Without Silent Gaps</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The backfill exits zero. Every window processed, nothing red in the logs, the object count in the bucket looks about right for the range. You close the ticket. Weeks later an analyst asks why the chart dips for one week in the middle of the historical range, and nobody can answer, because as far as every system involved is concerned, nothing failed.</p>



<p class="wp-block-paragraph">That&#8217;s the shape of the problem. When you backfill historical API data into S3, the failure to worry about isn&#8217;t the one that throws. It&#8217;s the one that writes fewer records than it should and reports success anyway. Bytes transferred look plausible. Nothing goes red. The data is quietly wrong, and you find out when someone downstream trusts it.</p>



<p class="wp-block-paragraph">This post covers the failure families that produce that outcome and the design decisions that make them impossible rather than merely unlikely: pagination that drifts under a mutating source, retries that write the same page twice under different keys, a prefix layout chosen while thinking about writes instead of reads, the seam where the backfill meets the live pipeline, and the storage class trap that makes a cheap-looking archive expensive to fix.</p>



<h2 class="wp-block-heading">Why a backfill that finishes cleanly can still be wrong</h2>



<p class="wp-block-paragraph">A live pipeline has a natural error signal. Data stops arriving, a lag metric climbs, someone notices. A backfill has none of that. It runs once, against a range nobody is watching in real time, and then it&#8217;s over. The only evidence it leaves behind is a set of objects in a bucket.</p>



<p class="wp-block-paragraph">So the question you need to answer months later isn&#8217;t &#8220;did the job succeed?&#8221; It&#8217;s &#8220;is this window complete, and how do I know?&#8221; If your design can&#8217;t answer that from artifacts on disk, you don&#8217;t have a backfill. You have a large amount of data of unknown provenance.</p>



<h2 class="wp-block-heading">Failure family 1: the source moved while you paginated</h2>



<p class="wp-block-paragraph">This one bites hardest and shows up least. Most REST APIs paginate historical queries with an offset or page number. You ask for page 1, page 2, page 3. Meanwhile the underlying table is still accepting writes.</p>



<p class="wp-block-paragraph">If a record is inserted into the middle of the result set between your page 4 and page 5 requests, everything after it shifts down by one. The row that would have started page 5 is now the last row of page 4, which you already fetched. You request page 5 and that record is gone. Not errored. Gone. Insertions cause skips, deletions cause duplicates, and the job still exits zero.</p>



<h3 class="wp-block-heading">What to do instead</h3>



<ul class="wp-block-list"><li><strong>Prefer keyset or cursor pagination.</strong> If the API offers an opaque cursor or a &#8220;records after ID X&#8221; parameter, use it. The read position is anchored to a value rather than a count, so inserts elsewhere in the set can&#8217;t shift it.</li><li><strong>Sort by an immutable column.</strong> If you&#8217;re stuck with offsets, sort by something that never changes after insert, usually the primary key or the creation timestamp. Sorting by a last-modified column while records are being modified is the worst case: rows physically move within the ordering as you walk it.</li><li><strong>Bound every request on both ends.</strong> Don&#8217;t ask for &#8220;everything before now.&#8221; Ask for a closed interval with an upper bound already in the past. A closed window can&#8217;t grow while you read it.</li><li><strong>Know which timestamp the filter applies to.</strong> Filtering on created-at gives you records that came into existence in that window. Filtering on updated-at gives you records touched in it, which for a mutable source is a different set entirely.</li></ul>



<p class="wp-block-paragraph">That last point causes arguments. Backfill on created-at, then run incremental syncs on updated-at, and your historical range never picks up edits made to old records. That may be correct for your use case. It is not correct by default, and it&#8217;s worth writing down which one you chose.</p>



<h2 class="wp-block-heading">Failure family 2: the retry that wrote the same page twice</h2>



<p class="wp-block-paragraph">Backfills retry. They have to, because you&#8217;re hammering an API for hours and something will time out. The problem is what the retry writes.</p>



<p class="wp-block-paragraph">If your object key contains a UUID, a run identifier or a wall-clock timestamp, a retried page produces a second object with a different key and the same records inside. You now have duplicates that listing will never reveal, because both objects are legitimately present and neither is obviously a copy. Derive the key from the request instead of the run, so the same window plus the same cursor always produces the same key and a retry overwrites rather than accumulates.</p>



<pre class="wp-block-code"><code>import hashlib
import boto3
from botocore.config import Config
from botocore.exceptions import ClientError

s3 = boto3.client(
    "s3",
    config=Config(
        retries={"total_max_attempts": 8, "mode": "standard"},
        max_pool_connections=32,
    ),
)

def page_key(dataset, window, cursor):
    # Deterministic: the same page always maps to the same key.
    token = hashlib.sha256(cursor.encode()).hexdigest()[:16]
    return f"raw/{dataset}/dt={window}/page-{token}.json.gz"

def put_once(bucket, key, body):
    try:
        s3.put_object(Bucket=bucket, Key=key, Body=body, IfNoneMatch="*")
        return "written"
    except ClientError as err:
        if err.response["Error"]["Code"] == "PreconditionFailed":
            return "already_present"
        raise</code></pre>



<p class="wp-block-paragraph">Two things in there are worth explaining rather than copying.</p>



<p class="wp-block-paragraph"><code>IfNoneMatch="*"</code> is a conditional write. S3 accepts the PUT only if no object exists at that key and returns a 412 <code>PreconditionFailed</code> otherwise. It&#8217;s supported on <code>PutObject</code> and <code>CompleteMultipartUpload</code>. That turns &#8220;did I already write this page?&#8221; into a single request with no read-before-write race. If two workers hit the same key concurrently, S3 can instead return a 409 <code>ConditionalRequestConflict</code>, which AWS documents as retryable, so handle the two codes differently: retry on 409, skip on 412.</p>



<p class="wp-block-paragraph">The same thing from the CLI, useful for testing the behaviour by hand:</p>



<pre class="wp-block-code"><code>aws s3api put-object 
  --bucket your-bucket 
  --key raw/orders/dt=WINDOW/page-abc123.json.gz 
  --body page.json.gz 
  --if-none-match "*"</code></pre>



<p class="wp-block-paragraph">The <code>retries</code> block matters for a reason that trips people up. In botocore, <code>max_attempts</code> means different things depending on where you set it: in the AWS config file or the environment variable it counts total requests including the first, but in a <code>Config</code> object it counts retries on top of the first. <code>total_max_attempts</code> exists only in a <code>Config</code> object and always means total, so using it removes the ambiguity. There&#8217;s also an <code>adaptive</code> mode that layers client-side rate limiting on top of <code>standard</code>; AWS still labels it experimental, so I reach for <code>standard</code> plus my own concurrency cap first.</p>



<h2 class="wp-block-heading">Failure family 3: a layout designed for writing, not reading</h2>



<p class="wp-block-paragraph">The prefix structure you pick during the backfill is the one you&#8217;re stuck with. Reorganising later means rewriting every object, and copy requests aren&#8217;t free.</p>



<h3 class="wp-block-heading">Partition on event time, not ingestion time</h3>



<p class="wp-block-paragraph">The tempting layout is the date you ran the job. It&#8217;s easy, it&#8217;s monotonic, and it&#8217;s useless. Every query anyone writes against historical data filters on when the thing happened, not when you happened to fetch it. If your partitions are ingestion dates, every one of those queries becomes a full scan. Use Hive-style keys so query engines prune without extra configuration:</p>



<pre class="wp-block-code"><code>s3://your-bucket/raw/orders/dt=YYYY-MM-DD/page-&lt;token&gt;.json.gz
s3://your-bucket/curated/orders/dt=YYYY-MM-DD/part-&lt;n&gt;.parquet</code></pre>



<p class="wp-block-paragraph">Keep the raw landing zone and the curated zone separate. Raw is whatever the API gave you, unmodified, so you can re-derive everything if your parsing logic turns out to be wrong. Curated is columnar and typed. Merging them saves nothing and costs you the ability to reprocess.</p>



<h3 class="wp-block-heading">Don&#8217;t let the write path throttle itself</h3>



<p class="wp-block-paragraph">S3 documents at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix, with no limit on the number of prefixes. The word doing the work is <em>partitioned</em>. That rate applies once S3 has split the keyspace out for that prefix, and AWS is explicit that the scaling is gradual rather than instant. During the ramp you can see 503 <code>SlowDown</code> responses even when your sustained rate looks well inside the documented ceiling.</p>



<p class="wp-block-paragraph">For a backfill this is mostly good news, since date-partitioned keys spread writes across many prefixes as you work through the range. It becomes a problem when you parallelise <em>within</em> a single day and drive thousands of writes per second into one <code>dt=</code> prefix. Two mitigations: back off with jitter instead of retrying 503s immediately, and shard workers across windows rather than within them.</p>



<h3 class="wp-block-heading">Register partitions without a scan</h3>



<p class="wp-block-paragraph">A backfill can create thousands of partitions at once. <code>MSCK REPAIR TABLE</code> handles that badly. AWS documents that it only adds partitions and never removes them, that it can take a long time on large tables, and that it can time out leaving the catalog partially updated. Partial success with no error is exactly the failure mode this whole post is about.</p>



<p class="wp-block-paragraph">Partition projection sidesteps the catalog. Athena computes the partition list from table properties instead of looking it up, so a backfilled range is queryable the moment the objects exist.</p>



<pre class="wp-block-code"><code>TBLPROPERTIES (
  'projection.enabled'          = 'true',
  'projection.dt.type'          = 'date',
  'projection.dt.format'        = 'yyyy-MM-dd',
  'projection.dt.range'         = 'START_DATE,NOW',
  'projection.dt.interval'      = '1',
  'projection.dt.interval.unit' = 'DAYS',
  'storage.location.template'   = 's3://your-bucket/curated/orders/dt=${dt}/'
)</code></pre>



<p class="wp-block-paragraph">Replace <code>START_DATE</code> with the earliest date in your backfill. The trade-off is worth stating plainly: projection generates every partition in the range whether or not data exists there, so an over-wide range on a sparse dataset wastes planning effort. And if the S3 layout ever stops matching <code>storage.location.template</code>, queries return nothing rather than erroring. Set the range to your actual data, not to a comfortable margin.</p>



<h2 class="wp-block-heading">Failure family 4: the seam between backfill and live pipeline</h2>



<p class="wp-block-paragraph">Two processes write to the same table: a backfill walking backwards through history, and a live ingest walking forwards. They meet somewhere, and that meeting point is where records go missing.</p>



<p class="wp-block-paragraph">The instinct is to make the handoff exact. Backfill up to timestamp T, start streaming from T. Don&#8217;t. Clock skew between your runner and the API, records that arrive late at the source, and rounding in the API&#8217;s own filtering will all conspire to open a hole at exactly T.</p>



<p class="wp-block-paragraph">Overlap deliberately instead. Run the backfill past the point where live ingest started, by a margin generously larger than any plausible source lag, then deduplicate on a natural key during curation. Duplicates you can remove with a query. Gaps you can only fix by going back to the API, and by then the retention window may have closed.</p>



<h2 class="wp-block-heading">Failure family 5: the storage class you can&#8217;t undo</h2>



<p class="wp-block-paragraph">Historical data feels like archive data, so the instinct is to write it straight into a cold class or transition it immediately. Read the mechanics first, because the billing rules for cold classes are structural and pro-rated against you.</p>



<ul class="wp-block-list"><li>S3 Standard-IA and One Zone-IA carry a 30-day minimum storage duration and a 128 KB minimum billable object size.</li><li>Glacier Instant Retrieval and Glacier Flexible Retrieval carry a 90-day minimum duration; Deep Archive carries 180 days. Delete, overwrite or transition earlier and you&#8217;re charged the remainder.</li><li>Objects in Glacier Flexible Retrieval and Deep Archive are billed for 40 KB of extra per-object metadata on top of the object itself.</li></ul>



<p class="wp-block-paragraph">Read those together and the conclusion is uncomfortable. A backfill that writes millions of small JSON pages into an archive class pays a size floor on every one of them, plus metadata overhead, plus a minimum duration that makes fixing the mistake expensive. Those pages are also the objects you&#8217;re most likely to want to reprocess, which is exactly the overwrite that triggers the early-deletion charge.</p>



<p class="wp-block-paragraph">What I&#8217;d actually do: land raw pages in Standard, compact them into larger Parquet files in the curated zone, verify completeness, and only then apply a lifecycle rule to the raw prefix. Compaction is what makes cold storage sane, because it turns a size floor per tiny object into a rounding error on a large one. It also fixes the small-file problem in Athena, which is a separate cost line.</p>



<p class="wp-block-paragraph">One more cost note: if your backfill runs from EC2 in a private subnet, every byte pulled from the API and pushed to S3 may route through a NAT gateway with per-GB processing charges. A gateway VPC endpoint for S3 removes the S3 half of that. It&#8217;s a five-minute change people usually discover after the bill.</p>



<h2 class="wp-block-heading">Proving the backfill is complete</h2>



<p class="wp-block-paragraph">Here&#8217;s the part most backfills skip, and the part that makes everything above verifiable rather than hopeful. Write a manifest object per window, alongside the data, describing what that window contains.</p>



<pre class="wp-block-code"><code>{
  "dataset": "orders",
  "window": "&lt;dt value&gt;",
  "filter_field": "created_at",
  "range_start_inclusive": "&lt;iso8601&gt;",
  "range_end_exclusive": "&lt;iso8601&gt;",
  "pages_written": "&lt;int&gt;",
  "records_written": "&lt;int&gt;",
  "source_reported_total": "&lt;int or null&gt;",
  "last_cursor": "&lt;opaque token&gt;",
  "schema_fingerprint": "&lt;sha256 of sorted field names&gt;"
}</code></pre>



<p class="wp-block-paragraph">The manifest turns completeness into something you can query. A window with no manifest was never finished. A window whose <code>records_written</code> disagrees with <code>source_reported_total</code> is suspect. A window whose <code>schema_fingerprint</code> differs from its neighbours tells you the API changed shape partway through your range, which is otherwise close to undetectable.</p>



<p class="wp-block-paragraph">Write the manifest last, after every page for that window is confirmed in S3. A manifest written optimistically at the start is worse than no manifest at all, because it asserts a completeness nobody checked.</p>



<h2 class="wp-block-heading">Troubleshooting a backfill that already ran</h2>



<ol class="wp-block-list"><li><strong>Count objects per window and look at the shape.</strong> Use S3 Inventory rather than a live <code>ListObjectsV2</code> walk on a large bucket. Plot objects per <code>dt=</code> partition. Real data has a rhythm, weekday peaks and weekend troughs. A window that breaks the rhythm without a business reason is your gap.</li><li><strong>Reconcile record counts against the source.</strong> Many APIs return a total count on the first page of a filtered query. Pull that for a sample of windows and compare with what&#8217;s in S3. If you kept manifests, this is one query.</li><li><strong>Check for duplicate natural keys within a partition.</strong> A <code>GROUP BY</code> on the record ID with <code>HAVING COUNT(*) &gt; 1</code> finds retry duplicates immediately. Duplicates clustered at window boundaries mean pagination drift; duplicates spread evenly mean your keys weren&#8217;t deterministic.</li><li><strong>Look at boundary hours, not whole days.</strong> Cursor drift and clock skew concentrate at the edges of each request window. Aggregate to the hour and the missing slice usually becomes obvious even when daily totals look fine.</li><li><strong>Verify the layout still matches the table definition.</strong> If you&#8217;re using partition projection and a query returns zero rows, the first suspect is <code>storage.location.template</code> not matching the actual prefix, trailing slashes included.</li></ol>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list"><li>Treating an exit code of zero as evidence of completeness.</li><li>Putting a run ID or timestamp in the object key, which makes retries additive instead of idempotent.</li><li>Partitioning by ingestion date because it&#8217;s what the job naturally knows.</li><li>Making the backfill and live-ingest boundary exact instead of overlapping.</li><li>Writing millions of small objects straight into an archive class and meeting the minimum object size and duration rules afterwards.</li><li>Running <code>MSCK REPAIR TABLE</code> over thousands of new partitions and trusting that it completed.</li><li>Backfilling on a last-modified field while assuming you captured creation events.</li><li>Sharding workers within a single day, concentrating writes on one prefix and inviting 503s.</li></ul>



<h2 class="wp-block-heading">Best practices when you backfill historical API data into S3</h2>



<ul class="wp-block-list"><li>Make every window independently re-runnable. If re-running one day is routine rather than scary, you&#8217;ll actually fix problems when you find them.</li><li>Derive object keys deterministically from window plus cursor, and use conditional writes so a retry is a no-op.</li><li>Bound every API request on both ends, and never let the upper bound be &#8220;now&#8221;.</li><li>Write raw responses untouched before parsing anything. Parsing bugs are common; re-fetching a closed API window often isn&#8217;t possible.</li><li>Emit a manifest per window, written last, and reconcile it against source counts.</li><li>Compact to Parquet before applying any lifecycle transition.</li><li>Run the job somewhere it can survive for hours. A long backfill on a laptop over hotel wifi is a bad plan; a small dedicated VPS from a provider like Contabo or InterServer, or a spot-priced EC2 instance, costs little and won&#8217;t sleep halfway through.</li><li>Watch progress somewhere other than the terminal. Pushing records-per-window into CloudWatch or Grafana gives you a curve you can eyeball, and an unusually flat stretch is a gap you can catch while the source data is still retrievable.</li></ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Should I write raw JSON or convert to Parquet during the backfill?</h3>



<p class="wp-block-paragraph">Both, in two zones. Write the raw response body first so the fetch step has no dependency on your schema assumptions, then convert as a separate job. Converting inline couples an expensive, rate-limited API walk to a parsing step that will eventually throw on an unexpected field, and you don&#8217;t want to re-fetch years of data because of a type error.</p>



<h3 class="wp-block-heading">How large should each object be?</h3>



<p class="wp-block-paragraph">In the raw zone, one object per API page is fine and keeps idempotency simple. In the curated zone, aim for files in the tens to low hundreds of megabytes. Query engines pay a fixed overhead per file, so thousands of tiny Parquet files scan slower and cost more than a few large ones holding identical data.</p>



<h3 class="wp-block-heading">Can I run the backfill in parallel across date windows?</h3>



<p class="wp-block-paragraph">Yes, and that&#8217;s the right axis, because each window is independent and lands on its own prefix. The binding constraint is almost always the source API&#8217;s rate limit rather than S3. Add a global concurrency cap and honour any documented limit or <code>Retry-After</code> header. Getting your credentials throttled or suspended mid-backfill is a worse outcome than finishing slowly.</p>



<h3 class="wp-block-heading">Do I need AWS Glue or Airflow for this?</h3>



<p class="wp-block-paragraph">Not to start. A single script that takes a window as an argument, writes pages and a manifest, and is safe to re-run covers most backfills. Orchestration earns its place when you need retries across hundreds of windows, dependency ordering, or a visible run history. Glue and Step Functions fit AWS-native stacks; Airflow gives you more control and more to operate. Pick after the script works, not before.</p>



<h3 class="wp-block-heading">What if the API caps how far back I can query?</h3>



<p class="wp-block-paragraph">Then the retention boundary is your real deadline and it should drive priority. Fetch the oldest reachable data first, not the most recent, because the newest data is still available tomorrow and the oldest may not be. If the vendor offers a bulk export or reporting endpoint separate from the standard API, it&#8217;s usually faster and less rate-limited than paginating.</p>



<h3 class="wp-block-heading">Does any of this change with S3-compatible storage instead of AWS?</h3>



<p class="wp-block-paragraph">The layout, idempotency and manifest patterns carry over unchanged to Cloudflare R2, Backblaze B2, MinIO and similar. What varies is API-level feature support. Conditional writes with <code>If-None-Match</code>, request-rate behaviour and storage class semantics differ by provider, so verify the specific behaviour you&#8217;re depending on against that provider&#8217;s documentation rather than assuming parity with S3.</p>



<h3 class="wp-block-heading">How do I know the backfill is actually finished?</h3>



<p class="wp-block-paragraph">When every expected window has a manifest, every manifest&#8217;s record count reconciles against the source, and a duplicate check on natural keys returns nothing. Not when the script exits.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A backfill isn&#8217;t finished when the job exits. It&#8217;s finished when you can prove the range is complete without re-reading the source. Deterministic keys, conditional writes, closed windows, manifests and deliberate overlap all exist to make that proof cheap.</p>



<p class="wp-block-paragraph">If you&#8217;re about to backfill historical API data into S3, spend the extra hour on the key derivation and the manifest before you start the run. The alternative is learning about a gap from a chart, months later, with the source retention window already closed behind you.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a hand with a backfill?</h2>



<p class="wp-block-paragraph">This is a large part of what I do. If you&#8217;re staring at a historical range and an API with a rate limit, I can help with:</p>



<ul class="wp-block-list"><li>Designing an idempotent, re-runnable backfill job with deterministic keys and per-window manifests, so a failed day is a one-command fix.</li><li>Choosing a partition layout that serves your actual query patterns, and setting up Athena or Glue tables with partition projection so thousands of backfilled partitions become queryable without catalog scans.</li><li>Auditing a backfill that already ran, reconciling record counts against the source and finding the gaps and duplicates before your dashboards do.</li><li>Handling awkward source APIs: offset-only pagination, aggressive rate limits, undocumented result caps, schema drift partway through a historical range.</li><li>Compaction and lifecycle design, so cold storage actually saves money instead of hitting minimum object size and minimum duration charges.</li><li>Wiring the seam between the backfill and your live pipeline so there&#8217;s no gap at the handover, plus the monitoring that would have caught it.</li></ul>



<p class="wp-block-paragraph">Send me something concrete: a sample API response, your current key layout, or the object counts per partition. That&#8217;s usually enough to spot where the gap is.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/backfill-historical-api-data-into-s3/">Backfilling Historical API Data into S3 Without Silent Gaps</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a Salesforce AI Assistant on Amazon Bedrock Without Leaking Your CRM</title>
		<link>https://john-nessime.com/blog/devops/salesforce-ai-assistant-amazon-bedrock/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 18 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[AgentCore]]></category>
		<category><![CDATA[Agentforce]]></category>
		<category><![CDATA[AI Agents]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[API Gateway]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Bedrock Guardrails]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Cognito]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[External Client Apps]]></category>
		<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[Identity Propagation]]></category>
		<category><![CDATA[MCP]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Salesforce]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=211</guid>

					<description><![CDATA[<p>The demo works, then someone sees records they shouldn't. Nothing errors. Here's how to build a Salesforce AI assistant with Amazon Bedrock that respects your sharing model: which doors AWS and Salesforce just closed, why identity propagation is the failure mode that bites, and how to shape the tool surface so the agent can't wander.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-ai-assistant-amazon-bedrock/">Building a Salesforce AI Assistant on Amazon Bedrock Without Leaking Your CRM</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The demo went fine. Someone from the support team asked the assistant which accounts looked at risk this quarter, and it answered. Good answer, too. Right opportunities, right open cases, sensible summary.</p>



<p class="wp-block-paragraph">Then somebody noticed that the person who asked couldn&#8217;t see half those opportunities in the Salesforce UI. Different role, different branch of the hierarchy. They saw them through the assistant.</p>



<p class="wp-block-paragraph">Nothing errored. No 403, no exception in the logs, no alert. The integration user had access, so the API call succeeded, so the model got the records, so it summarised them. Every layer did exactly what it was configured to do. That is the failure mode that should worry you most when you build a Salesforce AI assistant with Amazon Bedrock: it is silent, it is correct-looking, and you will not find it in a load test.</p>



<p class="wp-block-paragraph">This post covers what actually goes wrong when you wire a large language model into a CRM: which door AWS closed on the entry path, whose permissions your agent is really running under, how to shape the tool surface so the model can&#8217;t wander, and what happens to your org&#8217;s API budget when an agent starts looping. There&#8217;s a build walkthrough, but the build is the easy part.</p>



<h2 class="wp-block-heading">Two doors closed while you were reading the old tutorial</h2>



<p class="wp-block-paragraph">Most of what ranks for this topic was written against an architecture you can no longer build in a fresh account. Both vendors changed the front door within a few months of each other, and the tutorials haven&#8217;t caught up. Check this before you follow anything, including this post.</p>



<h3 class="wp-block-heading">On the AWS side: Bedrock Agents is now Bedrock Agents Classic</h3>



<p class="wp-block-paragraph">AWS renamed the original Bedrock Agents service to Bedrock Agents Classic and put it into maintenance mode. It stopped accepting new customers on 30 July 2026. This one is worth stating with the date attached, because the date is the whole point.</p>



<p class="wp-block-paragraph">The mechanics matter more than the branding:</p>



<ul class="wp-block-list">
<li>Only <code>CreateAgent</code> and <code>InvokeInlineAgent</code> are restricted. Every other API stays available to everyone.</li>

<li>Allowlisting is per AWS account, based on whether that account used Bedrock Agents in the previous twelve months. Multi-account estates get a split result: your old sandbox works, your new production account doesn&#8217;t.</li>

<li>A non-allowlisted account calling <code>CreateAgent</code> gets an <code>AccessDeniedException</code> with HTTP 403, not a quota error. If you&#8217;re debugging IAM and seeing 403s, this is a plausible cause that no policy change will fix.</li>

<li>There is no exception process. AWS documents this plainly.</li>

<li>The model catalogue inside Classic is frozen. Bedrock itself keeps getting new models; the Classic orchestration layer does not.</li>
</ul>



<p class="wp-block-paragraph">The replacement is Amazon Bedrock AgentCore. It splits into a managed harness (config-driven, closest to the old experience) and code-defined agents on the AgentCore runtime (bring Strands, LangGraph, CrewAI, or your own loop). For a Salesforce assistant, the harness is usually enough, and you can move to code later without re-plumbing the tools.</p>



<p class="wp-block-paragraph">Existing Classic agents keep running. There&#8217;s no announced end-of-life. But a frozen model catalogue sets a ceiling on the thing, and if your team can&#8217;t recreate the agent in a new account, your disaster recovery story has a hole in it.</p>



<h3 class="wp-block-heading">On the Salesforce side: Connected Apps gave way to External Client Apps</h3>



<p class="wp-block-paragraph">Salesforce blocked creation of new Connected Apps through both UI and API, with package installation as the exception. External Client Apps are the replacement. Existing Connected Apps keep working and keep their OAuth flows, so this is a wall on new creation rather than a sunset.</p>



<p class="wp-block-paragraph">The practical symptom is that the <em>New Connected App</em> button is greyed out with no explanation. Every AWS-to-Salesforce tutorial that opens with &#8220;first, create a Connected App&#8221; stops working right there.</p>



<p class="wp-block-paragraph">One difference matters for agent work specifically: External Client Apps do not support the username-password OAuth flow at all. If your integration design assumed you&#8217;d stash a service account username and password somewhere and call it a day, that design is gone. Which is a good thing, and it forces the question the next section is about.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: whose permissions is the assistant using?</h2>



<p class="wp-block-paragraph">This is the one from the opening, and it&#8217;s the reason most of these projects should slow down before they ship.</p>



<p class="wp-block-paragraph">Salesforce&#8217;s security model is genuinely good. An OAuth token runs in a user context, and object permissions, field-level security, and sharing rules all still apply to API calls. A token with the <code>api</code> scope does not bypass CRUD, FLS, or sharing. People hear that and relax.</p>



<p class="wp-block-paragraph">The catch is the phrase &#8220;a user context.&#8221; Which user?</p>



<p class="wp-block-paragraph">If you authenticate the gateway to Salesforce with client credentials, every request from every person runs as one integration user. Salesforce enforces that user&#8217;s permissions perfectly. It has no idea a different human asked the question. You&#8217;ve built what&#8217;s sometimes called a multiplexor: one licence fronting many people, with all the real access control now living in your application layer instead of the platform&#8217;s.</p>



<p class="wp-block-paragraph">At that point the sharing model is decoration. A prompt that talks the model into calling a query tool with a wider filter returns data the asker was never entitled to, and Salesforce logs it as a legitimate read by a legitimate user. There&#8217;s nothing to alert on, because nothing went wrong.</p>



<h3 class="wp-block-heading">Two-legged versus three-legged, and when each is honest</h3>



<p class="wp-block-paragraph">AgentCore Gateway supports both client credentials (two-legged) and authorization code (three-legged) flows for reaching downstream targets, and handles ingress and egress authorisation separately. That distinction is the control you need.</p>



<ul class="wp-block-list">
<li><strong>Two-legged is honest when the data is genuinely org-wide.</strong> Product catalogues, published knowledge articles, case deflection content. Nobody&#8217;s entitlement varies, so a single service identity represents the truth.</li>

<li><strong>Three-legged is the only correct answer for record data.</strong> Opportunities, accounts, contacts, custom objects with sharing rules on them. The token is minted for the actual human, Salesforce applies their sharing, and the answer is bounded by what they could have looked up themselves.</li>
</ul>



<p class="wp-block-paragraph">Three-legged costs you something real, and I&#8217;d rather say so than pretend it&#8217;s free. You need per-user consent, token storage, refresh handling, and a way to deal with a user whose token has expired mid-conversation. AgentCore Identity provides a token vault and handles the consent flow, including binding the authorisation URL to the session that started it so a shared link can&#8217;t be completed by someone else. It&#8217;s still more moving parts than a single secret in Secrets Manager.</p>



<p class="wp-block-paragraph">Pay the cost anyway for record data. The alternative is a system whose blast radius is &#8220;everything the integration user can see,&#8221; discovered by a customer or an auditor rather than by you.</p>



<p class="wp-block-paragraph">If you already run Okta, Auth0, or Ping as your identity provider, Gateway can sit in front of them for inbound auth rather than you standing up a parallel Cognito user pool. Reusing the IdP you already audit is worth more here than saving a config file.</p>



<h2 class="wp-block-heading">Failure family two: the shape of the tool surface</h2>



<p class="wp-block-paragraph">The second mistake is giving the model a tool called <code>run_soql</code> that accepts an arbitrary query string. It demos beautifully. Ask anything, get anything.</p>



<p class="wp-block-paragraph">You&#8217;ve now made the model your query planner and your authorisation layer at once. Every safety property of the system depends on prompt text. Guardrails help with content, not with whether a generated <code>WHERE</code> clause happened to be too generous.</p>



<p class="wp-block-paragraph">Narrow tools are the fix, and they&#8217;re boring on purpose. <code>get_open_cases_for_account(account_id)</code> beats <code>run_soql(query)</code> because the query shape is fixed in your code, the model only supplies an identifier, and you can log and rate-limit each tool independently. When something goes wrong you&#8217;re reading a parameter, not parsing SOQL to work out what the model asked for.</p>



<p class="wp-block-paragraph">AgentCore Gateway takes OpenAPI specs, Smithy models, and Lambda functions as tool input types, and exposes them to the agent as MCP tools. It also has one-click integrations for several SaaS products including Salesforce. Practically that gives you three routes:</p>



<ol class="wp-block-list">
<li><strong>The built-in Salesforce integration</strong> for the fastest path to something working. Good for a proof of concept. Look closely at which operations it exposes before you take it further, because breadth is exactly what you don&#8217;t want.</li>

<li><strong>An OpenAPI target</strong> describing a small purpose-built API in front of Salesforce. This is the one I reach for first. You control the surface completely, and the spec doubles as the tool description the model reads.</li>

<li><strong>Lambda targets</strong> when a tool needs to combine several Salesforce calls, or talk to something else at the same time.</li>
</ol>



<p class="wp-block-paragraph">MCP server targets are also supported, with the gateway aggregating tools from multiple targets into a single <code>tools/list</code> response. If you&#8217;re running your own MCP server as the Salesforce facade, it needs a stable HTTPS endpoint the gateway can reach; a small VPS from a provider like InterServer or Hetzner is plenty for the development and staging copies, with the production one behind whatever your standard ingress is.</p>



<h2 class="wp-block-heading">Failure family three: the write path</h2>



<p class="wp-block-paragraph">Reads leak. Writes destroy. They deserve different treatment and usually get the same treatment, which is how you end up with three hundred opportunities silently restaged.</p>



<p class="wp-block-paragraph">Start read-only and stay there longer than feels necessary. When you do add writes, the constraints that actually help:</p>



<ul class="wp-block-list">
<li><strong>One record per call.</strong> No bulk update tool. If the model wants to change twenty records it makes twenty calls, each individually logged and individually rate-limited. Slow is a feature here.</li>

<li><strong>Separate the write identity.</strong> A distinct External Client App and permission set for writes means you can revoke the write path at 2am without taking the assistant offline.</li>

<li><strong>Human confirmation on anything irreversible.</strong> The AgentCore harness supports inline function tools that pause the agent and hand control back to your client code, equivalent to return-of-control. That pause is where a person says yes.</li>

<li><strong>Stamp the source.</strong> A custom field recording that a change came from the assistant, with the session and actor ID, is the difference between a ten-minute rollback and an afternoon of forensics.</li>
</ul>



<p class="wp-block-paragraph">Salesforce field history tracking is on a limited number of fields per object. If the field your assistant writes to isn&#8217;t tracked, you have no before-value. Check that before go-live, not after.</p>



<h2 class="wp-block-heading">Failure family four: what happens under load</h2>



<p class="wp-block-paragraph">Salesforce meters API requests against a daily org-wide allocation that depends on your edition and licence count. Your assistant does not consume that budget the way a nightly sync does.</p>



<p class="wp-block-paragraph">A single conversational turn can be several tool calls. A model that doesn&#8217;t get a clean answer will often retry with a different filter. Ten support agents having ten conversations each becomes a request pattern nobody modelled, and the org-wide limit means the integration that breaks first might be your ETL job, your marketing platform, or your billing sync. Not the assistant. That&#8217;s what makes it hard to diagnose.</p>



<p class="wp-block-paragraph">Three things that help, in order of how much they buy you:</p>



<ul class="wp-block-list">
<li>Cap tool calls per session in your agent config, and return a clear message when the cap is hit rather than letting the loop run.</li>

<li>Cache anything org-wide and slow-moving. Picklist values, record type metadata, product catalogue. These are a surprising share of calls and almost never change mid-day.</li>

<li>Watch API usage as a leading indicator, not a post-mortem. Salesforce exposes remaining daily requests via the REST limits resource. Scrape it on a schedule into CloudWatch, Grafana Cloud, or whatever you already page from, and alert on the trend rather than the wall.</li>
</ul>



<h2 class="wp-block-heading">Wiring it up</h2>



<p class="wp-block-paragraph">With the design decisions settled, the build is short. The AgentCore CLI scaffolds and deploys the harness.</p>



<ol class="wp-block-list">
<li>Create an External Client App in Salesforce with the narrowest OAuth scopes the tools need. Assign it via permission set, not profile.</li>

<li>Build the tool layer: an OpenAPI spec for your purpose-built endpoints, or Lambda functions for composite operations.</li>

<li>Create an AgentCore Gateway, attach the target, and configure the credential provider in AgentCore Identity for the egress side.</li>

<li>Create the harness, point it at the gateway, and deploy.</li>
</ol>



<p class="wp-block-paragraph">Scaffolding a harness and attaching a gateway looks like this. The <code>--gateway-arn</code> value is the gateway you created in step three, and the <code>--type</code> tells the CLI this tool is a gateway rather than one of the built-in tools such as the browser or code interpreter:</p>



<pre class="wp-block-code"><code>agentcore create --name sf-assistant

agentcore add tool --harness sf-assistant 
  --type agentcore_gateway --name sf-gateway 
  --gateway-arn arn:aws:bedrock-agentcore:REGION:ACCOUNT_ID:gateway/GATEWAY_ID</code></pre>



<p class="wp-block-paragraph">Then declare the model and the system prompt, and deploy. Check the current model identifiers in the Bedrock console rather than copying one from any blog post, this one included, because they change:</p>



<pre class="wp-block-code"><code>agentcore add harness 
  --name sf-assistant 
  --model-id YOUR_MODEL_ID 
  --system-prompt "You answer questions about CRM records using only the provided tools. Never guess a record value." 
  --tools sf-gateway

agentcore deploy</code></pre>



<p class="wp-block-paragraph">Invoking it takes a session ID, and an actor ID if you want memory scoped per user. Skipping <code>--actor-id</code> is the memory equivalent of the integration user problem: one shared memory store, everyone&#8217;s context in it.</p>



<pre class="wp-block-code"><code>agentcore invoke --harness sf-assistant 
  --session-id "$(uuidgen)" --actor-id alice 
  "Which of my open opportunities have had no activity in three weeks?"</code></pre>



<p class="wp-block-paragraph">Note &#8220;my&#8221; in that question. With three-legged auth it means something. With an integration user it means nothing, and the model will cheerfully invent a definition.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<p class="wp-block-paragraph"><strong><code>AccessDeniedException</code> when creating an agent.</strong> If you&#8217;re calling <code>CreateAgent</code> or <code>InvokeInlineAgent</code> on the Classic API, check whether the account has prior Bedrock Agents usage before you touch IAM. No policy fixes this one. Build on AgentCore instead.</p>



<p class="wp-block-paragraph"><strong>The New Connected App button is greyed out.</strong> Expected. Create an External Client App. If you have a documented technical requirement that only a Connected App satisfies, that&#8217;s a Salesforce Support request, not a setting you can toggle.</p>



<p class="wp-block-paragraph"><strong>Tools appear in <code>tools/list</code> but calls fail.</strong> Gateway can cache tool schemas from a target without holding a working credential for calls. Discovery succeeding proves the schema synced, not that egress auth works. Look at the credential provider.</p>



<p class="wp-block-paragraph"><strong>The assistant returns fewer records than the user expects.</strong> Under three-legged auth this is often correct behaviour that looks like a bug. Have the user run the same query in the Salesforce UI. If the UI shows more, check FLS on the specific fields; if it shows the same, the assistant is right and the user&#8217;s mental model is wrong.</p>



<p class="wp-block-paragraph"><strong>Answers are confidently wrong about record values.</strong> Usually the model answering from training data instead of calling a tool. Tighten the system prompt to forbid answering record questions without a tool result, and check the traces to confirm which turns actually called out.</p>



<p class="wp-block-paragraph"><strong>Everything works, then breaks at the same time daily.</strong> Look at org-wide API consumption before you look at your own code. Something else in the org may be eating the allocation on a schedule.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Using a System Administrator profile for the integration identity because it makes the errors go away.</li>

<li>Exposing a generic SOQL tool and calling the prompt a security control.</li>

<li>Testing only in a sandbox with one user, where the sharing model is invisible because there&#8217;s nothing to share.</li>

<li>Treating Guardrails as an access control layer. They filter content; they don&#8217;t know who&#8217;s asking.</li>

<li>Shipping writes in the first release because the read-only version felt underwhelming in the demo.</li>

<li>Building on Bedrock Agents Classic in a new account without checking allowlist status first.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Use three-legged OAuth for anything governed by sharing rules. Reserve client credentials for genuinely org-wide data.</li>

<li>Give each tool one job and a fixed query shape. The model supplies parameters, never structure.</li>

<li>Separate read and write identities so you can kill one without the other.</li>

<li>Scope memory per actor, not per deployment.</li>

<li>Test with at least three users at different levels of the role hierarchy, and diff the answers. Identical answers to &#8220;show me my pipeline&#8221; is the signal that identity isn&#8217;t propagating.</li>

<li>Keep end-to-end traces and ship them somewhere you already look, so agent behaviour lands in the same place as the rest of your telemetry.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I still use Bedrock Agents for a new Salesforce project?</h3>



<p class="wp-block-paragraph">Only if the AWS account already used Bedrock Agents in the previous twelve months. Accounts without that history get an <code>AccessDeniedException</code> on <code>CreateAgent</code>, and AWS has said there&#8217;s no exception process. For anything new, build on AgentCore.</p>



<h3 class="wp-block-heading">Do I need Agentforce to build a Salesforce AI assistant with Amazon Bedrock?</h3>



<p class="wp-block-paragraph">No. Agentforce is Salesforce&#8217;s own agent layer, and it can call out to AWS, which is useful when you want the assistant to live inside the Salesforce UI and inherit the Einstein Trust Layer. But an AgentCore agent calling Salesforce APIs through a gateway is a complete architecture on its own, and it&#8217;s the better fit when the assistant also needs to reach systems outside Salesforce.</p>



<h3 class="wp-block-heading">Does the model see all my CRM data?</h3>



<p class="wp-block-paragraph">It sees whatever your tools return. That&#8217;s determined by the OAuth identity and the tool definitions, not by the model. This is why the identity decision matters more than the model choice.</p>



<h3 class="wp-block-heading">Will Bedrock Guardrails stop the assistant leaking records?</h3>



<p class="wp-block-paragraph">No, and this is worth being blunt about. Guardrails filter content categories and can mask sensitive data patterns. They have no concept of which user is asking or which records that user is entitled to. A record correctly retrieved under an over-permissioned token passes every guardrail check, because there&#8217;s nothing objectionable about it. Guardrails are a good layer. They are not this layer.</p>



<h3 class="wp-block-heading">How do I keep the assistant from burning the org&#8217;s API limit?</h3>



<p class="wp-block-paragraph">Cap tool calls per session, cache org-wide reference data, and monitor remaining daily requests via the REST limits resource on a schedule. The allocation is org-wide, so the symptom of overuse usually shows up in some other integration first.</p>



<h3 class="wp-block-heading">Should I migrate an existing Bedrock Agents Classic setup?</h3>



<p class="wp-block-paragraph">There&#8217;s no deadline and no announced end-of-life, so there&#8217;s no emergency. The reasons to move are the frozen model catalogue and the fact that new accounts can&#8217;t recreate your agent. If you&#8217;re still actively developing the thing, move. If it&#8217;s stable and finished, it can wait. AWS publishes a capability mapping and a CLI import path for existing configurations.</p>



<h3 class="wp-block-heading">Is an External Client App harder to set up than a Connected App?</h3>



<p class="wp-block-paragraph">Slightly different rather than harder. The main structural change is that developer settings and admin policies are separated, so the app definition and the per-org access controls are configured independently. The one real removal is the username-password flow, which no longer exists in the ECA model.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A Salesforce AI assistant with Amazon Bedrock is not really a model problem. Model quality is the part that will take care of itself, and the part you can change with one config line later.</p>



<p class="wp-block-paragraph">The part that won&#8217;t take care of itself is identity. If the assistant answers as one integration user, you have quietly moved your entire access control model from a platform that enforces it well into a prompt that enforces it not at all, and nothing in your monitoring will tell you. Get that right first. Then argue about models.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help building or auditing this?</h2>



<p class="wp-block-paragraph">Most of the work on these projects is unglamorous: deciding what the agent is allowed to touch, and proving it can&#8217;t touch anything else. That&#8217;s the part I&#8217;m useful for.</p>



<ul class="wp-block-list">
<li>Auditing an existing Salesforce AI integration for identity propagation gaps, including testing across role hierarchy levels to find silent over-permissioning.</li>

<li>Migrating a Bedrock Agents Classic deployment to AgentCore, including mapping action groups to gateway targets.</li>

<li>Designing the tool surface: replacing generic query tools with narrow, logged, individually rate-limited operations.</li>

<li>Setting up three-legged OAuth between AgentCore Gateway and Salesforce, including token vault and consent handling.</li>

<li>Moving integrations from Connected Apps to External Client Apps without breaking existing OAuth flows.</li>

<li>Instrumenting API consumption and agent traces so you see problems before an unrelated integration starts failing.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;ve got something concrete, send it over: a gateway target config, an agent trace, a tool schema, or just the error you&#8217;re stuck on. Easier to say something useful about a real setup than a hypothetical one.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-ai-assistant-amazon-bedrock/">Building a Salesforce AI Assistant on Amazon Bedrock Without Leaking Your CRM</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</title>
		<link>https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 17 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Cursor Pagination]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[GraphQL]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Orchestration]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[Step Functions]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=215</guid>

					<description><![CDATA[<p>A GraphQL source can hand you a 200 OK, a populated data block, and a quietly broken column in the same response. Here is how to build a GraphQL data ingestion pipeline on AWS that catches partial errors, respects cost-based rate limits, resumes cleanly from a cursor, and notices when the schema moves under you.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/">Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The row counts looked fine. That was the problem.</p>



<p class="wp-block-paragraph">A nightly sync from a partner&#8217;s GraphQL API had run for weeks without a single failed execution. Green across the board in the console. Then someone asked why one column had been empty since the previous Tuesday. Not sparse. Empty. Every row, every night, for six nights.</p>



<p class="wp-block-paragraph">The API had started returning a field-level error on that one field. Status code 200. A populated <code>data</code> block. An <code>errors</code> array sitting right next to it. The fetcher parsed <code>data</code>, wrote the records to S3, exited zero, and the orchestrator logged a success. Nothing in the stack was wrong, exactly. Everything in the stack was checking the wrong thing.</p>



<p class="wp-block-paragraph">This post covers how to build a GraphQL data ingestion pipeline on AWS that fails honestly. It is organised by failure family rather than by service, because the AWS wiring is the easy part. The hard part is that GraphQL breaks four assumptions that REST ingestion quietly relies on, and every one of them fails silently by default.</p>



<h2 class="wp-block-heading">Why GraphQL ingestion breaks the habits REST taught you</h2>



<p class="wp-block-paragraph">Most ETL code carries three inherited reflexes from years of REST work: the status code tells you whether it worked, the response shape is the server&#8217;s business, and rate limits are counted in requests. GraphQL invalidates all three.</p>



<ul class="wp-block-list">
<li><strong>The status code is not the contract.</strong> The GraphQL over HTTP specification says a server should return 200 even when field errors are raised during execution, because a partial response is still a successful execution.</li>

<li><strong>You own the response shape.</strong> The selection set is written by you, which means schema drift shows up as a query that stops being valid, not as a payload that changes underneath you.</li>

<li><strong>Cost is query-shaped, not endpoint-shaped.</strong> One endpoint can cost almost nothing or blow your entire budget depending on how deeply you nested it.</li>
</ul>



<p class="wp-block-paragraph">Each of those becomes a distinct failure family. Let&#8217;s take them in the order they bite.</p>



<h2 class="wp-block-heading">Failure one: the 200 OK that quietly ate a column</h2>



<p class="wp-block-paragraph">This is the one that costs the most and shows up the latest. GraphQL splits errors into two categories, and they behave completely differently on the wire.</p>



<ul class="wp-block-list">
<li><strong>Request errors</strong> happen before execution starts: syntax errors, validation failures, bad variable coercion. No <code>data</code> key is returned at all. Nothing executed.</li>

<li><strong>Field errors</strong> happen during execution: a resolver threw, a downstream service timed out, a permission check failed. Execution continues, the failed field goes null, and you get <code>data</code> and <code>errors</code> together.</li>
</ul>



<p class="wp-block-paragraph">Field errors are the dangerous ones for ingestion, because they look exactly like success to anything that only inspects the transport layer. Worse, GraphQL&#8217;s null propagation rules mean a single failing non-null field bubbles up to the nearest nullable ancestor, so one broken resolver can hollow out an entire nested object while the surrounding rows land normally.</p>



<p class="wp-block-paragraph">The fix is to make the response body the contract. Classify before you commit anything:</p>



<pre class="wp-block-code"><code>import json, urllib.request

class RequestError(Exception): pass
class FieldError(Exception): pass

def graphql(endpoint, token, query, variables):
    body = json.dumps({"query": query, "variables": variables}).encode()
    req = urllib.request.Request(
        endpoint,
        data=body,
        headers={
            "Content-Type": "application/json",
            "Accept": "application/graphql-response+json, application/json",
            "Authorization": f"Bearer {token}",
        },
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        payload = json.loads(resp.read())

    # The status code told us nothing. The body is the contract.
    if "data" not in payload:
        # Nothing executed. Query or auth problem. Do not retry blindly.
        raise RequestError(payload.get("errors"))

    if payload.get("errors"):
        # Partial response. data is populated but incomplete.
        raise FieldError(payload["errors"])

    return payload["data"]</code></pre>



<p class="wp-block-paragraph">Two things worth explaining. The <code>Accept</code> header asks for <code>application/graphql-response+json</code>, the media type the GraphQL over HTTP spec recommends, which lets a compliant server return a real 4xx for pre-execution failures instead of burying them in a 200. Servers that don&#8217;t support it fall back to <code>application/json</code>, so including both is free.</p>



<p class="wp-block-paragraph">The second is the split between the two exception types. They need different handling. A request error is a bug in your query or a credential problem, and retrying it just burns budget on the same failure. A field error is transient often enough to be worth retrying, but never worth committing.</p>



<h3 class="wp-block-heading">Deciding what &#8220;partial&#8221; means for your data</h3>



<p class="wp-block-paragraph">Failing the whole page on any error is the safe default and the one I reach for first. But it&#8217;s not always right. If you are pulling a hundred fields and one optional enrichment field is flaky, refusing every page forever is its own outage.</p>



<p class="wp-block-paragraph">The workable middle ground is to classify by path. Every error object carries a <code>path</code> array pointing at the exact field that failed. Maintain a small allowlist of paths that are permitted to be null, fail the page on anything outside it, and emit the error count as a CloudWatch metric so a rising trend is visible before someone in finance notices.</p>



<h2 class="wp-block-heading">Failure two: rate limits that never send a 429</h2>



<p class="wp-block-paragraph">Your retry logic almost certainly triggers on 429 and 5xx. Several major GraphQL APIs will throttle you without ever sending one.</p>



<p class="wp-block-paragraph">GitHub&#8217;s GraphQL API is explicit about this: exceeding the primary rate limit returns a 200 response with an error message in the body and <code>x-ratelimit-remaining</code> set to zero. Your Step Functions retrier sees a success. Your Lambda sees no data. The run completes, having ingested nothing, and reports green.</p>



<p class="wp-block-paragraph">The other half of the problem is that request counting is the wrong unit. Shopify&#8217;s GraphQL Admin API and several others price queries by <em>calculated cost</em>, using a leaky bucket. The server statically analyses your query before executing it, charges the bucket the worst-case <em>requested</em> cost, executes, then refunds the difference between requested and <em>actual</em> cost. That has a consequence people miss: a query asking for the maximum connection size gets charged as if every page were full, even when the last page returns three records.</p>



<p class="wp-block-paragraph">The good news is that these APIs tell you the state of the bucket on every response, under the <code>extensions</code> key:</p>



<pre class="wp-block-code"><code>{
  "extensions": {
    "cost": {
      "requestedQueryCost": 72,
      "actualQueryCost": 38,
      "throttleStatus": {
        "maximumAvailable": 1000.0,
        "currentlyAvailable": 962.0,
        "restoreRate": 100.0
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Read those numbers, never hardcode them. Bucket size and restore rate vary by plan tier, and a query that works fine against a large merchant&#8217;s store will fail instantly against a small one. The pattern that holds up is a pre-flight check: before dispatching, compare your estimated cost against <code>currentlyAvailable</code>, and if you&#8217;re short, sleep for the shortfall divided by <code>restoreRate</code>.</p>



<p class="wp-block-paragraph">On APIs that expose their own budget as a queryable field, ask for it in the same round trip rather than spending a second call to find out:</p>



<pre class="wp-block-code"><code>query BudgetCheck {
  rateLimit {
    limit
    cost
    remaining
    resetAt
  }
}</code></pre>



<p class="wp-block-paragraph">Two more things about concurrency. First, if you fan out across Lambda invocations, each worker&#8217;s local view of the bucket is wrong, because the bucket is shared. Multiple workers independently deciding they have headroom is the classic way to get throttled while every individual worker believes it&#8217;s behaving. Either centralise the token accounting or cap concurrency low enough that the arithmetic can&#8217;t go wrong.</p>



<p class="wp-block-paragraph">Second, if the source offers a bulk or async export operation, use it for backfills. Paginating fifty thousand records through the interactive endpoint burns the same budget your incremental syncs need, and it competes with itself.</p>



<h2 class="wp-block-heading">Failure three: cursors that don&#8217;t survive a restart</h2>



<p class="wp-block-paragraph">Most production GraphQL schemas follow the Relay cursor connections convention: <code>first</code> and <code>after</code> arguments, a <code>pageInfo</code> object with <code>hasNextPage</code> and <code>endCursor</code>, and edges carrying opaque per-item cursors.</p>



<pre class="wp-block-code"><code>query Orders($cursor: String) {
  orders(first: 100, after: $cursor) {
    pageInfo {
      hasNextPage
      endCursor
    }
    edges {
      cursor
      node {
        id
        updatedAt
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">The filter argument for incremental sync is not standardised. Some schemas take a query-string filter, some take a typed timestamp argument, some expose nothing at all. Run introspection against the schema and find out rather than guessing, because a wrong argument name is a request error and your run will fail before it starts.</p>



<p class="wp-block-paragraph">The failure here is subtle. Cursors are opaque and their validity is entirely up to the server. Some are stable positional pointers. Some encode a sort key and a snapshot identifier and expire. If a page-47 fetch fails and you retry an hour later with the stored cursor, you may get an error, or you may get results from a shifted window with a hole in the middle. The second outcome is worse, because it looks like it worked.</p>



<p class="wp-block-paragraph">Three habits make restarts safe:</p>



<ol class="wp-block-list">
<li><strong>Checkpoint after the write, not after the fetch.</strong> A cursor recorded before the page lands in S3 is a promise you can&#8217;t keep.</li>

<li><strong>Store a high watermark alongside the cursor.</strong> The cursor is your fast path. The watermark, taken from the newest record you actually committed, is your fallback when the cursor is rejected.</li>

<li><strong>Overlap the watermark deliberately.</strong> Restart a few minutes before the last committed timestamp and deduplicate on the primary key downstream. Re-reading a handful of rows is cheap. Missing rows that arrived during a clock skew is not.</li>
</ol>



<p class="wp-block-paragraph">A DynamoDB item per source stream is enough state for all of this:</p>



<pre class="wp-block-code"><code>{
  "source": "orders",
  "run_id": "01hq7k...",
  "cursor": "eyJsYXN0X2lkIjoxMjM0fQ==",
  "high_watermark": "&lt;updatedAt of newest committed record&gt;",
  "pages_committed": 47,
  "status": "IN_PROGRESS"
}</code></pre>



<p class="wp-block-paragraph">Overlap only works if downstream deduplication is real. Write raw pages with an ingest-time partition and a run identifier in the object key, then let the curated layer resolve duplicates by primary key and latest <code>updatedAt</code>. If you write straight into a merged table with no dedupe step, overlapping restarts will double-count and you&#8217;ll have traded silent loss for silent inflation.</p>



<h2 class="wp-block-heading">Failure four: the schema moved and nobody told you</h2>



<p class="wp-block-paragraph">GraphQL&#8217;s deprecation story is genuinely better than REST&#8217;s. Fields carry a <code>@deprecated</code> directive with a reason string, and introspection exposes it. The problem is that nothing forces you to look.</p>



<p class="wp-block-paragraph">The failure mode is a field that gets deprecated, stops being populated, then eventually gets removed. Stage one is invisible: your query still validates, the field returns null, and your warehouse fills with nulls. Stage two is loud but late: the query becomes invalid and the whole ingest breaks at 3am.</p>



<p class="wp-block-paragraph">Put an introspection diff in CI. Fetch the schema on a schedule, store it as an artifact, and diff against the last known good version. Fail the build on removals or type changes to fields your queries actually select, and warn on new deprecations. Tools in the Apollo and Hasura ecosystems ship schema-diff tooling for this, and rolling your own against the introspection result is not much work either. Either way, the point is that the check runs on a schedule rather than when the pager goes off.</p>



<p class="wp-block-paragraph">The complementary habit is to keep queries narrow. Every field you select is a field that can break you, a field that costs budget, and a field that widens your Glue table. Selecting a whole object because it was convenient during development is a liability you carry indefinitely.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Wiring the GraphQL data ingestion pipeline on AWS</h2>



<p class="wp-block-paragraph">With the failure families understood, the AWS assembly is mostly a set of trade-offs about where the fetch loop lives.</p>



<h3 class="wp-block-heading">Where the fetch loop runs</h3>



<p class="wp-block-paragraph">Step Functions can call an HTTPS endpoint directly with an HTTP Task, no Lambda involved. It uses an EventBridge connection to hold the credentials, which means your token lives in a Secrets Manager secret managed by EventBridge rather than in an environment variable:</p>



<pre class="wp-block-code"><code>"FetchPage": {
  "Type": "Task",
  "Resource": "arn:aws:states:::http:invoke",
  "Parameters": {
    "ApiEndpoint": "https://api.example.com/graphql",
    "Method": "POST",
    "InvocationConfig": {
      "ConnectionArn": "arn:aws:events:region:account-id:connection/name/id"
    },
    "Headers": { "Content-Type": "application/json" },
    "RequestBody": {
      "query.$": "$.query",
      "variables.$": "$.variables"
    }
  },
  "Retry": [{
    "ErrorEquals": [
      "States.Http.StatusCode.429",
      "States.Http.StatusCode.502",
      "States.Http.StatusCode.503",
      "States.Http.StatusCode.504"
    ],
    "IntervalSeconds": 1,
    "BackoffRate": 2,
    "MaxAttempts": 3,
    "JitterStrategy": "FULL"
  }],
  "Next": "InspectBody"
}</code></pre>



<p class="wp-block-paragraph">The execution role needs four permissions: <code>states:InvokeHTTPEndpoint</code>, <code>events:RetrieveConnectionCredentials</code>, and both <code>secretsmanager:GetSecretValue</code> and <code>secretsmanager:DescribeSecret</code> scoped to the EventBridge-managed secret.</p>



<p class="wp-block-paragraph">It&#8217;s a clean pattern, and I like it for small, high-value syncs. Two constraints decide whether it fits. HTTP Task requests time out after 60 seconds, which a deeply nested GraphQL query can exceed without being pathological. And Step Functions reads the response within its standard state payload ceiling of roughly 256 KB, so a page of a hundred rich records will not fit. Also note the retrier above: it fires on HTTP status codes, so it will not catch a throttle that arrives as a 200 with an error in the body. That check has to happen in the state after it.</p>



<p class="wp-block-paragraph">The alternative is a Lambda driver that paginates internally and writes pages straight to S3, returning only the cursor and a count to the state machine. This dodges both limits, gives you a real HTTP client with real backoff, and keeps large payloads out of the workflow&#8217;s state. It costs you per-page retry granularity, which you buy back with the checkpoint record.</p>



<p class="wp-block-paragraph">Distributed Map is worth mentioning for the fan-out case, such as one child execution per tenant or per shard. It gives you high concurrency and a per-item execution history, which is genuinely useful for debugging. Be aware of two things before committing: express child workflows cap at five minutes rather than Lambda&#8217;s fifteen, and practitioners have published benchmarks showing throughput well below what the headline concurrency numbers suggest for small, fast items. For a lot of ingestion work, SQS plus a DynamoDB job table remains the simpler answer.</p>



<p class="wp-block-paragraph">One case genuinely falls outside serverless: sources that require a long-lived session, a persistent websocket subscription, or a synchronisation window longer than fifteen minutes with no resumable cursor. A small always-on worker on a modest VPS from a provider like InterServer or Hetzner, feeding the same S3 landing zone, is less clever and more honest than contorting Lambda around the constraint.</p>



<h3 class="wp-block-heading">The landing zone</h3>



<p class="wp-block-paragraph">Land raw responses first, unmodified, including the <code>errors</code> and <code>extensions</code> keys. Storage is the cheapest thing in this architecture and the raw layer is what lets you reprocess after you discover a parsing bug six weeks later.</p>



<ul class="wp-block-list">
<li><strong>Partition by ingest time, not event time.</strong> Event-time partitioning requires reading the record before you know where to put it, which means you can&#8217;t write until you&#8217;ve parsed. Ingest-time keys let the raw write be dumb and fast.</li>

<li><strong>Put the run identifier in the key.</strong> When you need to unwind one bad run, being able to delete or ignore by prefix beats writing a surgical delete.</li>

<li><strong>Define Glue tables explicitly rather than crawling.</strong> You wrote the selection set, so you already know the schema. A crawler inferring types from JSON will surprise you the first time an optional field is absent from an entire page.</li>

<li><strong>Convert to columnar in the curated layer.</strong> Athena over raw JSON works and is fine for debugging. Parquet is what you want for anything anyone queries repeatedly.</li>
</ul>



<p class="wp-block-paragraph">If your source pushes rather than being polled, such as webhook deliveries carrying GraphQL payloads, Amazon Data Firehose (renamed from Kinesis Data Firehose, with APIs and IAM policies unchanged) will buffer, convert to Parquet and partition on the way to S3 without you running a consumer. For scheduled pull-based syncs it adds a hop you don&#8217;t need.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Runs succeed but a column is all null.</strong> Field errors. Pull a raw response from the landing zone and look at the <code>errors</code> array. The <code>path</code> tells you exactly which field.</li>

<li><strong>Runs succeed but ingest zero rows.</strong> Likely a throttle returned as a 200. Log the response body and the rate-limit headers on empty results, always.</li>

<li><strong>Intermittent failures partway through pagination.</strong> Cursor expiry. Fall back to the watermark and check whether the source documents a cursor lifetime.</li>

<li><strong>Query fails immediately with no <code>data</code> key.</strong> Request error. Validate against current introspection; something in the schema moved.</li>

<li><strong>Throttled despite staying under the documented limit.</strong> Either concurrent workers are sharing a bucket they each think they own, or you&#8217;re being charged requested cost on near-empty pages.</li>

<li><strong>HTTP Task fails on large pages.</strong> The response exceeds the state payload ceiling. Reduce page size or move the fetch into Lambda.</li>

<li><strong>Row counts drift upward over time.</strong> Overlap without deduplication. Check that the curated layer actually resolves on the primary key.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating a 200 as success and never reading the body.</li>

<li>Retrying request errors. They are deterministic; retrying just spends budget to fail identically.</li>

<li>Hardcoding bucket sizes and restore rates that differ per plan tier and change without notice.</li>

<li>Checkpointing the cursor before the data is durable.</li>

<li>Requesting the maximum page size on every call while being charged the worst-case cost for it.</li>

<li>Selecting entire objects because it was easier during development.</li>

<li>Alerting only on execution failure, so a job that succeeds while ingesting nothing never pages anyone.</li>

<li>Overlapping restart windows with no downstream deduplication.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Classify every response into clean, partial, or request-failed before anything downstream sees it.</li>

<li>Persist raw responses with <code>errors</code> and <code>extensions</code> intact, then parse from the raw layer.</li>

<li>Alarm on row-count deltas and partial-error rate, not just on execution status.</li>

<li>Read the throttle state the API hands you and back off before you hit the wall, not after.</li>

<li>Keep both a cursor and a watermark, and know which one is authoritative on restart.</li>

<li>Diff the schema on a schedule so deprecations reach you before they reach your data.</li>

<li>Hold credentials in Secrets Manager or an EventBridge connection, never in a Lambda environment variable.</li>

<li>Trace the pipeline end to end. CloudWatch covers the basics; Grafana or Datadog earn their keep once you&#8217;re correlating child executions across a fan-out.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can Step Functions call a GraphQL API without a Lambda function?</h3>



<p class="wp-block-paragraph">Yes. An HTTP Task using the <code>arn:aws:states:::http:invoke</code> resource posts your query directly, with an EventBridge connection handling authentication. It works well for small pages. The 60-second request timeout and the state payload size ceiling are what push most real ingestion work back into Lambda.</p>



<h3 class="wp-block-heading">How do I detect GraphQL errors when the status code is always 200?</h3>



<p class="wp-block-paragraph">Inspect the body. If there&#8217;s no <code>data</code> key, nothing executed and you have a request error. If <code>data</code> is present alongside a non-empty <code>errors</code> array, you have a partial response and the data is incomplete. Only a response with <code>data</code> and no errors is unambiguously clean.</p>



<h3 class="wp-block-heading">Does AWS AppSync fit into this?</h3>



<p class="wp-block-paragraph">Only if you&#8217;re on the serving side. AppSync is a managed GraphQL API for exposing your own data, so it&#8217;s what you&#8217;d reach for if you wanted clients to push events in via mutations. For pulling data out of somebody else&#8217;s GraphQL API, AppSync isn&#8217;t part of the picture.</p>



<h3 class="wp-block-heading">Should I store raw GraphQL responses or parsed records in S3?</h3>



<p class="wp-block-paragraph">Both, in separate layers. Raw responses are your audit trail and your reprocessing path, and they&#8217;re the only way to answer &#8220;was this field null in the source, or did my parser drop it?&#8221; weeks after the fact. Parsed and columnar output goes in a curated prefix that analysts query.</p>



<h3 class="wp-block-heading">How do I run an incremental sync from a GraphQL API?</h3>



<p class="wp-block-paragraph">Find the schema&#8217;s filter argument for a modification timestamp using introspection, pass your stored watermark into it, and page forward with cursors. Overlap the window slightly on each run and deduplicate on the primary key downstream. If the schema exposes no such filter, you&#8217;re limited to full pulls or webhook-driven change capture.</p>



<h3 class="wp-block-heading">Why am I throttled when I&#8217;m making very few requests?</h3>



<p class="wp-block-paragraph">Cost-based APIs charge for query complexity, not call count. A single deeply nested query pulling a connection inside a connection can consume a large fraction of the bucket by itself. Read the cost figures the API returns and flatten the query if the requested cost is consistently much higher than the actual cost.</p>



<h3 class="wp-block-heading">Is GraphQL a worse ingestion source than REST?</h3>



<p class="wp-block-paragraph">Different, not worse. You control the payload, which cuts transfer and lets you avoid the N+1 fan-out REST forces on nested data. Deprecation metadata is machine-readable, which REST rarely offers. The cost is that error signalling and rate limiting both moved into the body, so naive clients fail silently instead of loudly.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A GraphQL data ingestion pipeline on AWS fails differently from a REST one, and the difference is almost entirely about where the truth lives. In REST, the transport tells you what happened. In GraphQL, the transport tells you the request arrived, and the body tells you what happened. Every failure family in this post traces back to that: partial errors hidden behind a 200, throttles hidden behind a 200, cursors that fail by returning the wrong rows rather than an error.</p>



<p class="wp-block-paragraph">So build the pipeline so that a green run means something. Parse before you commit, alarm on row deltas rather than exit codes, and keep the raw responses so you can prove what the source actually said. Everything else is ordinary AWS plumbing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with a GraphQL ingestion pipeline?</h2>



<p class="wp-block-paragraph">This is the kind of work I take on regularly. Specifically:</p>



<ul class="wp-block-list">
<li>Auditing an existing GraphQL sync for silent partial-error loss and telling you which columns are already affected</li>

<li>Building the fetch, checkpoint and land loop in Step Functions and Lambda with cursors that survive restarts</li>

<li>Fitting a cost-aware throttle to whatever budget model your source actually uses, including multi-worker coordination</li>

<li>Designing the raw and curated S3 layout, Glue table definitions and Athena partitioning so queries stay cheap</li>

<li>Adding introspection diffing to CI so schema deprecations surface in a pull request, not in a dashboard</li>

<li>Wiring CloudWatch metrics and alarms on row-count deltas and partial-error rates, so a green run is actually green</li>
</ul>



<p class="wp-block-paragraph">Send me a run log, a state machine definition, or a raw response with an <code>errors</code> array in it, and I&#8217;ll tell you what it&#8217;s costing you.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/">Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Optimizing API Calls to Reduce SaaS Costs: Six Levers That Actually Move the Bill</title>
		<link>https://john-nessime.com/blog/devops/reduce-saas-api-costs/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Fri, 14 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[API Cost Optimization]]></category>
		<category><![CDATA[Batch Processing]]></category>
		<category><![CDATA[Cache Control]]></category>
		<category><![CDATA[Cache Stampede]]></category>
		<category><![CDATA[Caching]]></category>
		<category><![CDATA[Cloudflare]]></category>
		<category><![CDATA[Conditional Requests]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[ETag]]></category>
		<category><![CDATA[Exponential Backoff]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[HTTP Headers]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Nginx]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Polling]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[Redis]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Retry Budgets]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[Vendor Spend]]></category>
		<category><![CDATA[Webhooks]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=229</guid>

					<description><![CDATA[<p>Third-party API spend is the one production signal with no error rate attached to it, which is why it creeps up quietly. This is a working engineer's guide to reducing SaaS API costs by changing the shape of your calls: reading the billing unit before you optimise anything, killing pointless polling with conditional requests and webhooks, collapsing N+1 patterns, caching with stampede protection and per-tenant keys, stopping your own retry amplification, and attributing spend so you can prove the work paid off.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/reduce-saas-api-costs/">Optimizing API Calls to Reduce SaaS Costs: Six Levers That Actually Move the Bill</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually comes from finance, not from monitoring. A vendor line on the card went up again, nobody shipped a feature that month, and now somebody wants an explanation by Friday.</p>



<p class="wp-block-paragraph">Here is the awkward part: nothing was broken. No alert fired, no error rate moved, no dashboard turned red. Third-party API spend is one of the few production signals with no failure mode attached to it. A background job that polls a vendor every thirty seconds and gets back the exact same payload it got last time looks, from the outside, identical to a healthy integration. It just costs money every single time.</p>



<p class="wp-block-paragraph">This post is about how to reduce SaaS API costs by changing the shape of your calls rather than by renegotiating your contract. Six levers, in the order I&#8217;d pull them: read the billing unit, kill pointless polling, collapse N+1 patterns, cache honestly, stop your own retry amplification, and attribute spend so you can tell whether any of it worked.</p>



<h2 class="wp-block-heading">Start with the billing unit, not the call count</h2>



<p class="wp-block-paragraph">The single most common mistake I see is optimising HTTP requests when the vendor doesn&#8217;t bill HTTP requests. Before you touch any code, read the pricing page carefully enough to answer one question: <em>what is the countable thing?</em></p>



<p class="wp-block-paragraph">It varies more than people expect:</p>



<ul class="wp-block-list">
<li><strong>Per request.</strong> The obvious one. Geocoding, enrichment, most REST endpoints.</li>

<li><strong>Per session.</strong> Google Maps Platform bills Places Autocomplete this way when you pass a session token: the whole keystroke stream plus the terminating Place Details call is grouped and billed as one unit. Omit the token, or reuse one across sessions, and every keystroke request is billed individually.</li>

<li><strong>Per token issued.</strong> Auth0&#8217;s machine-to-machine quotas count access tokens obtained through the client credentials flow, not the API calls you make with them. A service that fetches a fresh token on every outbound request is burning the meter for no reason.</li>

<li><strong>Per input and output token.</strong> LLM providers. Output is typically priced higher than input, which makes &#8220;be concise&#8221; a cost control and not just a style note.</li>

<li><strong>Per record, event, or message.</strong> Twilio, SendGrid, most streaming and messaging platforms. One API call can carry many billable units.</li>

<li><strong>Not at all.</strong> Stripe doesn&#8217;t bill you per API call; the constraint there is a rate limit, not a line item. Optimising those calls buys you reliability headroom, not money.</li>
</ul>



<p class="wp-block-paragraph">Getting this wrong sends you down the wrong path for weeks. Batching requests against an endpoint that bills per record changes your request count and leaves the invoice exactly where it was. Read the unit first.</p>



<h2 class="wp-block-heading">Lever one: stop polling for things that rarely change</h2>



<p class="wp-block-paragraph">This is almost always the biggest single line, and it&#8217;s invisible because it&#8217;s steady. A sync job on a five-minute cron is 288 calls a day per resource. Multiply by resources, by environments, by the replicas you quietly scaled to three, and you get a number with no relationship at all to how often the data changed.</p>



<h3 class="wp-block-heading">Conditional requests, when the vendor honours them</h3>



<p class="wp-block-paragraph">If the API returns an <code>ETag</code> or <code>Last-Modified</code> header, store it and send it back on the next poll as <code>If-None-Match</code> or <code>If-Modified-Since</code>. If nothing changed you get a <code>304 Not Modified</code> with an empty body.</p>



<pre class="wp-block-code"><code># First call: capture the ETag from the response headers
curl -sS -D - -o /dev/null 
  -H "Authorization: Bearer $TOKEN" 
  https://api.github.com/repos/OWNER/REPO/pulls

# Second call: send it back and check the status code only
curl -sS -o /dev/null -w '%{http_code}n' 
  -H "Authorization: Bearer $TOKEN" 
  -H 'If-None-Match: "PASTE_THE_ETAG_HERE"' 
  https://api.github.com/repos/OWNER/REPO/pulls</code></pre>



<p class="wp-block-paragraph"><code>-D -</code> dumps response headers to stdout so you can see the <code>etag</code> value. <code>-w '%{http_code}'</code> on the second call prints just the status, which is all you care about: <code>304</code> means the resource is unchanged and you saved a payload.</p>



<p class="wp-block-paragraph">Whether that saves <em>money</em> depends on the vendor. GitHub documents that a conditional request returning 304 doesn&#8217;t count against your primary rate limit when it&#8217;s correctly authorised. Plenty of other vendors return 304 happily and bill it anyway. Test against your own invoice before you assume.</p>



<p class="wp-block-paragraph">Three things that bite people with ETags:</p>



<ul class="wp-block-list">
<li>They&#8217;re per-page on paginated endpoints. A 304 on page one says nothing about pages two through five, so store one ETag per page.</li>

<li>They can be scoped to the credential. Rotate a token and your stored ETags may stop matching, which quietly turns your conditional requests back into full ones.</li>

<li>GraphQL endpoints generally don&#8217;t do conditional requests at all. You cache those yourself, keyed on a hash of the query and its variables.</li>
</ul>



<h3 class="wp-block-heading">Webhooks, when they exist</h3>



<p class="wp-block-paragraph">The proper fix is to stop asking. If the vendor supports webhooks, the call count collapses from &#8220;one per interval&#8221; to &#8220;one per actual change&#8221;, which is usually a difference of two orders of magnitude on a slow-moving resource.</p>



<p class="wp-block-paragraph">Be honest about what it costs, though. Webhooks need a publicly reachable endpoint, signature verification, idempotent handling because vendors redeliver, a dead-letter path, and a reconciliation sweep for deliveries that never arrived. On a resource that changes every few seconds, polling is simpler and the savings are small. On one that changes twice a week, the webhook pays for itself immediately.</p>



<p class="wp-block-paragraph">A middle option people forget: many APIs expose a cheap &#8220;has anything changed&#8221; endpoint, a sync cursor, or a delta token. Poll that on a tight interval and only fetch the expensive payload when it moves.</p>



<h2 class="wp-block-heading">Lever two: collapse the N+1 pattern</h2>



<p class="wp-block-paragraph">You know this one from databases, and it shows up identically against third-party APIs. Fetch a list of 200 items, then fetch each item&#8217;s detail individually: that&#8217;s 201 billable calls where the vendor probably offers a way to do it in one or two. Look for:</p>



<ul class="wp-block-list">
<li><strong>Expansion or embed parameters.</strong> Many REST APIs let you inline related objects in the list response so the detail calls disappear entirely.</li>

<li><strong>Field selection.</strong> Asking for fewer fields is free on per-request billing but matters a lot where the field tier drives the price. Google Places is explicit about this: requesting a single higher-tier field upgrades the whole response to that tier&#8217;s pricing.</li>

<li><strong>Bulk and batch endpoints.</strong> Salesforce, Algolia, most CRMs and search platforms have one. They&#8217;re usually documented in a corner nobody reads.</li>

<li><strong>Larger pages.</strong> If the endpoint accepts a page size of 200 and you&#8217;re defaulting to 25, you&#8217;re paying eight times as many calls to fetch identical data.</li>
</ul>



<p class="wp-block-paragraph">The page-size one is the cheapest win in this entire post and it&#8217;s a one-line change. Check your client library defaults today.</p>



<h2 class="wp-block-heading">Lever three: cache with a boring, honest TTL</h2>



<p class="wp-block-paragraph">Caching third-party responses is the lever everyone reaches for first and gets subtly wrong. Three failure modes matter more than the cache itself.</p>



<p class="wp-block-paragraph"><strong>Stampede.</strong> A popular key expires, fifty concurrent requests all miss, and fifty identical calls go out to the vendor. You just paid fifty times for one refresh, at the worst possible moment.</p>



<p class="wp-block-paragraph"><strong>No negative caching.</strong> A lookup that returns &#8220;not found&#8221; is usually not cached at all, so the same failing lookup hammers the vendor forever. Cache your 404s. Short TTL, but cache them.</p>



<p class="wp-block-paragraph"><strong>Cache key design.</strong> If your key doesn&#8217;t include the credential or tenant, you will eventually serve one customer&#8217;s data to another. This is the one mistake in this post that turns a cost problem into an incident.</p>



<p class="wp-block-paragraph">You don&#8217;t necessarily need application code for this. An Nginx instance in front of the vendor handles all three, and it&#8217;s the approach I reach for first when several services hit the same API:</p>



<pre class="wp-block-code"><code>proxy_cache_path /var/cache/nginx/api levels=1:2 keys_zone=api_cache:20m
                 max_size=1g inactive=60m use_temp_path=off;

server {
    listen 127.0.0.1:8080;

    location / {
        proxy_pass https://api.vendor.example;
        proxy_set_header Host api.vendor.example;

        proxy_cache api_cache;
        proxy_cache_key "$request_method$request_uri$http_authorization";

        proxy_cache_valid 200 5m;
        proxy_cache_valid 404 30s;

        proxy_cache_lock on;
        proxy_cache_use_stale updating error timeout;
        proxy_cache_background_update on;

        add_header X-Cache-Status $upstream_cache_status;
    }
}</code></pre>



<p class="wp-block-paragraph">Reading that from the top down, because every directive is there for a reason:</p>



<ul class="wp-block-list">
<li><code>proxy_cache_key</code> includes <code>$http_authorization</code>, so responses are isolated per credential. That&#8217;s the tenant-leak fix.</li>

<li><code>proxy_cache_valid 404 30s</code> is the negative caching. Failed lookups stop being free traffic for the vendor.</li>

<li><code>proxy_cache_lock on</code> is the stampede fix. Only the first request for a given key goes upstream; the rest wait for it.</li>

<li><code>proxy_cache_use_stale updating</code> plus <code>proxy_cache_background_update on</code> serves the stale copy while a single background request refreshes it. Latency stays flat and you don&#8217;t pay a thundering herd at every expiry.</li>

<li><code>X-Cache-Status</code> gives you <code>HIT</code>, <code>MISS</code>, <code>STALE</code> or <code>UPDATING</code> in the response so you can actually measure your hit rate instead of guessing at it.</li>
</ul>



<p class="wp-block-paragraph">The trade-off is worth stating plainly: you&#8217;re now serving data that can be five minutes stale, and you own another moving part. Fine for a currency rate or a product catalogue. Not fine for an inventory count on a checkout page. Set the TTL from how wrong the data is allowed to be, never from how much you want to save.</p>



<p class="wp-block-paragraph">Where the caller is distributed, Redis with a short lock key does the same job in application code, and a small always-on VPS from something like InterServer or DigitalOcean is plenty to host either the proxy or the cache. At the edge, Cloudflare Workers with KV lets you cache vendor responses close to users, which helps when the calls originate from browsers rather than your backend.</p>



<h2 class="wp-block-heading">Lever four: stop paying for your own retries</h2>



<p class="wp-block-paragraph">This produces the sudden spike rather than the slow creep, and people miss it because retries feel like resilience.</p>



<p class="wp-block-paragraph">The vendor wobbles. Every client retries. With a fixed delay they retry in lockstep, so the vendor gets a synchronised wave on top of normal load, struggles longer, and generates more retries. AWS&#8217;s architecture write-up on exponential backoff and jitter is the canonical reference, and its central finding is worth repeating: backoff alone still leaves visible clusters of calls. Randomising each client&#8217;s delay is what decorrelates the herd, and in their simulation it cut total call count substantially.</p>



<pre class="wp-block-code"><code>import random, time

BASE = 0.5      # seconds
CAP  = 30.0     # never sleep longer than this

def sleep_before_retry(attempt):
    # capped exponential backoff, then full jitter
    window = min(CAP, BASE * (2 ** attempt))
    time.sleep(random.uniform(0, window))</code></pre>



<p class="wp-block-paragraph">The cap stops a long outage turning into hour-long sleeps. The <code>random.uniform(0, window)</code> is the jitter: two processes that failed at the same instant now wake at different times. Note that full jitter can occasionally produce a near-zero delay, which is fine for transient errors but too aggressive for throttling responses that reset on a per-minute window. For a <code>429</code>, honour the <code>Retry-After</code> header if the vendor sends one, and use a jitter formula with a guaranteed floor if they don&#8217;t.</p>



<p class="wp-block-paragraph">Three more rules that cost nothing to implement:</p>



<ol class="wp-block-list">
<li><strong>Never retry a 4xx except 429.</strong> A 400 or a 422 will fail identically forever. Every one of those retries is a billed call buying you nothing.</li>

<li><strong>Cap retries as a fraction of traffic.</strong> A retry budget (say, retries may not exceed some small percentage of total requests) is the only control that sees the system-wide picture. Per-client backoff can&#8217;t.</li>

<li><strong>Check your layers.</strong> If the SDK retries three times, your HTTP client retries three times, and your job runner retries three times, one logical operation is up to twenty-seven calls. This stacking is extremely common and almost never intentional.</li>
</ol>



<p class="wp-block-paragraph">For writes, retrying safely needs an idempotency key so the vendor deduplicates rather than creating two of whatever you asked for. Stripe&#8217;s <code>Idempotency-Key</code> header is the pattern most others copied. Without it, teams disable retries on writes entirely and then eat manual reconciliation work instead.</p>



<h2 class="wp-block-heading">Lever five: move latency-tolerant work off the synchronous path</h2>



<p class="wp-block-paragraph">Ask a blunt question about each integration: does a human wait on this response?</p>



<p class="wp-block-paragraph">Nightly enrichment, classification, report generation, bulk address validation, embedding backfills. Nobody is waiting. Several vendors, including the major LLM providers, offer asynchronous batch endpoints at a discount to the synchronous rate in exchange for a looser delivery window. Rates move, so check current terms rather than a number you read somewhere, but the mechanism is stable and the discount is usually substantial.</p>



<p class="wp-block-paragraph">Even without a batch endpoint, moving work behind a queue lets you coalesce. Ten events touching the same customer record within a minute become one enrichment call instead of ten. That deduplication window is often worth more than the batch discount.</p>



<h2 class="wp-block-heading">Lever six: attribute the spend or you&#8217;re guessing</h2>



<p class="wp-block-paragraph">Everything above is unfalsifiable until you can answer: which feature, which tenant, which job. The invoice gives you the total and never the cause. What I&#8217;d put in place, in order:</p>



<ol class="wp-block-list">
<li><strong>Route outbound calls through one place.</strong> A shared HTTP client wrapper or the caching proxy above. If every service instantiates its own client, you have no chokepoint and no measurement.</li>

<li><strong>Emit a counter per call</strong> with labels for vendor, endpoint, status class, and cache result. Keep the label set small; per-tenant labels will blow up your metrics cardinality and trade one bill for another.</li>

<li><strong>Build one unit-cost metric.</strong> Vendor spend per active tenant per month. It&#8217;s the only number that survives growth, because raw spend going up during a good quarter is not a problem.</li>

<li><strong>Alert on call rate, not on the bill.</strong> A rate alert fires in minutes. An invoice fires in thirty days. Grafana, Prometheus, Datadog, CloudWatch, whatever you already run is fine; the point is that the alert exists at all.</li>

<li><strong>Track cache hit rate as a first-class SLI.</strong> When someone changes a cache key format and the hit rate silently drops to zero, this is the graph that tells you before the invoice does.</li>
</ol>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting: the bill jumped and nothing shipped</h2>



<p class="wp-block-paragraph">Work down this list. In my experience the cause is usually in the first four.</p>



<ul class="wp-block-list">
<li><strong>Replica count changed.</strong> Autoscaling or a Kubernetes replica bump multiplies every cron-driven poller. Check whether your scheduled jobs have leader election, or whether all five pods are dutifully polling the same endpoint.</li>

<li><strong>Cache hit rate collapsed.</strong> A changed key format, a Redis eviction policy under memory pressure, or a TTL someone dropped &#8220;temporarily&#8221; during debugging. Graph the hit rate over the period, not the current value.</li>

<li><strong>Retry loop against a persistent 4xx.</strong> Look for a flat, high rate of identical failing calls. This has the distinctive signature of high call volume with zero successful responses.</li>

<li><strong>A non-production environment pointed at production credentials.</strong> Staging, a CI pipeline, or an integration test suite running on every commit. Check whether calls are arriving from IPs you don&#8217;t recognise.</li>

<li><strong>Session or token handling regressed.</strong> A refactor that dropped a session token parameter, or a service that stopped caching its auth token and now mints a new one per request.</li>

<li><strong>Legitimate growth in an expensive segment.</strong> A few large customers using a costly feature move the bill without moving call count much. Not a bug, but worth ruling in before you go hunting for one.</li>
</ul>



<p class="wp-block-paragraph">If the vendor exposes per-key usage breakdowns in their dashboard, split your credentials by service before your next incident. Separate API keys per environment and per major consumer turn a whodunnit into a two-minute lookup.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Optimising the endpoint that&#8217;s easy to change rather than the one that dominates the bill. Rank by spend first.</li>

<li>Caching without a per-credential key, and finding out the hard way in a multi-tenant system.</li>

<li>Treating 304 responses as free when your specific vendor bills them anyway.</li>

<li>Setting TTLs from a savings target rather than a staleness tolerance, and finding out during a customer escalation.</li>

<li>Leaving retry logic enabled at three separate layers and calling the result &#8220;defence in depth&#8221;.</li>

<li>Adding per-tenant labels to outbound-call metrics and moving the cost from the API vendor to the observability vendor.</li>

<li>Shipping the optimisation and never checking the next invoice against the prediction.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth keeping</h2>



<ul class="wp-block-list">
<li>One outbound HTTP client wrapper per service, with timeouts, retries, and metrics configured in exactly one place.</li>

<li>Separate API credentials per environment and per major consumer, so usage dashboards mean something.</li>

<li>Cache auth tokens until shortly before expiry. Refresh early, not on every call.</li>

<li>Add jitter to scheduled jobs as well as to retries. Everything set to run at the top of the hour arrives at the vendor as a spike.</li>

<li>Write down the staleness tolerance for each cached resource next to its TTL, so the next person doesn&#8217;t have to reverse-engineer your reasoning.</li>

<li>Re-check the pricing page after any vendor product update. Billing units change more often than endpoints do.</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">How do I find out which service is driving my API spend?</h3>



<p class="wp-block-paragraph">Start with the vendor&#8217;s own usage dashboard broken down by API key. If all your services share one key, that&#8217;s the first thing to fix. Failing that, log every outbound call at the client wrapper with the endpoint and calling service, and aggregate for a week. A week of honest logs beats a month of theorising.</p>



<h3 class="wp-block-heading">Do ETags actually reduce SaaS API costs, or just help with rate limits?</h3>



<p class="wp-block-paragraph">It depends entirely on the vendor. GitHub documents that an authorised conditional request returning 304 doesn&#8217;t count against the primary rate limit. Others count every request regardless of status. Run a controlled test: a known number of conditional calls over a billing period, then compare against the usage report. Don&#8217;t assume either way.</p>



<h3 class="wp-block-heading">Is caching third-party API responses allowed?</h3>



<p class="wp-block-paragraph">Usually yes, but not always, and some terms of service restrict how long you may store certain data. Mapping and place data is a common example of restricted caching. Check the terms for the specific vendor before you build a permanent store, particularly if you&#8217;re caching anything personally identifiable.</p>



<h3 class="wp-block-heading">How much can caching realistically cut an API bill?</h3>



<p class="wp-block-paragraph">Nobody can give you a number without knowing your traffic. Estimate it yourself: for a candidate endpoint, count distinct request keys versus total requests over a day. That ratio is roughly your ceiling for hit rate. Multiply by the endpoint&#8217;s share of spend. If the answer is a couple of percent, spend your time elsewhere.</p>



<h3 class="wp-block-heading">Should I always replace polling with webhooks?</h3>



<p class="wp-block-paragraph">No. The comparison is change frequency against poll frequency. If the resource changes almost as often as you poll, you save little and take on signature verification, redelivery handling, and a reconciliation job. Webhooks win decisively on slow-changing resources and on anything where you&#8217;re polling more than a handful of objects.</p>



<h3 class="wp-block-heading">Does rate limiting my own users reduce vendor costs?</h3>



<p class="wp-block-paragraph">Only when user actions map directly to vendor calls, such as a search box that fires an upstream request per keystroke. Debouncing input and enforcing per-plan quotas helps a lot there. For background processing driven by your own schedules, user rate limits change nothing.</p>



<h3 class="wp-block-heading">Should I disable retries to save money?</h3>



<p class="wp-block-paragraph">No. Fix them instead. Capped exponential backoff with jitter, no retries on non-429 client errors, a retry budget, and retry logic living at exactly one layer. Disabling retries entirely trades a cost problem for a reliability problem, and reliability problems are more expensive.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Most efforts to reduce SaaS API costs fail because they start with code. They should start with two questions: what is the vendor actually counting, and which of my calls returned information I already had?</p>



<p class="wp-block-paragraph">Answer those honestly and the work usually collapses to a handful of unglamorous changes. Raise a page size. Cache a token. Put a lock around a cache refresh. Delete a poller that was watching something that changes twice a week. None of it is clever. All of it shows up on the next invoice, which is the only review that counts.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help getting your API spend under control?</h2>



<p class="wp-block-paragraph">This is work I take on regularly. Typical engagements look like:</p>



<ul class="wp-block-list">
<li>Auditing outbound API traffic to find which endpoints, jobs and tenants are driving the bill, and ranking them by spend rather than by how obvious they are.</li>

<li>Building a caching layer in front of chatty vendor APIs, with Nginx, Redis or Cloudflare Workers, including stampede protection and per-tenant key isolation.</li>

<li>Replacing polling jobs with webhook receivers or delta-token syncs, with signature verification, idempotent handling and a dead-letter path.</li>

<li>Auditing timeout and retry configuration across services to remove stacked retries and add proper backoff, jitter and retry budgets.</li>

<li>Moving latency-tolerant workloads onto batch or queue-based paths, with deduplication windows that cut call volume before it reaches the vendor.</li>

<li>Setting up call-rate and cache-hit-rate dashboards and alerts in Prometheus, Grafana, Datadog or CloudWatch so the invoice is never the first signal.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;ve got a cost graph that&#8217;s climbing, a job you suspect is over-polling, or a client wrapper you&#8217;re not sure about, send it over and I&#8217;ll tell you what I&#8217;d look at first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/reduce-saas-api-costs/">Optimizing API Calls to Reduce SaaS Costs: Six Levers That Actually Move the Bill</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Apache Airflow on AWS: Building SaaS and API Pipelines That Don&#8217;t Lie to You</title>
		<link>https://john-nessime.com/blog/devops/apache-airflow-aws-saas-api-pipelines/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 05 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Airflow]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[MWAA]]></category>
		<category><![CDATA[Orchestration]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=145</guid>

					<description><![CDATA[<p>Most API pipeline failures are green DAGs producing incomplete data. A practical guide to running Apache Airflow on AWS for SaaS and API extraction: choosing between MWAA provisioned, MWAA Serverless and self-managed, the pool setting that silently stops throttling when you go deferrable, retry and pagination design, secrets handling, and the four cost lines that actually move.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/apache-airflow-aws-saas-api-pipelines/">Apache Airflow on AWS: Building SaaS and API Pipelines That Don&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually lands on a Monday: &#8220;the CRM numbers look wrong again.&#8221; Not missing. Wrong. The dashboard populated, every DAG run is green, and somewhere in the middle of last week&#8217;s data there is a hole where a paginated API returned a 429 and the task treated the empty body as a legitimate final page.</p>



<p class="wp-block-paragraph">That is the shape of most API pipeline incidents. Not a crash. A success that isn&#8217;t one.</p>



<p class="wp-block-paragraph">This post covers running Apache Airflow on AWS specifically for SaaS and API workloads: pulling from HubSpot, Salesforce, Stripe, Zendesk, Shopify, an internal partner API, whatever. It is organised by failure family rather than by feature, because the Airflow documentation already explains what an operator is and does a poor job of explaining which of these things will page you at 3am. I will cover choosing a deployment model, the concurrency trap that catches almost everyone, retry design, incremental state, secrets, and where the money actually goes.</p>



<h2 class="wp-block-heading">Why SaaS and API sources break differently</h2>



<p class="wp-block-paragraph">When your source is a database you control, failure is loud: connection refused, deadlock, disk full. When it is somebody else&#8217;s SaaS API, three things change.</p>



<ul class="wp-block-list">
<li><strong>You are a guest.</strong> The vendor decides your rate limit, and they can change it without telling you. Your pipeline&#8217;s correctness now depends on a number in someone else&#8217;s config file.</li>

<li><strong>Errors arrive as valid HTTP.</strong> A 429, a 200 with a truncated page, a 200 with an error object in the body. Your HTTP client is happy. Your data is not.</li>

<li><strong>Tasks spend most of their life waiting.</strong> API extraction is I/O bound almost end to end. That sounds harmless and is the root of the most expensive mistakes.</li>
</ul>



<h2 class="wp-block-heading">Pick the deployment model before you write a DAG</h2>



<p class="wp-block-paragraph">This decision constrains everything after it and is harder to reverse than people expect. Three realistic options.</p>



<h3 class="wp-block-heading">Amazon MWAA, provisioned</h3>



<p class="wp-block-paragraph">AWS runs the scheduler, web server, workers, triggerer and metadata database on Fargate; you drop DAGs into an S3 bucket and they get picked up.</p>



<p class="wp-block-paragraph">Where it wins: real Airflow, custom providers, custom plugins, full control over environment configuration. If your DAGs need arbitrary Python libraries, this option will not fight you.</p>



<p class="wp-block-paragraph">Where it doesn&#8217;t: the environment bills by the hour whether or not anything is running. There is no scale to zero on the base environment. If you sync six APIs once a day and each run takes twenty minutes, you are paying for a mostly idle cluster around the clock. The <code>mw1.micro</code> class exists precisely for the small case, but it collapses the scheduler and worker into a single Fargate task and caps worker autoscale low, so treat it as a dev or isolation tier rather than a cheap production tier.</p>



<h3 class="wp-block-heading">Amazon MWAA Serverless</h3>



<p class="wp-block-paragraph">You submit workflow definitions and AWS runs each task in its own Fargate container, billing per task duration with a one-minute minimum rather than per environment hour.</p>



<p class="wp-block-paragraph">Where it wins: spiky or infrequent schedules. If the workload is &#8220;six syncs a day, nothing overnight,&#8221; the cost profile beats a permanently running environment by a wide margin. Each workflow also gets its own IAM execution role, which is a real security improvement over one shared role per environment.</p>



<p class="wp-block-paragraph">Where it doesn&#8217;t: it leans on declarative YAML workflow definitions based on the DAG Factory format and a curated set of AWS operators. That is a deliberate trade: because the definition is declarative, the service can schedule tasks without executing your DAG code. It also means custom operators, exotic third-party providers and clever Python at parse time are not the sweet spot. It is also available in fewer regions than provisioned MWAA, so check your region before you design around it.</p>



<h3 class="wp-block-heading">Self-managed on ECS, EKS or a VPS</h3>



<p class="wp-block-paragraph">On EKS with the Kubernetes executor you get per-task pods and tight cost control. On a single VPS from a provider like InterServer or Hetzner, a Docker Compose stack with a Postgres metadata database will run a modest set of API syncs for a fraction of any managed price.</p>



<p class="wp-block-paragraph">Where it wins: cost at both extremes, and total control. Where it doesn&#8217;t: you now own metadata database upgrades, major version migrations, log retention and the 2am scheduler restart. Managed Airflow is a bet that your time is worth more than the hourly premium. For a solo engineer with three pipelines that bet often loses; for a data team of eight it usually wins. Astronomer is the main non-AWS managed option worth pricing alongside these.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: the throttle that silently stops throttling</h2>



<p class="wp-block-paragraph">You start with a normal setup: an Airflow pool named <code>crm_api</code> with four slots, and every task that touches the vendor assigned to it. Four concurrent requests, comfortably under the vendor&#8217;s limit. This works.</p>



<p class="wp-block-paragraph">Then you notice those tasks spend nearly all their runtime waiting on HTTP, burning worker slots to sit still. So you switch them to deferrable operators. A deferrable task suspends itself while waiting, releases its worker slot, and hands the waiting to the triggerer, which polls asynchronously. Worker pressure drops. Everything looks better.</p>



<p class="wp-block-paragraph">And your rate limiting quietly stops working.</p>



<p class="wp-block-paragraph">By default, a pool does not count tasks in the <em>deferred</em> state as occupying slots. That was deliberate, and the logic is sound in the abstract: a deferred task is not consuming a worker. But if you were using the pool to protect an external API rather than your own workers, it has just stopped doing the job you gave it. Every task can defer at once, and the vendor sees the full fan-out.</p>



<p class="wp-block-paragraph">The fix is a per-pool flag, <code>include_deferred</code>, which tells the scheduler to count deferred tasks against the slot budget. It is off by default. You can set it when editing the pool in the Airflow UI, or through the API.</p>



<p class="wp-block-paragraph">The failure signature is what makes this nasty. Nothing errors. Your DAG gets faster. The vendor starts returning 429s that your retry logic absorbs, and the only symptom is that runs take a little longer and occasionally a page goes missing. Weeks can pass. Two related traps in the same family:</p>



<ul class="wp-block-list">
<li><code>max_active_tasks</code> at the DAG level has the same blind spot with deferred tasks, and there is no equivalent opt-in flag. If you need a hard external concurrency cap, use a pool with <code>include_deferred</code> enabled, not DAG-level concurrency.</li>

<li>On MWAA, the triggerer runs alongside the scheduler on the same Fargate task, so scheduler count and triggerer capacity are linked. If you go heavily deferrable and your deferred tasks start stalling, scheduler capacity is the thing to look at.</li>
</ul>



<h2 class="wp-block-heading">Failure family two: retries that make the outage worse</h2>



<p class="wp-block-paragraph">The default instinct is to set <code>retries</code> high and move on. Against a rate-limited API, a fixed retry delay across many parallel tasks is just a slower version of the same stampede.</p>



<p class="wp-block-paragraph">What you want is exponential backoff with a ceiling. The shape:</p>



<pre class="wp-block-code"><code>from datetime import timedelta

from airflow.sdk import dag, task

@dag(
    schedule="0 5 * * *",
    catchup=False,
    max_active_runs=1,          # never let two runs of this DAG overlap
    default_args={
        "retries": 5,
        "retry_delay": timedelta(seconds=30),
        "retry_exponential_backoff": True,   # 30s, 60s, 120s, 240s...
        "max_retry_delay": timedelta(minutes=15),  # stop doubling here
        "pool": "crm_api",      # shared budget across every task touching this vendor
    },
    tags=["crm", "extract"],
)
def crm_extract():

    @task(max_active_tis_per_dag=4)
    def fetch_page(page_token: str) -&gt; str:
        ...

crm_extract()</code></pre>



<p class="wp-block-paragraph">The lines that matter:</p>



<ul class="wp-block-list">
<li><code>retry_exponential_backoff</code> turns <code>retry_delay</code> into a base rather than a constant, so repeated failures spread out instead of hammering in lockstep.</li>

<li><code>max_retry_delay</code> caps the doubling. Without it, a task that fails five times can sit idle for hours and blow past the window you actually cared about.</li>

<li><code>max_active_runs=1</code> is the one people skip. If a run overruns its schedule, the next one starts anyway, and now two runs are fetching the same pages from the same vendor with the same credentials. This is a common way to trigger a rate limit you have never hit before.</li>

<li><code>max_active_tis_per_dag</code> limits how many instances of that specific task run concurrently across DAG runs, which is the right knob for dynamically mapped extraction tasks.</li>
</ul>



<p class="wp-block-paragraph">One thing Airflow will not do for you: honour a <code>Retry-After</code> header. Airflow&#8217;s retry timing is computed from your config, not from the vendor&#8217;s response. If the API tells you exactly how long to wait, you have to catch that in your own code and sleep or reschedule accordingly. Ignoring a header the vendor bothered to send is a good way to get your API key throttled harder.</p>



<h2 class="wp-block-heading">Failure family three: pagination, cursors and the empty page</h2>



<p class="wp-block-paragraph">Back to the Monday message. The specific bug behind most &#8220;the numbers are wrong but nothing failed&#8221; incidents is a loop that treats any non-error response as a terminating condition. Three rules prevent it:</p>



<ol class="wp-block-list">
<li><strong>Never infer &#8220;done&#8221; from an empty result.</strong> Terminate on the explicit signal the API gives you: a null <code>next_cursor</code>, a missing <code>Link</code> header, a page count. An empty array with a valid cursor still has more data behind it.</li>

<li><strong>Assert the response shape before you use it.</strong> Check the status code explicitly and validate that the fields you depend on exist. A 200 carrying <code>{"error": "..."}</code> should raise, not return zero rows.</li>

<li><strong>Land raw, transform later.</strong> Write the untouched API response to S3 first, then parse from S3. When the vendor changes a field type, you can replay from raw instead of re-extracting from an API that no longer serves that window.</li>
</ol>



<h3 class="wp-block-heading">Where to keep incremental state</h3>



<p class="wp-block-paragraph">The tempting pattern is to store the last-seen timestamp in an Airflow Variable and update it at the end of a run. Do not make that your source of truth. If a run dies midway, the Variable is in an undefined state, and clearing and re-running the DAG will not restore it. Airflow&#8217;s retry and backfill machinery has no idea it exists.</p>



<p class="wp-block-paragraph">Better: make each run&#8217;s window a function of the run itself, and write output to a deterministic, run-scoped location such as <code>s3://bucket/source=crm/dt=&lt;logical-date&gt;/</code>. Re-running the same interval overwrites the same prefix. That is what makes a task idempotent, and idempotency is the difference between &#8220;clear the task and let it rerun&#8221; and a two-hour manual repair.</p>



<p class="wp-block-paragraph">Then overlap your windows deliberately. Many SaaS APIs order results by <em>modified</em> time with eventual consistency, so a record edited at the boundary can appear after you have already moved on. Query a window slightly wider than your schedule interval and rely on an idempotent upsert downstream to absorb the duplicates. Late-arriving data is not an edge case with SaaS sources. It is the normal case.</p>



<p class="wp-block-paragraph">Airflow&#8217;s asset-based scheduling is the clean way to trigger downstream DAGs from this: the extract DAG produces an asset, and the transform DAG runs when the asset updates, rather than being scheduled at a time you hope is late enough.</p>



<h2 class="wp-block-heading">Failure family four: credentials</h2>



<p class="wp-block-paragraph">API tokens rotate, sometimes on the vendor&#8217;s schedule rather than yours. Storing an API key in an Airflow Connection through the UI works, and is the wrong long-term answer: the value lives in the metadata database and there is no rotation story. On AWS, point Airflow&#8217;s secrets backend at AWS Secrets Manager. On MWAA that is an environment configuration option:</p>



<pre class="wp-block-code"><code>secrets.backend
  airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend

secrets.backend_kwargs
  {"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables"}</code></pre>



<p class="wp-block-paragraph">With that in place, a connection lookup for <code>crm_default</code> resolves against the secret at <code>airflow/connections/crm_default</code>. Rotating the credential is a Secrets Manager operation with no Airflow deployment involved.</p>



<p class="wp-block-paragraph">Two things to know before you turn it on. First, every connection and variable lookup becomes a Secrets Manager API call, and lookups fall through to the backend before hitting the metadata database, so a DAG that reads a Variable at parse time will generate a call on every parse cycle. Move those reads inside tasks. Second, the environment&#8217;s execution role needs explicit read permission on the relevant secret ARNs, and if you use a customer-managed KMS key, decrypt permission on that key too.</p>



<p class="wp-block-paragraph">Worth knowing if you are on Airflow 3: task code can no longer reach the metadata database directly. All runtime interaction goes through the Task Execution API. If you inherited custom operators that open a session and query Airflow&#8217;s own tables, that is a migration blocker, not a warning.</p>



<h2 class="wp-block-heading">Failure family five: the bill</h2>



<p class="wp-block-paragraph">Nobody is surprised by the environment line item. They are surprised by the other four.</p>



<ul class="wp-block-list">
<li><strong>Idle time.</strong> A provisioned MWAA environment bills continuously. Compute the ratio of hours billed to hours doing work. If it is bad, that is the argument for MWAA Serverless or for consolidating several thin pipelines into one environment.</li>

<li><strong>NAT Gateway.</strong> This is the classic one. Private-subnet workers calling public SaaS APIs route through a NAT Gateway, which charges hourly <em>and</em> per gigabyte processed. A high-volume extraction pipeline can spend more on NAT than on Airflow. VPC endpoints remove that cost for AWS service traffic, but they do nothing for calls to a third-party API, which is exactly the traffic an API pipeline generates.</li>

<li><strong>CloudWatch Logs.</strong> Task logs go to CloudWatch, and ingestion is billed per gigabyte. Set the Airflow log level per component rather than globally at DEBUG, and set a retention policy on the log groups. The default is to keep logs forever.</li>

<li><strong>S3 requests.</strong> Landing raw API responses one small object per page generates a lot of PUTs. Batch pages into larger objects where you can.</li>
</ul>



<p class="wp-block-paragraph">Rates and dimensions change, so model your own workload against the current pricing page rather than trusting a number from a blog post. The point is knowing which four lines to look at.</p>



<h2 class="wp-block-heading">Troubleshooting Apache Airflow on AWS when API pipelines misbehave</h2>



<h3 class="wp-block-heading">Tasks sit in &#8220;queued&#8221; and never start</h3>



<p class="wp-block-paragraph">Usually a slot problem, not a broken scheduler. Check, in order: is the pool full; has DAG-level <code>max_active_tasks</code> been hit; is worker autoscaling at its configured maximum. On MWAA, the container and queue utilisation metrics published to CloudWatch tell you which of the three it is far faster than reading scheduler logs.</p>



<h3 class="wp-block-heading">DAG file changes don&#8217;t appear</h3>



<p class="wp-block-paragraph">On MWAA, DAGs sync from S3 on an interval; it is not instant. If a file has been there for several minutes and still hasn&#8217;t appeared, it almost always failed to parse. Check the DAG processing logs in CloudWatch rather than the scheduler logs, because a broken import raises there and never reaches the scheduler.</p>



<h3 class="wp-block-heading">A new provider package won&#8217;t install</h3>



<p class="wp-block-paragraph">MWAA installs from your <code>requirements.txt</code> in the DAGs bucket, and from Airflow 2.7.2 onward that file must include a constraint line. Without one, MWAA picks a constraint for you, and pip is free to resolve a provider version that conflicts with the Airflow build in the image.</p>



<pre class="wp-block-code"><code>--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-&lt;AIRFLOW_VERSION&gt;/constraints-&lt;PYTHON_VERSION&gt;.txt"

apache-airflow-providers-http
apache-airflow-providers-salesforce</code></pre>



<p class="wp-block-paragraph">Substitute the literal Airflow version your environment runs and the Python version bundled with it. MWAA does not expand shell variables in that file. Test the requirements file against a local Airflow image before you upload it, because a failed install on MWAA surfaces as a partially working environment rather than a clean error.</p>



<h3 class="wp-block-heading">A backfill is stuck and you need to clear it</h3>



<p class="wp-block-paragraph">You do not need a web login token for this. MWAA exposes the Airflow REST API through a signed AWS API call, so you can drive it from CI or a runbook with normal IAM credentials:</p>



<pre class="wp-block-code"><code>aws mwaa invoke-rest-api 
  --name MyMWAAEnvironment 
  --path "/dags/crm_extract/clearTaskInstances" 
  --method POST 
  --body '{"dry_run": true}'</code></pre>



<p class="wp-block-paragraph">Start with <code>dry_run</code> set to true so the response tells you which task instances would be cleared before you actually clear them. Note that the resource paths differ between Airflow 2 and Airflow 3 environments, so confirm against the API version your environment exposes.</p>



<h3 class="wp-block-heading">Deferred tasks stall forever</h3>



<p class="wp-block-paragraph">If deferred tasks stop resuming while the environment reports healthy, suspect the triggerer rather than your DAG. A triggerer that has lost its ability to process triggers can keep heartbeating normally, so the scheduler sees nothing wrong while every deferred task drifts toward timeout. This class of bug has been fixed and re-fixed upstream, so check your Airflow version&#8217;s release notes before assuming it is your code.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Switching to deferrable operators without enabling <code>include_deferred</code> on the pools that were protecting the API.</li>

<li>Leaving <code>max_active_runs</code> unset, so a slow run and the next scheduled run compete for the same rate limit budget.</li>

<li>Treating an empty response page as the end of pagination.</li>

<li>Storing the incremental watermark in an Airflow Variable and updating it mid-run.</li>

<li>Calling an API or reading a Variable at DAG parse time, which executes on every parse cycle rather than once per run.</li>

<li>Transforming during extraction, so a vendor schema change means re-pulling data the API may no longer serve.</li>

<li>Sizing the environment for peak concurrency when the actual constraint is the vendor&#8217;s rate limit.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>One pool per vendor, sized to their published limit with headroom, and <code>include_deferred</code> enabled on every one of them.</li>

<li>Land raw responses to S3 before parsing. Extraction and transformation are separate tasks with separate failure modes.</li>

<li>Make every task idempotent and window-scoped, so &#8220;clear and rerun&#8221; is always a safe repair.</li>

<li>Overlap extraction windows and deduplicate downstream rather than trusting a vendor&#8217;s timestamps to be exact.</li>

<li>Secrets Manager for credentials, with the execution role scoped to specific secret ARNs.</li>

<li>Alert on row counts and freshness, not just task state. A green DAG that produced 40% of yesterday&#8217;s rows is the failure you actually care about. Shipping Airflow&#8217;s StatsD metrics into Prometheus, Grafana Cloud or Datadog makes that a dashboard rather than a discovery.</li>

<li>Define the environment in Terraform or OpenTofu. Recreating an MWAA environment by hand after a bad configuration change is a bad afternoon.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is MWAA worth it compared to self-hosting Airflow on EC2?</h3>



<p class="wp-block-paragraph">It depends almost entirely on how many people share the platform. MWAA&#8217;s premium buys you managed metadata database upgrades, patched images and version migration support. If one engineer maintains three DAGs, self-hosting on a modest VPS is cheaper and the operational load is real but small. Once several teams depend on the scheduler being up, the premium is easy to justify.</p>



<h3 class="wp-block-heading">Should I use Step Functions instead of Airflow for API pipelines?</h3>



<p class="wp-block-paragraph">Step Functions is genuinely better for event-driven, AWS-service-centric orchestration with modest branching, and it scales to zero. Airflow wins when you need scheduled batch semantics, backfills over historical windows, dependencies between many pipelines, and a UI that non-platform engineers can use to see why last Tuesday failed. Backfill is usually the deciding feature.</p>



<h3 class="wp-block-heading">Do deferrable operators reduce my AWS bill?</h3>



<p class="wp-block-paragraph">On provisioned MWAA, they reduce worker <em>pressure</em>, which reduces autoscaling into additional worker instances. The base environment cost is unchanged. On a Kubernetes executor setup where each task is a pod, the saving is more direct. Either way, do not adopt them purely for cost without revisiting your pool configuration first.</p>



<h3 class="wp-block-heading">How do I handle a vendor with no documented rate limit?</h3>



<p class="wp-block-paragraph">Start conservative, one or two concurrent requests, and instrument the response status codes. Raise the pool size gradually and watch for 429s or rising latency. Latency creeping up under load is often the earlier signal, because some vendors throttle by slowing you down before they start rejecting.</p>



<h3 class="wp-block-heading">Can Airflow read a Retry-After header automatically?</h3>



<p class="wp-block-paragraph">No. Airflow computes retry timing from <code>retry_delay</code> and the backoff settings on the task. If a vendor sends <code>Retry-After</code>, you need to handle it in your own request code or in a custom operator.</p>



<h3 class="wp-block-heading">What breaks when upgrading to Airflow 3?</h3>



<p class="wp-block-paragraph">The big one for API pipelines is that task code can no longer access the metadata database directly; everything goes through the Task Execution API. Imports also move to the <code>airflow.sdk</code> namespace, and several core operators now live in the standard provider package. Audit custom operators first, since that is where direct database access hides. MWAA requires you to be on the latest Airflow 2 minor version before a major upgrade, so plan two steps.</p>



<h3 class="wp-block-heading">How many DAGs can one MWAA environment handle?</h3>



<p class="wp-block-paragraph">The binding constraint is usually the metadata database and scheduler CPU, not DAG count. Watch metadata database memory and scheduler CPU utilisation; when either saturates, you either move up an environment class or split into multiple environments. Splitting also gives you blast-radius isolation, which matters more than people expect.</p>



<h2 class="wp-block-heading">Wrapping up</h2>



<p class="wp-block-paragraph">Running Apache Airflow on AWS for SaaS and API pipelines is mostly not an Airflow problem. The scheduler works. The operators work. What bites is the gap between &#8220;the task succeeded&#8221; and &#8220;the data is correct,&#8221; and that gap lives in concurrency settings, pagination logic and retry design rather than anywhere Airflow will warn you about.</p>



<p class="wp-block-paragraph">If you take one thing away: <strong>a green DAG is not a signal that your data is complete.</strong> Enable <code>include_deferred</code> on the pools protecting your vendors, terminate pagination on an explicit signal instead of an empty page, make every task idempotent, and alert on row counts. Those four things prevent most of the incidents that never show up as a failed task.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with your Airflow pipelines on AWS?</h2>



<p class="wp-block-paragraph">I work with teams running data and API pipelines on AWS, usually somewhere between &#8220;it works but nobody trusts it&#8221; and &#8220;we need to move off cron.&#8221; Things I can help with:</p>



<ul class="wp-block-list">
<li>Reviewing existing DAGs for silent data loss: pagination logic, retry behaviour, pool and concurrency configuration.</li>

<li>Choosing between MWAA provisioned, MWAA Serverless and self-managed Airflow, with a cost model for your actual schedule rather than a generic comparison.</li>

<li>Building SaaS extraction pipelines that are idempotent and safely re-runnable, landing raw to S3 with incremental windows that survive failure.</li>

<li>Cutting MWAA cost: environment right-sizing, NAT Gateway traffic, CloudWatch log volume and dependency install time.</li>

<li>Airflow 2 to 3 migration audits, focused on custom operators and direct metadata database access.</li>

<li>Data freshness and volume alerting in Grafana or CloudWatch, so you learn about a partial sync before the business does.</li>
</ul>



<p class="wp-block-paragraph">If something specific is broken, send me the DAG file, the task log, or the CloudWatch metrics for the run that went wrong. It is usually faster to look at the real thing than to describe it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/apache-airflow-aws-saas-api-pipelines/">Apache Airflow on AWS: Building SaaS and API Pipelines That Don&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a Jira Analytics Pipeline with AWS Lambda and Athena (Without Double-Counting Everything)</title>
		<link>https://john-nessime.com/blog/devops/jira-analytics-pipeline-aws-lambda-athena/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 04 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Business Intelligence]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Engineering Metrics]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Jira]]></category>
		<category><![CDATA[Partition Projection]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[SQL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=125</guid>

					<description><![CDATA[<p>Jira's built-in reports stop at the board boundary. This guide walks through a Jira analytics pipeline built on AWS Lambda, S3 and Athena, organised around the four failure families that actually bite: the removed search endpoint, silently truncated changelogs, incremental loads that duplicate rows, and an S3 layout that quietly inflates your query bill.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/jira-analytics-pipeline-aws-lambda-athena/">Building a Jira Analytics Pipeline with AWS Lambda and Athena (Without Double-Counting Everything)</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone in a delivery review asks why cycle time went up last quarter. You open the Jira dashboard, and it can tell you what is in progress right now, roughly. It cannot tell you what &#8220;in progress&#8221; meant three months ago, how long each ticket sat in code review, or whether the increase came from one team or all six. The control chart resets when the board configuration changes, the sprint report only knows about sprints, and the CSV export tops out somewhere unhelpful.</p>



<p class="wp-block-paragraph">That is usually the moment someone says &#8220;let&#8217;s just pull it into a warehouse.&#8221; A <strong>Jira analytics pipeline</strong> built on AWS Lambda, S3 and Athena is a reasonable answer to that, and it is genuinely cheap to run. It is also easy to build a version that looks correct for two weeks and then quietly reports numbers that are thirty percent wrong.</p>



<p class="wp-block-paragraph">This post covers the extraction and modelling problems that actually cost you time: the search endpoint Atlassian removed, the change history that truncates without erroring, the incremental load pattern that duplicates rows across partitions, and the S3 layout decisions that decide whether Athena costs you pennies or hundreds. Code where it clarifies something, and honest notes on what I would skip.</p>



<h2 class="wp-block-heading">Where Jira&#8217;s own reporting genuinely stops</h2>



<p class="wp-block-paragraph">Give the built-in tooling its due first. Jira&#8217;s velocity, burndown and control charts are fine for a single team inspecting its own recent work, they need no infrastructure, and they update instantly. Marketplace apps like eazyBI and Custom Charts cover a lot of ground without you writing a line of Python. If your question is &#8220;how did this sprint go,&#8221; you do not need a pipeline.</p>



<p class="wp-block-paragraph">Where it stops is anything that crosses a boundary. Comparing lead time across projects that use different workflows. Joining ticket data to deploy events from your CI system or incident data from PagerDuty. Retaining a consistent view of history after someone renames a status or archives a board. Answering a question nobody anticipated when the board was configured. Those need the raw data somewhere you control, in a shape you decide.</p>



<h2 class="wp-block-heading">The shape of the pipeline</h2>



<p class="wp-block-paragraph">The architecture is unremarkable, which is the point:</p>



<ol class="wp-block-list">
<li>EventBridge Scheduler triggers a Lambda function on a schedule.</li>

<li>Lambda reads a Jira API token from Secrets Manager and pages through the Jira Cloud REST API.</li>

<li>It writes Parquet files to S3, partitioned by load date.</li>

<li>The Glue Data Catalog holds the table definitions, with partition projection so nothing has to crawl.</li>

<li>Athena queries S3 directly. Grafana, Power BI, Metabase or QuickSight sit on top of Athena.</li>
</ol>



<p class="wp-block-paragraph">No cluster, no always-on database, nothing to patch. The whole thing costs about as much as a small EC2 instance for a mid-sized Jira site, and most of that is S3 storage. Terraform or CloudFormation to define it, GitHub Actions to deploy it.</p>



<p class="wp-block-paragraph">The complexity is not in the wiring. It is in four places, and they are worth taking in order.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family 1: the search endpoint you were probably going to use is gone</h2>



<p class="wp-block-paragraph">Almost every Jira extraction tutorial and a good number of client libraries still reach for <code>GET /rest/api/3/search</code>. Atlassian removed it from Jira Cloud. It returns 410 Gone. If you are copying a script from a blog post that predates the change, this is the first thing that breaks, and at least it breaks loudly.</p>



<p class="wp-block-paragraph">The replacement is <code>/rest/api/3/search/jql</code>, available as both GET and POST. Use POST for anything real, because JQL strings get long and you avoid URL encoding entirely. Three behavioural changes matter more than the URL:</p>



<h3 class="wp-block-heading">Pagination is cursor-based, and there is no total</h3>



<p class="wp-block-paragraph"><code>startAt</code> is gone. You get a <code>nextPageToken</code> back and hand it to the next request. There is no <code>total</code> in the response, which means any progress bar, any &#8220;expected N rows&#8221; sanity check, and any loop that terminated on <code>startAt &gt;= total</code> needs rewriting.</p>



<p class="wp-block-paragraph">If you only need a count, there is a separate operation, <code>POST /rest/api/3/search/approximate-count</code>, which takes a JQL body and returns an approximate figure without paging through results. It is genuinely useful as a reconciliation check: run it before extraction, compare against the row count you actually wrote, and alert on a large gap.</p>



<p class="wp-block-paragraph">There have been persistent community reports of <code>isLast</code> behaving unreliably on this endpoint, including tokens that chain without ever terminating. Do not trust <code>isLast</code> as your loop condition. Terminate on the absence of <code>nextPageToken</code>, and put a hard page cap in as a circuit breaker so a bad token cannot burn your entire Lambda budget in one invocation.</p>



<pre class="wp-block-code"><code>def search_issues(session, base_url, jql, fields, max_pages=2000):
    """Page through /search/jql. Terminates on missing nextPageToken,
    not on isLast, which has been reported as unreliable."""
    token = None
    for _ in range(max_pages):
        body = {"jql": jql, "fields": fields, "maxResults": 100}
        if token:
            body["nextPageToken"] = token
        r = session.post(f"{base_url}/rest/api/3/search/jql",
                         json=body, timeout=60)
        r.raise_for_status()
        page = r.json()
        for issue in page.get("issues", []):
            yield issue
        token = page.get("nextPageToken")
        if not token:
            return
    raise RuntimeError("page cap hit, refusing to loop further")</code></pre>



<h3 class="wp-block-heading">You have to ask for fields explicitly</h3>



<p class="wp-block-paragraph">The new endpoint does not hand you every field by default. Omit <code>fields</code> and you get essentially nothing back. This is the failure that looks like success: the pipeline runs, files land in S3, row counts look plausible, and every analytical column is null.</p>



<p class="wp-block-paragraph">Be explicit and be narrow. Every field you request costs response size, and Jira sites accumulate hundreds of custom fields nobody uses. Name what you need:</p>



<pre class="wp-block-code"><code>FIELDS = [
    "summary", "status", "issuetype", "project", "priority",
    "assignee", "reporter", "created", "updated", "resolutiondate",
    "labels", "components", "parent",
    "customfield_10016",   # story points on this site, verify yours
]</code></pre>



<p class="wp-block-paragraph">Custom field IDs are per-site. Do not hardcode one you read in someone else&#8217;s blog post. Pull <code>/rest/api/3/field</code> once, find the field by name, and either store the mapping in config or resolve it at runtime and log what it resolved to. When a Jira admin rebuilds a field, an ID-based pipeline goes null and a name-resolving pipeline keeps working.</p>



<h3 class="wp-block-heading">Rate limits and the fifteen-minute wall</h3>



<p class="wp-block-paragraph">Jira Cloud applies cost-based rate limiting and returns HTTP 429 when you exceed it, typically with a <code>Retry-After</code> header. Respect that header rather than inventing your own backoff. A naive retry loop that ignores it turns a brief throttle into a sustained one.</p>



<p class="wp-block-paragraph">Lambda&#8217;s hard ceiling is fifteen minutes. A full historical backfill of a large Jira site will not finish in one invocation, and the ugly failure mode is a function that times out at minute fifteen having written half its data with no record of where it stopped. Two ways out:</p>



<ul class="wp-block-list">
<li><strong>Shard the work.</strong> Fan out one Lambda invocation per project key, or per month of created date. Each one is small, independently retryable, and finishes well inside the limit.</li>

<li><strong>Checkpoint and continue.</strong> Persist the current <code>nextPageToken</code> to DynamoDB or S3 after each page. When the function is close to its deadline, stop cleanly and let Step Functions re-invoke it from the checkpoint.</li>
</ul>



<p class="wp-block-paragraph">Sharding is simpler and I reach for it first. Checkpointing is what you need when a single project is itself too large. Either way, watch Lambda&#8217;s ephemeral storage: the default <code>/tmp</code> allocation is 512 MB, and buffering a large Parquet write there will fail before your API calls do. Raise it or stream to S3 in chunks.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family 2: the change history is the whole point, and it truncates silently</h2>



<p class="wp-block-paragraph">Current issue state answers almost none of the interesting questions. How long a ticket spent waiting for review, how many times it bounced back from QA, when it actually entered development rather than when someone remembered to drag the card, all of that lives in the changelog. Without it you have a list of tickets. With it you have a process.</p>



<p class="wp-block-paragraph">Here is the trap. Requesting an issue with <code>expand=changelog</code> returns a capped number of history entries, commonly the first hundred, and it does not tell you it truncated. Well-worn tickets with lots of field edits blow past that easily. Your data does not error, it just quietly loses the later transitions, which are usually the ones near completion. Cycle time comes out looking better than reality.</p>



<p class="wp-block-paragraph">Two correct approaches:</p>



<ul class="wp-block-list">
<li><code>GET /rest/api/3/issue/{issueIdOrKey}/changelog</code> and page it properly. Correct, but it is one request per issue, which is brutal against rate limits on a large site.</li>

<li><code>POST /rest/api/3/changelog/bulkfetch</code>, which accepts <code>issueIdsOrKeys</code> and an optional <code>fieldIds</code> filter, with the same <code>nextPageToken</code> pagination. Far fewer round trips.</li>
</ul>



<p class="wp-block-paragraph">Bulk fetch is the one I would use, with a caveat: it has carried an experimental designation, so pin your expectations and keep the per-issue path available as a fallback. Filter <code>fieldIds</code> to <code>status</code> if status history is all you model. That cuts the response size enormously, because most changelog volume is description edits and label churn nobody analyses.</p>



<p class="wp-block-paragraph">Store changelog as its own narrow table, one row per field change, not nested inside the issue record. Athena can handle nested structures, but flat is dramatically easier to reason about in SQL and much cheaper to scan.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family 3: the incremental load that duplicates everything</h2>



<p class="wp-block-paragraph">This is the one that bites hardest, because nothing fails. Everything runs green for weeks and the numbers are wrong the entire time.</p>



<p class="wp-block-paragraph">The obvious incremental design is a watermark: track the last successful run, then pull <code>updated &gt;= watermark</code> on each run and append the results to S3. It is the standard pattern and it works for immutable event data.</p>



<p class="wp-block-paragraph">Jira issues are not immutable. A ticket created in January and touched every week appears in every weekly extract. If you append each run into its own partition, that one issue now exists as a row in twenty partitions. Athena is doing exactly what you asked: <code>SELECT count(*) FROM jira_issues WHERE status = 'Open'</code> counts twenty things. Your open bug count is inflated, your throughput is inflated, and the inflation is proportional to how much a ticket gets edited, which correlates with how contentious it was. The busiest work is the most over-counted.</p>



<p class="wp-block-paragraph">Nobody catches this from the dashboard, because the numbers move in the right direction and look approximately sane. It surfaces months later when someone reconciles against a JQL query in Jira and the two disagree.</p>



<h3 class="wp-block-heading">Pick one of three fixes, deliberately</h3>



<p class="wp-block-paragraph"><strong>Full snapshot per load, dedupe at read time.</strong> Keep every version, partition by <code>load_date</code>, and always read through a view that takes the latest row per issue key. Storage is cheap, history is free, and you can answer &#8220;what did the board look like in March&#8221; without any extra machinery. The cost is that every query pays for the deduplication.</p>



<pre class="wp-block-code"><code>CREATE OR REPLACE VIEW jira_issues_current AS
SELECT * FROM (
  SELECT
    i.*,
    ROW_NUMBER() OVER (
      PARTITION BY issue_key
      ORDER BY load_date DESC, updated DESC
    ) AS rn
  FROM jira_issue_snapshot i
  WHERE load_date &gt;= date_format(current_date - interval '7' day, '%Y-%m-%d')
) WHERE rn = 1;</code></pre>



<p class="wp-block-paragraph">The <code>load_date</code> filter inside the view matters. Without it the deduplication window scans the entire table on every query, which is the single most common way a cheap Athena setup becomes an expensive one.</p>



<p class="wp-block-paragraph"><strong>Overwrite the affected partitions.</strong> Partition by something stable, usually issue created month, and rewrite whole partitions when any issue in them changes. Clean reads, no dedupe cost, but you now own read-modify-write logic in Lambda and a concurrency problem if two runs overlap.</p>



<p class="wp-block-paragraph"><strong>Use an ACID table format.</strong> Apache Iceberg gives you real <code>MERGE INTO</code> semantics on S3, and Athena supports it natively. This is the right answer if you are already running Iceberg elsewhere or if the pipeline will grow to a dozen sources. It is not worth adopting solely to load one Jira site, because you inherit compaction and snapshot expiry as ongoing maintenance.</p>



<p class="wp-block-paragraph">For a single Jira site feeding a handful of dashboards, snapshot plus a dedupe view is what I would build. It has the fewest moving parts and it gives you point-in-time history as a side effect, which you will want the first time someone asks a retrospective question.</p>



<h3 class="wp-block-heading">Deletions and moves</h3>



<p class="wp-block-paragraph">A JQL watermark query never returns deleted issues, so they persist in your data forever. Same for issues moved out of scope or into an archived project. Periodically reconcile: pull the full set of issue keys with a minimal <code>fields</code> list, compare against what you hold, and mark the difference. Monthly is usually enough. Skip this and your historical counts drift upward permanently.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family 4: the S3 layout that decides your Athena bill</h2>



<p class="wp-block-paragraph">Athena&#8217;s standard pricing model bills on bytes scanned, rounded up, with a small per-query minimum. DDL statements are free. That means your storage layout, not your SQL, is what determines cost.</p>



<p class="wp-block-paragraph">Three things do almost all the work:</p>



<ul class="wp-block-list">
<li><strong>Write Parquet, not JSON.</strong> Columnar storage lets Athena read only the columns your query touches. A dashboard selecting five columns from a forty-column table scans a small fraction of what the equivalent JSON would. Snappy or Zstd compression on top of that.</li>

<li><strong>Avoid tiny files.</strong> A Lambda that runs hourly and writes one small file per invocation produces thousands of objects. Athena spends more time opening files than reading them. Batch writes so files land in the low hundreds of megabytes, or run a periodic compaction job.</li>

<li><strong>Use partition projection.</strong> This is the one people skip and then wonder why queries have a fixed few-second overhead.</li>
</ul>



<p class="wp-block-paragraph">Partition projection lets Athena calculate partition locations from table properties instead of calling <code>GetPartitions</code> against the Glue Data Catalog. On a table with a couple of years of daily partitions, that lookup is real latency on every single query, and it grows as the table does. Projection removes it, and removes your need for a Glue crawler entirely, which is one less scheduled job and one less IAM role.</p>



<pre class="wp-block-code"><code>CREATE EXTERNAL TABLE jira_issue_snapshot (
  issue_id        string,
  issue_key       string,
  project_key     string,
  issue_type      string,
  status          string,
  status_category string,
  assignee_id     string,
  created         timestamp,
  updated         timestamp,
  resolutiondate  timestamp,
  story_points    double
)
PARTITIONED BY (load_date string)
STORED AS PARQUET
LOCATION 's3://example-jira-lake/issue_snapshot/'
TBLPROPERTIES (
  'projection.enabled'                = 'true',
  'projection.load_date.type'         = 'date',
  'projection.load_date.format'       = 'yyyy-MM-dd',
  'projection.load_date.range'        = 'NOW-3YEARS,NOW',
  'projection.load_date.interval'     = '1',
  'projection.load_date.interval.unit'= 'DAYS',
  'storage.location.template'         =
    's3://example-jira-lake/issue_snapshot/load_date=${load_date}'
);</code></pre>



<p class="wp-block-paragraph">One caution that surprises people: projection describes partitions Athena will look for, not partitions that exist. Set a range wider than your data and queries without a <code>load_date</code> filter will probe empty prefixes. Keep the range tight to what you actually hold.</p>



<p class="wp-block-paragraph">Finally, set <code>BytesScannedCutoffPerQuery</code> on the Athena workgroup. It kills any query that exceeds a scan threshold. One analyst running <code>SELECT *</code> against three years of data in a BI tool&#8217;s preview pane is the classic surprise line item, and this stops it at the source.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Modelling time in status</h2>



<p class="wp-block-paragraph">Once status changelog rows are landed flat, the core metric is a window function. Each transition&#8217;s duration is the gap to the next transition on the same issue:</p>



<pre class="wp-block-code"><code>WITH transitions AS (
  SELECT
    issue_key,
    to_status,
    changed_at,
    LEAD(changed_at) OVER (
      PARTITION BY issue_key ORDER BY changed_at
    ) AS next_changed_at
  FROM jira_changelog
  WHERE field_id = 'status'
    AND load_date &gt;= date_format(current_date - interval '90' day, '%Y-%m-%d')
)
SELECT
  issue_key,
  to_status,
  SUM(date_diff('second', changed_at,
                COALESCE(next_changed_at, current_timestamp))) / 3600.0
    AS hours_in_status
FROM transitions
GROUP BY issue_key, to_status;</code></pre>



<p class="wp-block-paragraph">The <code>COALESCE</code> handles the current status, which has no successor transition. Two modelling decisions to make consciously: whether to subtract non-working hours, and how to treat a ticket that moves backwards through the workflow. Both are business questions, not technical ones, and both should be settled in a documented view rather than reimplemented in each dashboard.</p>



<p class="wp-block-paragraph">Map raw status names to a stable category early. Teams rename statuses constantly, and a metric keyed on the literal string breaks the moment someone changes &#8220;In Review&#8221; to &#8220;Peer Review&#8221;. A small mapping table joined at query time keeps history comparable.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Every analytical column is null.</strong> You did not pass <code>fields</code>, or you passed a custom field ID from another site. Log the resolved field list on every run.</li>

<li><strong>Counts higher than the same JQL in Jira.</strong> Duplicate rows across partitions. Check whether your query goes through the dedupe view or straight at the base table.</li>

<li><strong>Extraction loops forever.</strong> Do not terminate on <code>isLast</code>. Terminate on missing <code>nextPageToken</code> and keep a page cap.</li>

<li><strong>HTTP 410 from the API.</strong> You are still calling the removed <code>/rest/api/3/search</code>. Check your client library version too, not just your own code.</li>

<li><strong>Cycle times suspiciously low.</strong> Truncated changelog. Move to the dedicated changelog endpoint or bulk fetch.</li>

<li><strong>Athena returns zero rows but the files are there.</strong> Partition projection range does not cover the partition, or the S3 prefix does not match <code>storage.location.template</code>. Compare a real object key against the template character by character.</li>

<li><strong>Queries slow down as the table grows.</strong> Either you have no projection and Glue lookups dominate, or you have accumulated small files.</li>

<li><strong>Authentication failures overnight with no deploy.</strong> API token expired or was revoked. Alert on the specific status code rather than on &#8220;run failed&#8221;.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Appending incremental extracts without a deduplication strategy.</li>

<li>Building on current issue state and adding changelog later. Retrofitting history is far more work than including it from the start.</li>

<li>Hardcoding custom field IDs copied from documentation.</li>

<li>Storing the API token in a Lambda environment variable instead of Secrets Manager.</li>

<li>Running an hourly schedule when the dashboard is read once a day. You pay in small files and API quota for freshness nobody uses.</li>

<li>Letting BI tools query base tables directly instead of curated views.</li>

<li>Never reconciling against Jira. A scheduled check comparing approximate count to your row count catches drift within a day.</li>
</ul>



<h2 class="wp-block-heading">Best practices for a Jira analytics pipeline that survives</h2>



<ul class="wp-block-list">
<li>Land raw API responses to S3 before transforming. When your parsing is wrong, and it will be, you replay from raw rather than re-hammering the API.</li>

<li>Make the load idempotent. Re-running for the same date should produce the same result, not a second copy.</li>

<li>Define the pipeline in Terraform or CloudFormation and deploy it from CI. The IAM policy for Lambda and Athena is fiddly and you do not want to rebuild it from memory.</li>

<li>Alert on a run that succeeds with zero rows, not just on runs that error. Silent empty loads are the more common failure.</li>

<li>Scope the Jira token to a service account with read access to exactly the projects you need.</li>

<li>Put an S3 lifecycle policy on the raw zone. It grows faster than you expect and nobody queries last year&#8217;s raw JSON.</li>

<li>Expose curated views, not tables, to Grafana, Metabase, Power BI or QuickSight, and enable Athena&#8217;s query result reuse for repeated dashboard loads.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Should I use Lambda or Glue for Jira extraction?</h3>



<p class="wp-block-paragraph">Lambda, for an API-driven pull. The work is mostly waiting on HTTP responses, which Spark&#8217;s distributed compute does nothing for, and you would be paying for a Glue job&#8217;s minimum billing on something that is idle. Glue earns its place downstream, if you have heavy joins across several sources. Athena CTAS often covers that too.</p>



<h3 class="wp-block-heading">How often should the pipeline run?</h3>



<p class="wp-block-paragraph">Match the decision cadence, not the data cadence. Delivery metrics are reviewed weekly or in sprint ceremonies, so daily is almost always enough and produces far better file sizes. Reserve hourly for something with a real-time consumer, and know that you are buying that freshness with small-file overhead and API quota.</p>



<h3 class="wp-block-heading">Can I use webhooks instead of polling?</h3>



<p class="wp-block-paragraph">You can, and for near-real-time reaction it is the right tool. For analytics it is a poor primary source, because a missed delivery leaves a permanent hole you have no way to detect. The pattern that works is webhooks for freshness plus a scheduled reconciliation pull as the source of truth. If you only build one, build the scheduled pull.</p>



<h3 class="wp-block-heading">Does this work with Jira Data Center or Server?</h3>



<p class="wp-block-paragraph">The AWS half is identical. The extraction half is not. Cursor pagination and the <code>/search/jql</code> endpoint are Cloud-only changes, so self-hosted instances still use the older offset-based <code>/rest/api/2/search</code>. Write the extractor behind an interface if you need to support both, and expect the auth model to differ as well.</p>



<h3 class="wp-block-heading">How much does an Athena-based Jira pipeline cost to run?</h3>



<p class="wp-block-paragraph">The mechanism matters more than any figure I could quote. You pay for S3 storage, Lambda invocation time, and Athena per byte scanned with a small per-query minimum. Jira issue data compresses extremely well as Parquet, so storage stays small. The variable is dashboard query volume, which is why partitioning and workgroup scan limits matter. Model it against current AWS rates and your own expected query count.</p>



<h3 class="wp-block-heading">Why not just use eazyBI or a Jira reporting app?</h3>



<p class="wp-block-paragraph">Often you should. If your questions stay inside Jira, a Marketplace app gets you there in an afternoon with no infrastructure. The case for a pipeline is joining Jira to data that lives elsewhere, retaining history the app does not, or needing your data in a warehouse you already run. Build the pipeline when the app has actually failed you, not in anticipation.</p>



<h3 class="wp-block-heading">Should I load into Redshift instead of querying S3?</h3>



<p class="wp-block-paragraph">Only if you have concurrent BI users hitting the same tables constantly and Athena&#8217;s per-query latency is a real complaint. Jira data volumes are small, the query pattern is bursty, and Athena&#8217;s zero-idle-cost model fits that far better. Redshift makes sense as a consolidation layer across many sources, not for one issue tracker.</p>



<h2 class="wp-block-heading">The one thing to carry away</h2>



<p class="wp-block-paragraph">The hard part of a Jira analytics pipeline is not the AWS wiring. Lambda, S3, Glue and Athena will be working within a day. The hard part is that Jira issues are mutable, so an incremental load that appends is an incremental load that duplicates, and it does so without a single error in your logs.</p>



<p class="wp-block-paragraph">Decide your deduplication strategy before you write the first extract, not after someone reconciles a dashboard against JQL and finds a gap. Snapshot with a read-time dedupe view, partition projection so queries stay fast as history accumulates, and a scheduled reconciliation check that alerts on drift. Get those three right and the rest is plumbing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need this built or fixed?</h2>



<p class="wp-block-paragraph">I design and run serverless data pipelines on AWS, and Jira extraction has more sharp edges than most sources. Things I can help with directly:</p>



<ul class="wp-block-list">
<li>Migrating an extractor off the removed <code>/rest/api/3/search</code> endpoint to cursor-based <code>/search/jql</code> without losing rows in the cutover</li>

<li>Auditing an existing Jira pipeline for duplicate rows and reconciling your numbers back against JQL</li>

<li>Building changelog-based cycle time and time-in-status models that survive workflow renames</li>

<li>Restructuring an S3 layer with Parquet, partition projection and file compaction to cut Athena scan costs</li>

<li>Packaging the whole thing as Terraform or CloudFormation with CI deployment and least-privilege IAM</li>

<li>Connecting Athena to Grafana, Power BI, Metabase or QuickSight with curated views instead of raw tables</li>
</ul>



<p class="wp-block-paragraph">Send me your extractor code, a Glue table definition, or an Athena query that is scanning more than it should, and I will tell you what I would change.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/jira-analytics-pipeline-aws-lambda-athena/">Building a Jira Analytics Pipeline with AWS Lambda and Athena (Without Double-Counting Everything)</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
