That is the moment. Not a crash, not an error banner. A plain question about the system that nobody can answer from the canvas.

GoHighLevel is good at what it was built for and most accounts never need anything else. But there is a point where the next workflow makes the account worse instead of better, and past it, GoHighLevel custom development costs less than another six months of patching. Below: the failure mode hiding under all the others, six signals you have crossed that line, the honest case for staying in the builder, and three levels of code to try before committing to a full app.

The failure hiding under all the others

The workflow builder has no failure semantics. Everything below is a consequence of that.

When a step works, the contact moves on. When a step fails, the contact usually also moves on. No transaction, no rollback, no retry budget you control, no way to say “this contact is in a bad state, stop and tell someone”. A Custom Webhook that gets a 500 back does not park the contact for review. It becomes a line in an execution log nobody reads, and the contact continues as though the other system had answered.

Fine when the step sends a text message. Not fine when the step is the only thing keeping two systems in agreement. So the useful test is not “is this hard to build in the builder”, because plenty of hard things build fine. The test is: what happens when this step fails, and would I find out?

Signal one: you are storing state in tags

Tags start as labels. Then someone needs to know whether a lead was called, so there is a called tag. Then whether they were called twice, so called-2. Then a workflow that removes called when a new form arrives, and another that puts it back if an appointment was booked first.

Those are no longer labels. That is a state machine with no diagram, no validation, and no guarantee that only one workflow writes to it at a time. Two workflows fired by the same event will both read the old value and both write, and whichever finishes last wins. That is a race condition, and it hides during testing because you test one contact at a time. The tell is a tag name with a number in it, or a tag whose meaning needs the word “unless” to explain.

Signal two: the data has to be right, not eventually right

Marketing automation tolerates drift. One extra nurture email hurts nobody. The moment GoHighLevel holds something that must reconcile with another system, that tolerance is gone. Order status from a store, invoice state from an accounting package, availability shared with a clinic system.

Those need idempotency, meaning a repeated message produces the same result as a single one, and reconciliation, meaning something periodically compares both sides and reports the difference. The builder gives you neither: no natural home for an idempotency key, no scheduled job that reads both systems and complains. You can fake reconciliation with a workflow on a timer, but that is a batch job written in a UI designed for follow-up sequences, and the first time it half-runs you will not hear about it.

Signal three: per-execution billing became a per-lead tax

The actions that make workflows powerful are premium actions, billed per execution against the agency wallet. Custom Webhook, Inbound Webhook, Custom Code and the workflow AI actions all sit in that category. HighLevel gives each sub-account a small free allowance when premium actions are first switched on, publishes volume tiers above that, and lets you rebill sub-accounts individually instead of absorbing the charge. Take the current rate and tiers from HighLevel’s billing documentation rather than any blog post, including this one.

The mechanism is the point. You pay per execution, not per useful outcome. A workflow calling out three times to enrich, score and push a lead is three executions per lead. Multiply by volume and sub-accounts, and the plumbing line can pass the cost of a small service doing all three in one request.

Two caveats. Consolidating three calls into one endpoint is usually cheaper than moving the whole thing to code. And whatever you move still costs something to run, whether that is a VPS from a provider like Contabo or InterServer, a hosted platform such as Make or n8n, or a serverless function. Compare both sides.

Signal four: the same logic lives in twenty sub-accounts

Snapshots are the platform’s answer and better than people expect. You can refresh a snapshot and push selected assets to sub-accounts that previously imported it, choosing both the assets and the recipients, and the push is selective rather than destructive.

Two limits. Push Update only reaches sub-accounts inside your own agency, so anything shared externally has to be re-imported through a new link. And HighLevel documents that when a snapshot refresh deletes steps, contacts waiting on those steps in workflows created from that snapshot are removed. Fine for a copy change, destructive for a structural one.

Snapshots distribute structure. They do not give you versioning, a rollback path, a diff between intended and running state, or a test that fails before the change reaches a client. Once a change has to land correctly in twenty accounts on one day, you want those, and they come from putting the logic behind an API you control and calling it from a thin step in each account.

Signal five: the integration must survive the other side being down

A Custom Webhook fires the moment the contact reaches the step. If your endpoint is slow, restarting or rate limited, that single firing is what you get. No queue in front, no backoff behind.

The API side has numbers worth knowing. HighLevel documents a burst limit of 100 requests per 10 seconds and a daily limit of 200,000 requests, counted per marketplace app per resource, where a resource is a location or the company. Every response carries headers telling you where you stand, so backoff can follow what the server says instead of a guessed sleep.

curl -sS -D - -o /dev/null 
  -H "Authorization: Bearer $GHL_TOKEN" 
  -H "Version: $GHL_API_VERSION" 
  "$GHL_API_BASE/contacts/" 
  | grep -i '^x-ratelimit'

-D - writes response headers to stdout while -o /dev/null discards the body, so you read the rate limit state without dumping contact data into your shell history. Set the base URL and the required Version value from the current API reference. Watch X-RateLimit-Max and X-RateLimit-Remaining for the burst window, X-RateLimit-Interval-Milliseconds for its length, and X-RateLimit-Limit-Daily with X-RateLimit-Daily-Remaining for the day.

The daily budget sounds enormous until a migration. Reading a contact, upserting it, then writing custom fields is three calls each, so forty thousand contacts is 120,000 calls before live traffic touches the account. Batch the reads, skip writes where nothing changed, run backfills overnight. Watch for loops too: writes made through the API can themselves fire workflow triggers, so a receiver that writes back has built a cycle.

Signal six: someone needs a screen, not a form

Forms collect. They do not display. Once a client asks to see a filtered view of their own data, approve something in two clicks, or work a queue that is not a pipeline, you are looking at an interface.

The platform supports this. Marketplace apps can add Custom Pages, which render inside HighLevel in an embedded iframe loading a URL you host, appearing on the app details page or in the left navigation. HighLevel publishes an app template to start from, plus an SSO mechanism so your page can identify the user instead of asking them to log in again. That is still a real application with hosting, auth and deployment: the biggest step on this list, the one I would defer longest, and the only honest answer when the requirement is a screen.

Where the workflow builder genuinely wins

A one-sided argument is not worth reading, so here is the case against writing code.

• The person who owns the outcome can change it. An account manager adjusts send timing on a Friday with no deploy and no developer awake.
• Messaging and scheduling are solved. SMS, email, calendars, reminders and the delivery infrastructure behind them are the platform’s core. Rebuilding any of it is a bad trade.
• Timezone handling is built in. Time-based steps run in the account timezone or the contact’s own, falling back to the account when a contact has none.
• No pipeline to maintain. No repository, no secrets rotation, no dependency upgrades, no server needing patched at an awkward hour.
• Execution history is right there. For a linear sequence, reading what happened to a contact beats correlating logs across two systems.

If the automation is a follow-up sequence with branching, the builder is correct and code is the expensive mistake.

Three levels before you build an app

“Needs a developer” is not one decision. It is three, and most accounts only need the first.

Level one: the Custom Code action

The builder has a Custom Code premium action that runs JavaScript inside the workflow. Values from earlier steps are mapped in as named input properties and arrive in an InputData object. The action returns a JavaScript object or an array of objects for later steps to reference, and console.log output is captured so you can see what ran.

// Normalise a phone number before it reaches a branch step.
// rawPhone and country are input properties mapped from earlier steps.
const digits = String(InputData.rawPhone || "").replace(/D/g, "");
const cc = InputData.country === "US" ? "1" : "";

if (!digits) {
  console.log("no phone on contact, routing to manual branch");
  return { e164: "", usable: false };
}

return {
  e164: "+" + (digits.startsWith(cc) ? digits : cc + digits),
  usable: digits.length >= 10
};

This kills a lot of branch sprawl. Ten If/Else steps existing only to reshape a value collapse into one action returning a clean value and a boolean. Still inside the platform, still billed per execution, still no retries. But logic smeared across a canvas now reads in one place.

Level two: a private integration and a script you own

For one account you do not need a marketplace app. Create a Private Integration inside the sub-account, select only the scopes the job needs, and use the token as a bearer credential. No OAuth handshake, no refresh cycle, no listing.

This is where reconciliation jobs, nightly audits, migrations and bulk cleanups belong. Run it as a cron job on a small VPS, behind a Cloudflare Tunnel if it needs an inbound endpoint, with the token in a secrets store rather than a config file. Treat that token as a credential with a rotation plan, not a string you paste once and forget.

Level three: a marketplace app

You need this when the thing spans many accounts, has a user interface, or is a product you intend to sell. It means OAuth 2.0 on the authorization code grant, token storage and refresh per installed location, webhook handling, and somewhere to run it. Note the version boundary: the v1 API is end of life and unmaintained, and new builds target the OAuth-based v2. If a tutorial hands you a permanent location API key, it is describing the old world.

Marketplace apps can also register their own workflow triggers and actions, which appear in the builder for sub-accounts that installed the app. That is the pattern I reach for first: the hard logic lives in your service, and the person who owns the campaign still drags a step onto a canvas. It requires the workflows read scope, and those custom actions are themselves premium actions billed per execution.

Arguments that do not survive contact

• “We will just add one more workflow.” The cost is not building it. It is that every future workflow now has to be reasoned about alongside it. Ten workflows listening for one tag is not ten problems, it is the interactions between them.
• “No-code means no developer.” It means no deployment. Someone still holds the whole model in their head, and when they leave, an undocumented canvas is harder to inherit than a repository with a README.
• “Zapier or Make will handle it.” Often true and worth trying first. What they do not give you is one place where business logic lives. Split rules across three tools and debugging means three log formats.
• “It works, so it is fine.” Every silent failure worked right up until someone counted. Absence of errors in a system with no error reporting is not evidence.
• “Custom code is lock-in.” Backwards. Logic on a canvas is bound to one vendor’s UI and exports only as a snapshot. Logic in a repository moves.

How I would decide

1. Write down what happens when each external call fails. If the answer anywhere is “the contact continues as if it succeeded” and that matters, you have found your requirement.
2. Count the workflows writing to the same tag or field. More than two means you have a state machine. Draw it before touching anything.
3. Count premium executions per lead, then multiply by monthly volume. Compare against a few hours a month maintaining a small service. Sometimes the builder still wins. Do the arithmetic instead of assuming.
4. Ask how a change reaches every account. If it involves opening sub-accounts one at a time, that is the real cost, and it grows with every client.
5. Try the smallest level that fits, and leave the trigger in the builder. Custom Code action, then private integration script, then marketplace app. Whatever moves out, keep the entry point as a workflow step so the marketing side can still change timing and audience without you.

Whatever you build, monitor it. An uptime check on your receiver and an alert when the reconciliation job reports a non-zero difference will catch more real problems than any amount of careful workflow naming.

Frequently asked questions

Do I need a marketplace app to use the GoHighLevel API?

No. For a single sub-account, create a Private Integration inside that account, grant only the scopes you need, and use the token as a bearer credential. Marketplace apps and OAuth are for anything spanning accounts or distributed to other people.

How much automation should stay in workflows?

The parts a non-developer should be able to change: timing, audience, message copy, branching on obvious contact attributes. Move out the parts where being wrong is expensive and being wrong is silent.

Why does my workflow show as successful when the integration clearly failed?

A step firing and a step achieving something are different events, and the builder records the first. A webhook returning an error still counts as having run. If the outcome matters, verify it explicitly rather than trusting the log.

Should I avoid premium workflow actions on cost grounds?

Not on principle. They cost per execution because they are the expensive verbs. The problem is fragmentation: three separate calls where one would do triples the bill for the same outcome. Consolidate before considering a move off the platform.

What does a GoHighLevel custom development project usually start with?

An audit, not a build. Map which workflows write to which fields, where external calls happen, and what the failure path is for each. Often that document alone removes the need for the project.

The one thing worth remembering

Ask what happens when a step fails. That question separates automation belonging in the builder from automation needing code, more reliably than counting steps, counting workflows, or judging how clever the branching has become.

If failure is visible and cheap, stay in the builder and enjoy that your client can edit it themselves. If failure is silent and expensive, GoHighLevel custom development stops being an upgrade and becomes maintenance you already deferred. Then start small: a Custom Code action beats a marketplace app you will not want to maintain.

Working on this with me

I work on the engineering side of GoHighLevel accounts, usually the parts failing quietly rather than the parts that look broken.

• Auditing an account for silent failures: which workflows write to which tags and fields, and what happens on every external call that returns an error.
• Replacing branch sprawl with Custom Code actions, so logic reads as one block instead of fifteen steps on a canvas.
• Webhook receivers and sync services that retry, deduplicate and reconcile, with the rate limit headers driving backoff instead of a fixed sleep.
• Migrations and backfills planned against the daily API budget, batched and run in the account’s timezone so live traffic keeps working.
• Private integration setup with least-privilege scopes, token storage and a rotation plan that is written down.
• Marketplace app work when genuinely warranted: OAuth handling, custom workflow actions, and Custom Pages hosted somewhere you control.

If you want a second opinion, send something concrete: a screenshot of the workflow you are least sure about, an execution log entry that makes no sense, or the response body your webhook endpoint is returning. That is a faster conversation than a scoping call.

The post Six Signs GoHighLevel Needs a Developer, Not Another Workflow appeared first on John Nessime.

That’s the shape of the problem. When you backfill historical API data into S3, the failure to worry about isn’t the one that throws. It’s the one that writes fewer records than it should and reports success anyway. Bytes transferred look plausible. Nothing goes red. The data is quietly wrong, and you find out when someone downstream trusts it.

This post covers the failure families that produce that outcome and the design decisions that make them impossible rather than merely unlikely: pagination that drifts under a mutating source, retries that write the same page twice under different keys, a prefix layout chosen while thinking about writes instead of reads, the seam where the backfill meets the live pipeline, and the storage class trap that makes a cheap-looking archive expensive to fix.

Why a backfill that finishes cleanly can still be wrong

A live pipeline has a natural error signal. Data stops arriving, a lag metric climbs, someone notices. A backfill has none of that. It runs once, against a range nobody is watching in real time, and then it’s over. The only evidence it leaves behind is a set of objects in a bucket.

So the question you need to answer months later isn’t “did the job succeed?” It’s “is this window complete, and how do I know?” If your design can’t answer that from artifacts on disk, you don’t have a backfill. You have a large amount of data of unknown provenance.

Failure family 1: the source moved while you paginated

This one bites hardest and shows up least. Most REST APIs paginate historical queries with an offset or page number. You ask for page 1, page 2, page 3. Meanwhile the underlying table is still accepting writes.

If a record is inserted into the middle of the result set between your page 4 and page 5 requests, everything after it shifts down by one. The row that would have started page 5 is now the last row of page 4, which you already fetched. You request page 5 and that record is gone. Not errored. Gone. Insertions cause skips, deletions cause duplicates, and the job still exits zero.

What to do instead

• Prefer keyset or cursor pagination. If the API offers an opaque cursor or a “records after ID X” parameter, use it. The read position is anchored to a value rather than a count, so inserts elsewhere in the set can’t shift it.
• Sort by an immutable column. If you’re stuck with offsets, sort by something that never changes after insert, usually the primary key or the creation timestamp. Sorting by a last-modified column while records are being modified is the worst case: rows physically move within the ordering as you walk it.
• Bound every request on both ends. Don’t ask for “everything before now.” Ask for a closed interval with an upper bound already in the past. A closed window can’t grow while you read it.
• Know which timestamp the filter applies to. Filtering on created-at gives you records that came into existence in that window. Filtering on updated-at gives you records touched in it, which for a mutable source is a different set entirely.

That last point causes arguments. Backfill on created-at, then run incremental syncs on updated-at, and your historical range never picks up edits made to old records. That may be correct for your use case. It is not correct by default, and it’s worth writing down which one you chose.

Failure family 2: the retry that wrote the same page twice

Backfills retry. They have to, because you’re hammering an API for hours and something will time out. The problem is what the retry writes.

If your object key contains a UUID, a run identifier or a wall-clock timestamp, a retried page produces a second object with a different key and the same records inside. You now have duplicates that listing will never reveal, because both objects are legitimately present and neither is obviously a copy. Derive the key from the request instead of the run, so the same window plus the same cursor always produces the same key and a retry overwrites rather than accumulates.

import hashlib
import boto3
from botocore.config import Config
from botocore.exceptions import ClientError

s3 = boto3.client(
    "s3",
    config=Config(
        retries={"total_max_attempts": 8, "mode": "standard"},
        max_pool_connections=32,
    ),
)

def page_key(dataset, window, cursor):
    # Deterministic: the same page always maps to the same key.
    token = hashlib.sha256(cursor.encode()).hexdigest()[:16]
    return f"raw/{dataset}/dt={window}/page-{token}.json.gz"

def put_once(bucket, key, body):
    try:
        s3.put_object(Bucket=bucket, Key=key, Body=body, IfNoneMatch="*")
        return "written"
    except ClientError as err:
        if err.response["Error"]["Code"] == "PreconditionFailed":
            return "already_present"
        raise

Two things in there are worth explaining rather than copying.

IfNoneMatch="*" is a conditional write. S3 accepts the PUT only if no object exists at that key and returns a 412 PreconditionFailed otherwise. It’s supported on PutObject and CompleteMultipartUpload. That turns “did I already write this page?” into a single request with no read-before-write race. If two workers hit the same key concurrently, S3 can instead return a 409 ConditionalRequestConflict, which AWS documents as retryable, so handle the two codes differently: retry on 409, skip on 412.

The same thing from the CLI, useful for testing the behaviour by hand:

aws s3api put-object 
  --bucket your-bucket 
  --key raw/orders/dt=WINDOW/page-abc123.json.gz 
  --body page.json.gz 
  --if-none-match "*"

The retries block matters for a reason that trips people up. In botocore, max_attempts means different things depending on where you set it: in the AWS config file or the environment variable it counts total requests including the first, but in a Config object it counts retries on top of the first. total_max_attempts exists only in a Config object and always means total, so using it removes the ambiguity. There’s also an adaptive mode that layers client-side rate limiting on top of standard; AWS still labels it experimental, so I reach for standard plus my own concurrency cap first.

Failure family 3: a layout designed for writing, not reading

The prefix structure you pick during the backfill is the one you’re stuck with. Reorganising later means rewriting every object, and copy requests aren’t free.

Partition on event time, not ingestion time

The tempting layout is the date you ran the job. It’s easy, it’s monotonic, and it’s useless. Every query anyone writes against historical data filters on when the thing happened, not when you happened to fetch it. If your partitions are ingestion dates, every one of those queries becomes a full scan. Use Hive-style keys so query engines prune without extra configuration:

s3://your-bucket/raw/orders/dt=YYYY-MM-DD/page-<token>.json.gz
s3://your-bucket/curated/orders/dt=YYYY-MM-DD/part-<n>.parquet

Keep the raw landing zone and the curated zone separate. Raw is whatever the API gave you, unmodified, so you can re-derive everything if your parsing logic turns out to be wrong. Curated is columnar and typed. Merging them saves nothing and costs you the ability to reprocess.

Don’t let the write path throttle itself

S3 documents at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix, with no limit on the number of prefixes. The word doing the work is partitioned. That rate applies once S3 has split the keyspace out for that prefix, and AWS is explicit that the scaling is gradual rather than instant. During the ramp you can see 503 SlowDown responses even when your sustained rate looks well inside the documented ceiling.

For a backfill this is mostly good news, since date-partitioned keys spread writes across many prefixes as you work through the range. It becomes a problem when you parallelise within a single day and drive thousands of writes per second into one dt= prefix. Two mitigations: back off with jitter instead of retrying 503s immediately, and shard workers across windows rather than within them.

Register partitions without a scan

A backfill can create thousands of partitions at once. MSCK REPAIR TABLE handles that badly. AWS documents that it only adds partitions and never removes them, that it can take a long time on large tables, and that it can time out leaving the catalog partially updated. Partial success with no error is exactly the failure mode this whole post is about.

Partition projection sidesteps the catalog. Athena computes the partition list from table properties instead of looking it up, so a backfilled range is queryable the moment the objects exist.

TBLPROPERTIES (
  'projection.enabled'          = 'true',
  'projection.dt.type'          = 'date',
  'projection.dt.format'        = 'yyyy-MM-dd',
  'projection.dt.range'         = 'START_DATE,NOW',
  'projection.dt.interval'      = '1',
  'projection.dt.interval.unit' = 'DAYS',
  'storage.location.template'   = 's3://your-bucket/curated/orders/dt=${dt}/'
)

Replace START_DATE with the earliest date in your backfill. The trade-off is worth stating plainly: projection generates every partition in the range whether or not data exists there, so an over-wide range on a sparse dataset wastes planning effort. And if the S3 layout ever stops matching storage.location.template, queries return nothing rather than erroring. Set the range to your actual data, not to a comfortable margin.

Failure family 4: the seam between backfill and live pipeline

Two processes write to the same table: a backfill walking backwards through history, and a live ingest walking forwards. They meet somewhere, and that meeting point is where records go missing.

The instinct is to make the handoff exact. Backfill up to timestamp T, start streaming from T. Don’t. Clock skew between your runner and the API, records that arrive late at the source, and rounding in the API’s own filtering will all conspire to open a hole at exactly T.

Overlap deliberately instead. Run the backfill past the point where live ingest started, by a margin generously larger than any plausible source lag, then deduplicate on a natural key during curation. Duplicates you can remove with a query. Gaps you can only fix by going back to the API, and by then the retention window may have closed.

Failure family 5: the storage class you can’t undo

Historical data feels like archive data, so the instinct is to write it straight into a cold class or transition it immediately. Read the mechanics first, because the billing rules for cold classes are structural and pro-rated against you.

• S3 Standard-IA and One Zone-IA carry a 30-day minimum storage duration and a 128 KB minimum billable object size.
• Glacier Instant Retrieval and Glacier Flexible Retrieval carry a 90-day minimum duration; Deep Archive carries 180 days. Delete, overwrite or transition earlier and you’re charged the remainder.
• Objects in Glacier Flexible Retrieval and Deep Archive are billed for 40 KB of extra per-object metadata on top of the object itself.

Read those together and the conclusion is uncomfortable. A backfill that writes millions of small JSON pages into an archive class pays a size floor on every one of them, plus metadata overhead, plus a minimum duration that makes fixing the mistake expensive. Those pages are also the objects you’re most likely to want to reprocess, which is exactly the overwrite that triggers the early-deletion charge.

What I’d actually do: land raw pages in Standard, compact them into larger Parquet files in the curated zone, verify completeness, and only then apply a lifecycle rule to the raw prefix. Compaction is what makes cold storage sane, because it turns a size floor per tiny object into a rounding error on a large one. It also fixes the small-file problem in Athena, which is a separate cost line.

One more cost note: if your backfill runs from EC2 in a private subnet, every byte pulled from the API and pushed to S3 may route through a NAT gateway with per-GB processing charges. A gateway VPC endpoint for S3 removes the S3 half of that. It’s a five-minute change people usually discover after the bill.

Proving the backfill is complete

Here’s the part most backfills skip, and the part that makes everything above verifiable rather than hopeful. Write a manifest object per window, alongside the data, describing what that window contains.

{
  "dataset": "orders",
  "window": "<dt value>",
  "filter_field": "created_at",
  "range_start_inclusive": "<iso8601>",
  "range_end_exclusive": "<iso8601>",
  "pages_written": "<int>",
  "records_written": "<int>",
  "source_reported_total": "<int or null>",
  "last_cursor": "<opaque token>",
  "schema_fingerprint": "<sha256 of sorted field names>"
}

The manifest turns completeness into something you can query. A window with no manifest was never finished. A window whose records_written disagrees with source_reported_total is suspect. A window whose schema_fingerprint differs from its neighbours tells you the API changed shape partway through your range, which is otherwise close to undetectable.

Write the manifest last, after every page for that window is confirmed in S3. A manifest written optimistically at the start is worse than no manifest at all, because it asserts a completeness nobody checked.

Troubleshooting a backfill that already ran

1. Count objects per window and look at the shape. Use S3 Inventory rather than a live ListObjectsV2 walk on a large bucket. Plot objects per dt= partition. Real data has a rhythm, weekday peaks and weekend troughs. A window that breaks the rhythm without a business reason is your gap.
2. Reconcile record counts against the source. Many APIs return a total count on the first page of a filtered query. Pull that for a sample of windows and compare with what’s in S3. If you kept manifests, this is one query.
3. Check for duplicate natural keys within a partition. A GROUP BY on the record ID with HAVING COUNT(*) > 1 finds retry duplicates immediately. Duplicates clustered at window boundaries mean pagination drift; duplicates spread evenly mean your keys weren’t deterministic.
4. Look at boundary hours, not whole days. Cursor drift and clock skew concentrate at the edges of each request window. Aggregate to the hour and the missing slice usually becomes obvious even when daily totals look fine.
5. Verify the layout still matches the table definition. If you’re using partition projection and a query returns zero rows, the first suspect is storage.location.template not matching the actual prefix, trailing slashes included.

Common mistakes

• Treating an exit code of zero as evidence of completeness.
• Putting a run ID or timestamp in the object key, which makes retries additive instead of idempotent.
• Partitioning by ingestion date because it’s what the job naturally knows.
• Making the backfill and live-ingest boundary exact instead of overlapping.
• Writing millions of small objects straight into an archive class and meeting the minimum object size and duration rules afterwards.
• Running MSCK REPAIR TABLE over thousands of new partitions and trusting that it completed.
• Backfilling on a last-modified field while assuming you captured creation events.
• Sharding workers within a single day, concentrating writes on one prefix and inviting 503s.

Best practices when you backfill historical API data into S3

• Make every window independently re-runnable. If re-running one day is routine rather than scary, you’ll actually fix problems when you find them.
• Derive object keys deterministically from window plus cursor, and use conditional writes so a retry is a no-op.
• Bound every API request on both ends, and never let the upper bound be “now”.
• Write raw responses untouched before parsing anything. Parsing bugs are common; re-fetching a closed API window often isn’t possible.
• Emit a manifest per window, written last, and reconcile it against source counts.
• Compact to Parquet before applying any lifecycle transition.
• Run the job somewhere it can survive for hours. A long backfill on a laptop over hotel wifi is a bad plan; a small dedicated VPS from a provider like Contabo or InterServer, or a spot-priced EC2 instance, costs little and won’t sleep halfway through.
• Watch progress somewhere other than the terminal. Pushing records-per-window into CloudWatch or Grafana gives you a curve you can eyeball, and an unusually flat stretch is a gap you can catch while the source data is still retrievable.

FAQ

Should I write raw JSON or convert to Parquet during the backfill?

Both, in two zones. Write the raw response body first so the fetch step has no dependency on your schema assumptions, then convert as a separate job. Converting inline couples an expensive, rate-limited API walk to a parsing step that will eventually throw on an unexpected field, and you don’t want to re-fetch years of data because of a type error.

How large should each object be?

In the raw zone, one object per API page is fine and keeps idempotency simple. In the curated zone, aim for files in the tens to low hundreds of megabytes. Query engines pay a fixed overhead per file, so thousands of tiny Parquet files scan slower and cost more than a few large ones holding identical data.

Can I run the backfill in parallel across date windows?

Yes, and that’s the right axis, because each window is independent and lands on its own prefix. The binding constraint is almost always the source API’s rate limit rather than S3. Add a global concurrency cap and honour any documented limit or Retry-After header. Getting your credentials throttled or suspended mid-backfill is a worse outcome than finishing slowly.

Do I need AWS Glue or Airflow for this?

Not to start. A single script that takes a window as an argument, writes pages and a manifest, and is safe to re-run covers most backfills. Orchestration earns its place when you need retries across hundreds of windows, dependency ordering, or a visible run history. Glue and Step Functions fit AWS-native stacks; Airflow gives you more control and more to operate. Pick after the script works, not before.

What if the API caps how far back I can query?

Then the retention boundary is your real deadline and it should drive priority. Fetch the oldest reachable data first, not the most recent, because the newest data is still available tomorrow and the oldest may not be. If the vendor offers a bulk export or reporting endpoint separate from the standard API, it’s usually faster and less rate-limited than paginating.

Does any of this change with S3-compatible storage instead of AWS?

The layout, idempotency and manifest patterns carry over unchanged to Cloudflare R2, Backblaze B2, MinIO and similar. What varies is API-level feature support. Conditional writes with If-None-Match, request-rate behaviour and storage class semantics differ by provider, so verify the specific behaviour you’re depending on against that provider’s documentation rather than assuming parity with S3.

How do I know the backfill is actually finished?

When every expected window has a manifest, every manifest’s record count reconciles against the source, and a duplicate check on natural keys returns nothing. Not when the script exits.

The one thing worth remembering

A backfill isn’t finished when the job exits. It’s finished when you can prove the range is complete without re-reading the source. Deterministic keys, conditional writes, closed windows, manifests and deliberate overlap all exist to make that proof cheap.

If you’re about to backfill historical API data into S3, spend the extra hour on the key derivation and the manifest before you start the run. The alternative is learning about a gap from a chart, months later, with the source retention window already closed behind you.

Need a hand with a backfill?

This is a large part of what I do. If you’re staring at a historical range and an API with a rate limit, I can help with:

• Designing an idempotent, re-runnable backfill job with deterministic keys and per-window manifests, so a failed day is a one-command fix.
• Choosing a partition layout that serves your actual query patterns, and setting up Athena or Glue tables with partition projection so thousands of backfilled partitions become queryable without catalog scans.
• Auditing a backfill that already ran, reconciling record counts against the source and finding the gaps and duplicates before your dashboards do.
• Handling awkward source APIs: offset-only pagination, aggressive rate limits, undocumented result caps, schema drift partway through a historical range.
• Compaction and lifecycle design, so cold storage actually saves money instead of hitting minimum object size and minimum duration charges.
• Wiring the seam between the backfill and your live pipeline so there’s no gap at the handover, plus the monitoring that would have caught it.

Send me something concrete: a sample API response, your current key layout, or the object counts per partition. That’s usually enough to spot where the gap is.

The post Backfilling Historical API Data into S3 Without Silent Gaps appeared first on John Nessime.

Here is the awkward part: nothing was broken. No alert fired, no error rate moved, no dashboard turned red. Third-party API spend is one of the few production signals with no failure mode attached to it. A background job that polls a vendor every thirty seconds and gets back the exact same payload it got last time looks, from the outside, identical to a healthy integration. It just costs money every single time.

This post is about how to reduce SaaS API costs by changing the shape of your calls rather than by renegotiating your contract. Six levers, in the order I’d pull them: read the billing unit, kill pointless polling, collapse N+1 patterns, cache honestly, stop your own retry amplification, and attribute spend so you can tell whether any of it worked.

Start with the billing unit, not the call count

The single most common mistake I see is optimising HTTP requests when the vendor doesn’t bill HTTP requests. Before you touch any code, read the pricing page carefully enough to answer one question: what is the countable thing?

It varies more than people expect:

• Per request. The obvious one. Geocoding, enrichment, most REST endpoints.
• Per session. Google Maps Platform bills Places Autocomplete this way when you pass a session token: the whole keystroke stream plus the terminating Place Details call is grouped and billed as one unit. Omit the token, or reuse one across sessions, and every keystroke request is billed individually.
• Per token issued. Auth0’s machine-to-machine quotas count access tokens obtained through the client credentials flow, not the API calls you make with them. A service that fetches a fresh token on every outbound request is burning the meter for no reason.
• Per input and output token. LLM providers. Output is typically priced higher than input, which makes “be concise” a cost control and not just a style note.
• Per record, event, or message. Twilio, SendGrid, most streaming and messaging platforms. One API call can carry many billable units.
• Not at all. Stripe doesn’t bill you per API call; the constraint there is a rate limit, not a line item. Optimising those calls buys you reliability headroom, not money.

Getting this wrong sends you down the wrong path for weeks. Batching requests against an endpoint that bills per record changes your request count and leaves the invoice exactly where it was. Read the unit first.

Lever one: stop polling for things that rarely change

This is almost always the biggest single line, and it’s invisible because it’s steady. A sync job on a five-minute cron is 288 calls a day per resource. Multiply by resources, by environments, by the replicas you quietly scaled to three, and you get a number with no relationship at all to how often the data changed.

Conditional requests, when the vendor honours them

If the API returns an ETag or Last-Modified header, store it and send it back on the next poll as If-None-Match or If-Modified-Since. If nothing changed you get a 304 Not Modified with an empty body.

# First call: capture the ETag from the response headers
curl -sS -D - -o /dev/null 
  -H "Authorization: Bearer $TOKEN" 
  https://api.github.com/repos/OWNER/REPO/pulls

# Second call: send it back and check the status code only
curl -sS -o /dev/null -w '%{http_code}n' 
  -H "Authorization: Bearer $TOKEN" 
  -H 'If-None-Match: "PASTE_THE_ETAG_HERE"' 
  https://api.github.com/repos/OWNER/REPO/pulls

-D - dumps response headers to stdout so you can see the etag value. -w '%{http_code}' on the second call prints just the status, which is all you care about: 304 means the resource is unchanged and you saved a payload.

Whether that saves money depends on the vendor. GitHub documents that a conditional request returning 304 doesn’t count against your primary rate limit when it’s correctly authorised. Plenty of other vendors return 304 happily and bill it anyway. Test against your own invoice before you assume.

Three things that bite people with ETags:

• They’re per-page on paginated endpoints. A 304 on page one says nothing about pages two through five, so store one ETag per page.
• They can be scoped to the credential. Rotate a token and your stored ETags may stop matching, which quietly turns your conditional requests back into full ones.
• GraphQL endpoints generally don’t do conditional requests at all. You cache those yourself, keyed on a hash of the query and its variables.

Webhooks, when they exist

The proper fix is to stop asking. If the vendor supports webhooks, the call count collapses from “one per interval” to “one per actual change”, which is usually a difference of two orders of magnitude on a slow-moving resource.

Be honest about what it costs, though. Webhooks need a publicly reachable endpoint, signature verification, idempotent handling because vendors redeliver, a dead-letter path, and a reconciliation sweep for deliveries that never arrived. On a resource that changes every few seconds, polling is simpler and the savings are small. On one that changes twice a week, the webhook pays for itself immediately.

A middle option people forget: many APIs expose a cheap “has anything changed” endpoint, a sync cursor, or a delta token. Poll that on a tight interval and only fetch the expensive payload when it moves.

Lever two: collapse the N+1 pattern

You know this one from databases, and it shows up identically against third-party APIs. Fetch a list of 200 items, then fetch each item’s detail individually: that’s 201 billable calls where the vendor probably offers a way to do it in one or two. Look for:

• Expansion or embed parameters. Many REST APIs let you inline related objects in the list response so the detail calls disappear entirely.
• Field selection. Asking for fewer fields is free on per-request billing but matters a lot where the field tier drives the price. Google Places is explicit about this: requesting a single higher-tier field upgrades the whole response to that tier’s pricing.
• Bulk and batch endpoints. Salesforce, Algolia, most CRMs and search platforms have one. They’re usually documented in a corner nobody reads.
• Larger pages. If the endpoint accepts a page size of 200 and you’re defaulting to 25, you’re paying eight times as many calls to fetch identical data.

The page-size one is the cheapest win in this entire post and it’s a one-line change. Check your client library defaults today.

Lever three: cache with a boring, honest TTL

Caching third-party responses is the lever everyone reaches for first and gets subtly wrong. Three failure modes matter more than the cache itself.

Stampede. A popular key expires, fifty concurrent requests all miss, and fifty identical calls go out to the vendor. You just paid fifty times for one refresh, at the worst possible moment.

No negative caching. A lookup that returns “not found” is usually not cached at all, so the same failing lookup hammers the vendor forever. Cache your 404s. Short TTL, but cache them.

Cache key design. If your key doesn’t include the credential or tenant, you will eventually serve one customer’s data to another. This is the one mistake in this post that turns a cost problem into an incident.

You don’t necessarily need application code for this. An Nginx instance in front of the vendor handles all three, and it’s the approach I reach for first when several services hit the same API:

proxy_cache_path /var/cache/nginx/api levels=1:2 keys_zone=api_cache:20m
                 max_size=1g inactive=60m use_temp_path=off;

server {
    listen 127.0.0.1:8080;

    location / {
        proxy_pass https://api.vendor.example;
        proxy_set_header Host api.vendor.example;

        proxy_cache api_cache;
        proxy_cache_key "$request_method$request_uri$http_authorization";

        proxy_cache_valid 200 5m;
        proxy_cache_valid 404 30s;

        proxy_cache_lock on;
        proxy_cache_use_stale updating error timeout;
        proxy_cache_background_update on;

        add_header X-Cache-Status $upstream_cache_status;
    }
}

Reading that from the top down, because every directive is there for a reason:

• proxy_cache_key includes $http_authorization, so responses are isolated per credential. That’s the tenant-leak fix.
• proxy_cache_valid 404 30s is the negative caching. Failed lookups stop being free traffic for the vendor.
• proxy_cache_lock on is the stampede fix. Only the first request for a given key goes upstream; the rest wait for it.
• proxy_cache_use_stale updating plus proxy_cache_background_update on serves the stale copy while a single background request refreshes it. Latency stays flat and you don’t pay a thundering herd at every expiry.
• X-Cache-Status gives you HIT, MISS, STALE or UPDATING in the response so you can actually measure your hit rate instead of guessing at it.

The trade-off is worth stating plainly: you’re now serving data that can be five minutes stale, and you own another moving part. Fine for a currency rate or a product catalogue. Not fine for an inventory count on a checkout page. Set the TTL from how wrong the data is allowed to be, never from how much you want to save.

Where the caller is distributed, Redis with a short lock key does the same job in application code, and a small always-on VPS from something like InterServer or DigitalOcean is plenty to host either the proxy or the cache. At the edge, Cloudflare Workers with KV lets you cache vendor responses close to users, which helps when the calls originate from browsers rather than your backend.

Lever four: stop paying for your own retries

This produces the sudden spike rather than the slow creep, and people miss it because retries feel like resilience.

The vendor wobbles. Every client retries. With a fixed delay they retry in lockstep, so the vendor gets a synchronised wave on top of normal load, struggles longer, and generates more retries. AWS’s architecture write-up on exponential backoff and jitter is the canonical reference, and its central finding is worth repeating: backoff alone still leaves visible clusters of calls. Randomising each client’s delay is what decorrelates the herd, and in their simulation it cut total call count substantially.

import random, time

BASE = 0.5      # seconds
CAP  = 30.0     # never sleep longer than this

def sleep_before_retry(attempt):
    # capped exponential backoff, then full jitter
    window = min(CAP, BASE * (2 ** attempt))
    time.sleep(random.uniform(0, window))

The cap stops a long outage turning into hour-long sleeps. The random.uniform(0, window) is the jitter: two processes that failed at the same instant now wake at different times. Note that full jitter can occasionally produce a near-zero delay, which is fine for transient errors but too aggressive for throttling responses that reset on a per-minute window. For a 429, honour the Retry-After header if the vendor sends one, and use a jitter formula with a guaranteed floor if they don’t.

Three more rules that cost nothing to implement:

1. Never retry a 4xx except 429. A 400 or a 422 will fail identically forever. Every one of those retries is a billed call buying you nothing.
2. Cap retries as a fraction of traffic. A retry budget (say, retries may not exceed some small percentage of total requests) is the only control that sees the system-wide picture. Per-client backoff can’t.
3. Check your layers. If the SDK retries three times, your HTTP client retries three times, and your job runner retries three times, one logical operation is up to twenty-seven calls. This stacking is extremely common and almost never intentional.

For writes, retrying safely needs an idempotency key so the vendor deduplicates rather than creating two of whatever you asked for. Stripe’s Idempotency-Key header is the pattern most others copied. Without it, teams disable retries on writes entirely and then eat manual reconciliation work instead.

Lever five: move latency-tolerant work off the synchronous path

Ask a blunt question about each integration: does a human wait on this response?

Nightly enrichment, classification, report generation, bulk address validation, embedding backfills. Nobody is waiting. Several vendors, including the major LLM providers, offer asynchronous batch endpoints at a discount to the synchronous rate in exchange for a looser delivery window. Rates move, so check current terms rather than a number you read somewhere, but the mechanism is stable and the discount is usually substantial.

Even without a batch endpoint, moving work behind a queue lets you coalesce. Ten events touching the same customer record within a minute become one enrichment call instead of ten. That deduplication window is often worth more than the batch discount.

Lever six: attribute the spend or you’re guessing

Everything above is unfalsifiable until you can answer: which feature, which tenant, which job. The invoice gives you the total and never the cause. What I’d put in place, in order:

1. Route outbound calls through one place. A shared HTTP client wrapper or the caching proxy above. If every service instantiates its own client, you have no chokepoint and no measurement.
2. Emit a counter per call with labels for vendor, endpoint, status class, and cache result. Keep the label set small; per-tenant labels will blow up your metrics cardinality and trade one bill for another.
3. Build one unit-cost metric. Vendor spend per active tenant per month. It’s the only number that survives growth, because raw spend going up during a good quarter is not a problem.
4. Alert on call rate, not on the bill. A rate alert fires in minutes. An invoice fires in thirty days. Grafana, Prometheus, Datadog, CloudWatch, whatever you already run is fine; the point is that the alert exists at all.
5. Track cache hit rate as a first-class SLI. When someone changes a cache key format and the hit rate silently drops to zero, this is the graph that tells you before the invoice does.

Troubleshooting: the bill jumped and nothing shipped

Work down this list. In my experience the cause is usually in the first four.

• Replica count changed. Autoscaling or a Kubernetes replica bump multiplies every cron-driven poller. Check whether your scheduled jobs have leader election, or whether all five pods are dutifully polling the same endpoint.
• Cache hit rate collapsed. A changed key format, a Redis eviction policy under memory pressure, or a TTL someone dropped “temporarily” during debugging. Graph the hit rate over the period, not the current value.
• Retry loop against a persistent 4xx. Look for a flat, high rate of identical failing calls. This has the distinctive signature of high call volume with zero successful responses.
• A non-production environment pointed at production credentials. Staging, a CI pipeline, or an integration test suite running on every commit. Check whether calls are arriving from IPs you don’t recognise.
• Session or token handling regressed. A refactor that dropped a session token parameter, or a service that stopped caching its auth token and now mints a new one per request.
• Legitimate growth in an expensive segment. A few large customers using a costly feature move the bill without moving call count much. Not a bug, but worth ruling in before you go hunting for one.

If the vendor exposes per-key usage breakdowns in their dashboard, split your credentials by service before your next incident. Separate API keys per environment and per major consumer turn a whodunnit into a two-minute lookup.

Common mistakes

• Optimising the endpoint that’s easy to change rather than the one that dominates the bill. Rank by spend first.
• Caching without a per-credential key, and finding out the hard way in a multi-tenant system.
• Treating 304 responses as free when your specific vendor bills them anyway.
• Setting TTLs from a savings target rather than a staleness tolerance, and finding out during a customer escalation.
• Leaving retry logic enabled at three separate layers and calling the result “defence in depth”.
• Adding per-tenant labels to outbound-call metrics and moving the cost from the API vendor to the observability vendor.
• Shipping the optimisation and never checking the next invoice against the prediction.

Best practices worth keeping

• One outbound HTTP client wrapper per service, with timeouts, retries, and metrics configured in exactly one place.
• Separate API credentials per environment and per major consumer, so usage dashboards mean something.
• Cache auth tokens until shortly before expiry. Refresh early, not on every call.
• Add jitter to scheduled jobs as well as to retries. Everything set to run at the top of the hour arrives at the vendor as a spike.
• Write down the staleness tolerance for each cached resource next to its TTL, so the next person doesn’t have to reverse-engineer your reasoning.
• Re-check the pricing page after any vendor product update. Billing units change more often than endpoints do.

FAQ

How do I find out which service is driving my API spend?

Start with the vendor’s own usage dashboard broken down by API key. If all your services share one key, that’s the first thing to fix. Failing that, log every outbound call at the client wrapper with the endpoint and calling service, and aggregate for a week. A week of honest logs beats a month of theorising.

Do ETags actually reduce SaaS API costs, or just help with rate limits?

It depends entirely on the vendor. GitHub documents that an authorised conditional request returning 304 doesn’t count against the primary rate limit. Others count every request regardless of status. Run a controlled test: a known number of conditional calls over a billing period, then compare against the usage report. Don’t assume either way.

Is caching third-party API responses allowed?

Usually yes, but not always, and some terms of service restrict how long you may store certain data. Mapping and place data is a common example of restricted caching. Check the terms for the specific vendor before you build a permanent store, particularly if you’re caching anything personally identifiable.

How much can caching realistically cut an API bill?

Nobody can give you a number without knowing your traffic. Estimate it yourself: for a candidate endpoint, count distinct request keys versus total requests over a day. That ratio is roughly your ceiling for hit rate. Multiply by the endpoint’s share of spend. If the answer is a couple of percent, spend your time elsewhere.

Should I always replace polling with webhooks?

No. The comparison is change frequency against poll frequency. If the resource changes almost as often as you poll, you save little and take on signature verification, redelivery handling, and a reconciliation job. Webhooks win decisively on slow-changing resources and on anything where you’re polling more than a handful of objects.

Does rate limiting my own users reduce vendor costs?

Only when user actions map directly to vendor calls, such as a search box that fires an upstream request per keystroke. Debouncing input and enforcing per-plan quotas helps a lot there. For background processing driven by your own schedules, user rate limits change nothing.

Should I disable retries to save money?

No. Fix them instead. Capped exponential backoff with jitter, no retries on non-429 client errors, a retry budget, and retry logic living at exactly one layer. Disabling retries entirely trades a cost problem for a reliability problem, and reliability problems are more expensive.

The one thing worth remembering

Most efforts to reduce SaaS API costs fail because they start with code. They should start with two questions: what is the vendor actually counting, and which of my calls returned information I already had?

Answer those honestly and the work usually collapses to a handful of unglamorous changes. Raise a page size. Cache a token. Put a lock around a cache refresh. Delete a poller that was watching something that changes twice a week. None of it is clever. All of it shows up on the next invoice, which is the only review that counts.

Need help getting your API spend under control?

This is work I take on regularly. Typical engagements look like:

• Auditing outbound API traffic to find which endpoints, jobs and tenants are driving the bill, and ranking them by spend rather than by how obvious they are.
• Building a caching layer in front of chatty vendor APIs, with Nginx, Redis or Cloudflare Workers, including stampede protection and per-tenant key isolation.
• Replacing polling jobs with webhook receivers or delta-token syncs, with signature verification, idempotent handling and a dead-letter path.
• Auditing timeout and retry configuration across services to remove stacked retries and add proper backoff, jitter and retry budgets.
• Moving latency-tolerant workloads onto batch or queue-based paths, with deduplication windows that cut call volume before it reaches the vendor.
• Setting up call-rate and cache-hit-rate dashboards and alerts in Prometheus, Grafana, Datadog or CloudWatch so the invoice is never the first signal.

If you’ve got a cost graph that’s climbing, a job you suspect is over-polling, or a client wrapper you’re not sure about, send it over and I’ll tell you what I’d look at first.

The post Optimizing API Calls to Reduce SaaS Costs: Six Levers That Actually Move the Bill appeared first on John Nessime.