Nothing is broken in the way you are looking for. The records exist in Zoho. Your job did not error. What happened is that you asked for every deal modified since your last watermark, sorted, and then walked through it page by page while sales reps kept working. Records got modified during the read, changed their position in the sort order, and shifted from page 4 to page 2 after you had already read page 2. So you never saw them.

That is the failure that defines building an ETL pipeline from Zoho CRM to Amazon S3, and it does not look like a bug. It looks like flaky data. It only shows up on busy modules, it never produces an error, and re-running the job produces a different wrong answer, which is the single most confusing symptom in data engineering.

The fix is one rule, and everything else in this post follows from it. This covers that rule, which extraction API to use and when, authentication and the data centre trap that wastes an afternoon, handling deletions (Zoho is genuinely good here), shaping nested CRM JSON for S3, and staying inside your API credits.

The rule: bound both ends of the window

Almost every incremental extract is written like this:

Modified_Time > {last_run}

That query has no upper bound, which means the result set keeps growing while you read it. You are paginating a moving target. Add an upper bound and put it slightly in the past:

Modified_Time > {last_run} AND Modified_Time < {now_minus_lag}

Now the set is frozen. Records modified during your run land after the upper bound and get picked up next time. The extract becomes repeatable: run it twice, get the same rows twice. That property is worth more than any amount of retry logic, because it means a failed run costs you nothing and a suspicious number can be checked by re-running.

Two details that matter. The lag needs to comfortably exceed how long your extract takes plus any clock skew between you and Zoho; ten minutes is a sensible starting point and costs you ten minutes of freshness. And only advance the watermark after the entire window has landed in S3, never after the API call succeeds. Those are different moments, and the gap between them is where data goes missing.

#!/usr/bin/env bash
set -euo pipefail

LAG_MINUTES=10
T1=$(cat state/deals.watermark)
T2=$(date -u -d "-${LAG_MINUTES} minutes" +%Y-%m-%dT%H:%M:%S+00:00)

extract_window "$T1" "$T2"

# Watermark advances only once the data is durably in S3.
echo "$T2" > state/deals.watermark

Keep that state somewhere durable and versioned, not on the box running the job. DynamoDB, Parameter Store, or a small object in S3 all work. A watermark file on an ephemeral runner is a watermark you will lose.

Authentication, and the trap that costs an afternoon

Zoho uses OAuth 2.0. You register a client in the Zoho API Console, get a client ID and secret, exchange a grant token for a refresh token once, and then trade the refresh token for short-lived access tokens from then on. Access tokens last an hour, so cache them rather than requesting one per call.

Here is the part that catches everyone: Zoho runs multiple data centres and they are separate worlds. An account in the EU data centre authenticates against a different accounts domain and calls a different API domain than a US account. A token issued in one region is meaningless in another, and the error you get back does not say “wrong data centre”, it says the token is invalid.

So make the domain a configuration value from the first line of code, never a hardcoded string. The examples below use the US domain. If your org lives elsewhere, both the accounts domain and the API domain change together.

Two more things worth getting right at the start. Request the narrowest OAuth scopes that work, read-only on the modules you actually extract, because scopes are easy to widen later and awkward to explain in an audit. And store the refresh token in Secrets Manager rather than an environment variable, since it does not expire and is effectively a permanent key to your CRM.

Which API to extract with

Three options, and the choice is mostly about volume.

Bulk Read, for anything large

This is the right default for a lake. You POST a job description, Zoho runs the export server-side, and you poll for status or supply a callback URL. When it finishes you get a download URL for a ZIP of CSV. A single job handles up to 200,000 records, with paging beyond that, and it does not consume your standard API limits the way record-by-record calls do.

The important property for our purposes: the pagination problem disappears. Zoho assembles the export on its side, so there is no page-by-page read for records to slip between.

curl "https://www.zohoapis.com/crm/bulk/v8/read" 
  -X POST 
  -H "Authorization: Zoho-oauthtoken $ACCESS_TOKEN" 
  -H "Content-Type: application/json" 
  -d @job.json

{
  "query": {
    "module": { "api_name": "Deals" },
    "fields": ["id", "Deal_Name", "Amount", "Stage", "Modified_Time"],
    "criteria": {
      "group_operator": "and",
      "group": [
        {
          "field": { "api_name": "Modified_Time" },
          "comparator": "greater_than",
          "value": "{T1}"
        },
        {
          "field": { "api_name": "Modified_Time" },
          "comparator": "less_than",
          "value": "{T2}"
        }
      ]
    },
    "page": 1
  }
}

Two limits to design around. The download URL is only valid for about a day, so fetch and land the file promptly rather than queueing it for a later step. And downloads are rate limited per minute, so if you fan out across many modules at once you will start collecting 429s.

COQL, for moderate volumes and real filtering

Zoho’s SQL-like query API. Up to 2,000 records per call and up to 100,000 records total per unique criteria through pagination. Past that, Zoho’s own documentation tells you to use Bulk Read, which is good advice to take rather than work around.

If you do paginate COQL, paginate by key rather than by offset. Sort by id and carry the last ID you saw into the next call, so shifting records cannot cause a skip:

{
  "select_query": "select id, Deal_Name, Amount, Stage, Modified_Time from Deals where (Modified_Time > '{T1}' and Modified_Time < '{T2}') and id > {last_id} order by id asc limit 2000"
}

Note the shape: a closed time window that freezes the set, plus a keyset cursor that walks it deterministically. Offset pagination gives you neither. Zoho’s own documentation uses this pattern in its examples, which is a fair hint about what they expect.

Get Records, for small modules and metadata

The plain module endpoint returns up to 200 records a page and supports an If-Modified-Since header for incremental reads. Fine for lookup tables and small custom modules. Not what you want pointed at a Deals module with six figures of rows.

Worth saying plainly: if all of this sounds like a fortnight of work you would rather not own, a managed connector from Fivetran or Airbyte solves the extraction half and you spend your time on the modelling instead. Price it against your own hours honestly, because the build-it-yourself option is habitually costed at zero.

Deletions, which Zoho actually handles well

An incremental extract can never see a deletion, because a record that no longer exists cannot appear in a query for modified records. Most CRMs make you infer this. Zoho gives you a direct endpoint, and it is one of the nicer parts of the API.

# type: all | recycle | permanent
curl "https://www.zohoapis.com/crm/v8/Deals/deleted?type=all&per_page=200&page=1" 
  -X GET 
  -H "Authorization: Zoho-oauthtoken $ACCESS_TOKEN" 
  -H "If-Modified-Since: {T1}"

The retention windows are generous: records in the recycle bin are retrievable for up to 60 days from deletion, and permanently deleted records for up to 120 days. Compared with warehouses where the detection window is a couple of weeks and not guaranteed, that is a lot of slack. It also means there is no excuse for missing a deletion, since any sane sync interval sits comfortably inside it.

Run the deleted-records call as part of every extract cycle, using the same closed window, and mark the matching rows as deleted in your curated layer rather than removing them from raw. Soft-delete preserves the audit trail and lets you answer “when did this disappear” later.

Even with a good endpoint, run a periodic full ID reconciliation as a backstop, monthly is plenty. Pull just the ID column for the whole module, diff it against the lake, and flag anything you still believe in that Zoho does not. It catches the cases the deletion endpoint does not describe cleanly, such as records merged or moved between modules.

Shaping CRM data for S3

Land raw first, exactly as returned, in its own prefix. Then convert to Parquet in a second step. Skipping the raw layer feels efficient right up to the first time you need to reprocess a month with corrected logic.

# Partition by the window's upper bound, not by a business date.
# One bad run is then one partition to replace.
s3://acme-lake/raw/zoho_crm/deals/window_end=YYYY-MM-DDTHH/
s3://acme-lake/curated/zoho_crm/deals/window_end=YYYY-MM-DDTHH/

Zoho returns nested structures that do not map onto a flat table. Lookup fields come back as objects with an ID and a display name. Multi-select fields and tags come back as arrays. Subforms come back as arrays of objects. You have three choices per field and should make each one deliberately:

• Flatten into columns. A lookup becomes account_id and account_name. Correct for anything you filter or join on.
• Split into a child table. Subforms and line items become their own Parquet dataset keyed by parent ID. Correct when the nested rows are things people count.
• Keep as a JSON string column. Fine for rarely-queried arrays, and Athena can parse it on demand. Not fine for anything in a regular report.

Pin your types explicitly during conversion rather than letting the writer infer them from a batch. Inference is the reason a column is a string in January’s files and a double in February’s, and Athena will happily read both and quietly fail to reconcile them.

Register the result in the Glue Data Catalog and query with Athena. If you expect to apply updates and deletes in place rather than rebuilding partitions, Apache Iceberg earns its extra setup: row-level operations and schema evolution are exactly what a CRM feed generates.

On the AWS side, the orchestration is unremarkable and should stay that way: EventBridge on a schedule, Lambda or a small container for the extract, Step Functions if you need to poll a Bulk Read job and branch on the result. Resist Glue jobs for the extraction itself unless you are already deep in Glue; a Python container you can run locally is easier to debug at seven in the morning.

API credits

Zoho meters API usage as credits, allocated by edition and user count, over a rolling 24-hour window, and shared with every other integration on the org. There are also concurrency limits, so hammering the API in parallel fails differently from exceeding your daily allowance.

• Bulk Read for volume. It sidesteps the standard limits and is the whole reason the API exists.
• Ask for fewer fields. Selecting only the columns you actually land reduces payload and processing on both sides.
• Mind the COQL limit value. Credits scale with the page size you request, so grabbing 2,000 rows costs more than grabbing 200. Larger pages are still usually the better trade, but know you are making it.
• Give the pipeline its own connected app and user. Then consumption is attributable when someone asks who exhausted the credits.
• Back off properly on 429. Exponential backoff with jitter, not a fixed sleep, and treat rate limiting as an expected condition rather than an error.

Troubleshooting

Two runs of the same window return different rows

Open-ended window, offset pagination, or both. Add the upper bound, switch to keyset pagination or Bulk Read, and the symptom disappears. This is the one to check before anything else, because it makes every other investigation unreliable.

Invalid token, but the credentials are definitely right

Data centre mismatch. Confirm which region the org lives in and that your accounts domain and API domain both match it. This is far more common than an actually bad token.

Row counts drift upward over time

Deletions are not being applied. Run the deleted-records endpoint for the last 60 days and see how much comes back, then run a full ID reconciliation to catch the rest.

Bulk Read job finished but the file is gone

The download URL expires after roughly a day. Land the file as soon as the job reports complete, rather than deferring it to a downstream step that might not run until tomorrow.

Athena fails with a schema mismatch across partitions

Type inference changed between runs, usually because a nullable field was all-null in one batch and populated in the next. Define the schema explicitly at write time and reprocess the affected partitions from raw.

A new custom field never appears

You are selecting fields explicitly, which is correct, and nobody told you a field was added. Poll the module’s field metadata on a schedule and alert on changes, so schema drift is a notification rather than a discovery six weeks later.

Frequent 429s

Either concurrency or the download rate limit, depending on which call is failing. Serialise the module extracts rather than fanning them all out at once, and add jittered backoff.

Common mistakes

• An open-ended Modified_Time filter with no upper bound.
• Offset pagination over a dataset that is being modified while you read it.
• Advancing the watermark when the API call succeeds rather than when the data lands.
• Keeping watermark state on an ephemeral runner.
• Hardcoding the Zoho API domain and discovering data centres the hard way.
• Never calling the deleted-records endpoint, so counts only ever grow.
• Deferring the Bulk Read download until after the URL has expired.
• Landing only Parquet with no raw layer, so reprocessing means re-extracting.
• Letting the Parquet writer infer types per batch.
• Flattening subforms into a wide table and double-counting parent rows.
• Fanning out every module in parallel and collecting rate limits.
• Sharing the pipeline’s connected app with other integrations, so nobody can attribute credit usage.
• Storing the refresh token in an environment variable and forgetting it never expires.

Best practices

• Closed read windows with a deliberate lag, always.
• Keyset pagination when you paginate at all, Bulk Read when you can avoid it.
• Watermarks in durable storage, advanced only on confirmed landing.
• Idempotent runs, so re-running a window is always safe.
• Deleted-records endpoint every cycle, full ID reconciliation monthly.
• Raw layer untouched, curated layer derived, partitions keyed by window end.
• Explicit schemas on write, explicit field lists on read.
• Its own connected app, narrow scopes, refresh token in Secrets Manager.
• Jittered exponential backoff and serialised module extracts.
• Alerting on schema drift and on row-count delta against Zoho.
• Encryption and a retention policy on CRM data in S3 from day one, since it is personal data.

FAQ

What is the best way to export Zoho CRM data to S3?

Bulk Read for anything of size, on a schedule, with closed time windows. COQL for moderate volumes where you want real filtering. The plain records endpoint only for small modules. Land raw output first, convert to Parquet second.

Why does my extract return different results each run?

Your query has no upper time bound, so the result set changes while you paginate through it. Bound both ends of the window and put the upper bound a few minutes in the past. That single change makes runs repeatable.

How do I capture deleted records?

Call the module’s deleted-records endpoint each cycle. Recycle-bin deletions stay retrievable for 60 days and permanent deletions for 120, so a daily or hourly sync has plenty of margin. Add a monthly full ID reconciliation as a backstop.

Will this exhaust my Zoho API credits?

Not if you use Bulk Read for the heavy lifting, since it does not draw on standard API limits the way per-record calls do. The credit pool is shared across the org, so give the pipeline its own connected app and monitor consumption rather than finding out when someone else’s integration breaks.

How do I handle subforms and multi-select fields?

Decide per field. Flatten lookups into ID and name columns, split subforms and line items into their own child datasets keyed by parent ID, and keep genuinely peripheral arrays as JSON strings. Flattening a subform into the parent row is how you end up double-counting deals.

Build it or buy a connector?

Buy it if Zoho CRM is one source among several and you have no unusual requirements; managed connectors handle pagination, deletes and schema drift, which is most of the work described here. Build it when you need control over the shape of the output, want to avoid per-row pricing, or already run the surrounding infrastructure.

How fresh can the data realistically be?

Hourly is comfortable and suits nearly all reporting. Every fifteen minutes is achievable with COQL on modest modules. Anything closer to real time means webhooks or notification subscriptions rather than polling, which is a considerably larger commitment for a benefit most dashboards do not use.

The one thing to remember

An extract that returns a different answer each time it runs is not a data quality problem, it is a read consistency problem, and no amount of downstream validation will fix it. Bound both ends of the window, put the upper bound in the past, paginate by key rather than offset, and advance the watermark only when the data is safely in S3.

Get that right and the rest of the pipeline becomes ordinary engineering: land it, convert it, catalogue it, query it. Get it wrong and you will spend months chasing numbers that move every time you look at them.

Want this built or reviewed?

Most CRM pipelines I get handed work fine on a quiet module and lose rows on the busy one, which is the hardest version to notice. Work I take on:

• Building a Zoho CRM to S3 pipeline end to end: OAuth, Bulk Read extraction, Parquet conversion, Glue catalog, Athena query layer.
• Auditing an existing pipeline for read consistency and telling you whether it is silently dropping records.
• Reworking incremental logic into closed windows, keyset pagination and durable watermarks.
• Deletion handling and reconciliation jobs so row counts stay equal to the CRM.
• Modelling nested CRM data into a schema that does not double-count.
• Orchestration and alerting on AWS: EventBridge, Lambda or containers, Step Functions, and monitoring that catches drift early.

Send me your extract query and how you paginate, and I will tell you whether it can lose rows.

The post Run It Twice, Get Two Answers: Building an ETL Pipeline From Zoho CRM to Amazon S3 appeared first on John Nessime.