Integrations very rarely fail on launch day. They fail in week six. That is roughly when the OAuth token first expires, when an admin adds a field nobody mentioned, when a second tool starts consuming the same API allowance, and when enough records have been deleted for the row counts to visibly diverge. None of those conditions exist in staging, which is why staging said yes.

So this is a Salesforce AWS integration checklist aimed at week six rather than at Tuesday. Every item has a pass test, because “we thought about that” is not the same as “we checked.”

1. Credentials and identity

• The integration has its own Salesforce user. Pass: the username is not a person’s, and nobody uses it interactively.
• That user is API-only and cannot log in through the UI. Pass: someone tried and was refused.
• Its profile grants only the objects and fields you actually read. Pass: you have the field-level permission list and can explain every entry.
• Credentials live in a secrets manager, not in environment variables or config files. Pass: the secret is retrievable by ARN and nothing in the repo contains it.
• You have rotated the credential at least once, in production, and the pipeline survived. Pass: there is a dated record of the rotation and no incident attached to it.
• You know what invalidates the refresh token. Pass: you can name the events that revoke it, including password resets and session policy changes on the integration user.

That last item is the classic week-four failure. A refresh token feels permanent right up until an admin does something routine to the user account it belongs to.

2. Network, versions and deadlines

• You know whether the compute needs to be in a VPC, and why. Pass: if it is, you can name the private resource it reaches.
• If it is in a VPC, S3 traffic goes through a gateway endpoint. Pass: the endpoint exists and the route table references it.
• The Salesforce API version in your endpoints is pinned and current. Pass: it is not a version on a retirement list.
• IAM permissions are scoped to the specific resources involved. Pass: no wildcard on S3 buckets or Secrets Manager paths.
• Login IP restrictions and connected app policies are configured deliberately. Pass: you know whether IP relaxation is on and made that choice on purpose.

The API version item has real dates attached, so it is worth stating plainly rather than leaving as a vague “keep current”. Salesforce retired versions 7.0 through 20.0 in Summer ’22 and versions 21.0 through 30.0 in Summer ’25, and has announced that 31.0 through 40.0 retire in June 2028, with the SOAP login() call retiring separately in June 2027. Retired versions return HTTP 410 on REST rather than degrading gracefully.

Usefully, Salesforce warns you first. Calls to legacy versions come back with a Warning header, so this is detectable today rather than on the morning it breaks:

# If this matches anything, you have a deadline you did not know about.
grep -i "Warning:" integration.log | sort -u

3. Correctness

• Deletes are handled, not just creates and updates. Pass: you can point at the code path and name the mechanism it uses.
• Merged records do not double-count. Pass: you have checked what happens to the losing record of a merge.
• Formula and computed fields are either recalculated downstream or documented as snapshots. Pass: there is a written list of which columns are derived.
• Re-running the same window produces the same result. Pass: you ran it twice and diffed the output.
• One real aggregate reconciles against Salesforce for a closed period. Pass: daily and monthly totals both match, not just daily.
• Timezone handling is explicit at the boundary. Pass: someone can say which timezone the date column is in without guessing.

If daily figures reconcile and monthly ones do not, that is a timezone problem at period boundaries, not missing data. Find it now, because finding it inside somebody’s board pack is a worse day.

4. Failure handling

• Retries are scoped to failures that deserve them. Pass: transient errors retry, data errors do not, and you can show which is which.
• Backoff is exponential with jitter. Pass: a fixed sleep does not appear anywhere in the retry path.
• A partial failure leaves recoverable state. Pass: the watermark advances only after data lands, not when the API call returns.
• You can replay an arbitrary historical window on demand. Pass: someone has done it, in production, with a command you could hand to a colleague.
• Rate limiting is treated as an expected condition. Pass: a 429 or a limit error produces a backoff, not an alert and a failed run.

5. Observability

• The pipeline emits a metric on successful completion. Pass: the metric exists and you can graph it.
• An alarm fires when that metric stops arriving. Pass: you disabled the schedule on purpose and the alarm went red.
• Data freshness is measured on the destination side. Pass: you can answer “how far behind are we right now” from a dashboard.
• Salesforce API allowance consumption is graphed. Pass: there is a percentage on a chart with a threshold line.
• Alerts describe symptoms and link to a runbook. Pass: the alarm description contains a URL somebody could follow at 3am.

The second item is the one people skip and it is the only one that matters. An alarm you have never seen fire is a belief, not a control.

6. Cost

• Every resource carries a cost allocation tag. Pass: Cost Explorer filtered to that tag returns a number.
• Log groups have a retention period. Pass: none of them say “Never expire”.
• The schedule matches how the output is actually used. Pass: somebody named the decision the data supports and the latency it tolerates.
• Average object size in the destination is sensible. Pass: you have looked, and it is not thousands of tiny files per partition.
• A budget alert exists on the pipeline’s tag. Pass: it has a threshold and a recipient who is not on holiday.

7. Operations and handover

• The integration has a named owner. Pass: a person, not a team inbox, and they know.
• A runbook exists covering the three most likely failures. Pass: someone who did not build it followed the runbook successfully.
• Infrastructure is defined as code. Pass: you could rebuild it in a fresh account from the repository.
• Schema drift produces a notification. Pass: adding a field in a sandbox generated an alert somewhere.
• Personal data has an access policy and a retention policy. Pass: both are written down and somebody outside the team has read them.
• You know what a Salesforce sandbox refresh does to it. Pass: you have been through one, or you know what will break when you do.

If you only do five

Nobody clears a thirty-item list before a deadline. These five catch most of what actually goes wrong:

1. Rotate the credential once before go-live. It proves the whole credential path works and it is the failure with the longest fuse.
2. Break it deliberately and confirm someone is told. Disable the schedule, watch the alarm, put it back.
3. Reconcile one real number against Salesforce for a closed period, daily and monthly.
4. Set log retention and one cost allocation tag. Five minutes, and it is the difference between a knowable bill and a mystery.
5. Write the runbook and name the owner. If it is nobody’s, it is nobody’s at 3am too.

FAQ

What actually breaks first in production?

Credentials, usually around the first expiry or the first time an admin touches the integration user. After that, schema drift and API allowance contention with a tool somebody else added.

How do I know if I’m on a retiring API version?

Look at the version string in your endpoint URLs, and search your logs for the Warning header Salesforce returns on legacy versions. Retired versions fail with an HTTP 410 rather than degrading, so there is no soft landing.

Is a staging environment enough to sign off?

No, because the conditions that break integrations barely exist there: real volume, real credential lifecycles, real deletions, and a shared API budget. Staging proves the code runs. It cannot prove the thing survives contact with an organisation.

How often should I revisit the list?

Quarterly for the alarm test and the reconciliation, annually for permissions, credentials and API versions. Put both in a calendar, because an integration that works is one nobody thinks about until it does not.

Does this apply to managed connectors too?

Most of it. You inherit retries and pagination from the vendor, but credentials, permissions, reconciliation, cost, alerting and ownership are all still yours. A managed connector removes code, not accountability.

The one thing to remember

“It works” is a statement about today. “It’s ready” is a claim about week six, and the only honest way to make it is to have tested the things that only happen later: rotate the credential, break the schedule, reconcile a number, follow the runbook.

If you cannot point at evidence for an item, it is not done. It is intended, which is a different thing, and the gap between the two is where you will spend your next quarter.

Want a second pair of eyes before go-live?

Readiness reviews are cheap compared with the incidents they prevent, and they go faster with someone who has seen the same six failures repeatedly. Work I take on:

• Pre-launch readiness review against this checklist, with evidence collected rather than assurances taken.
• Credential and permission audit: integration users, profiles, field-level security, secret storage and rotation.
• Reconciliation harness so row counts and aggregates are checked automatically rather than when someone complains.
• Alerting and runbooks, including a failure drill that proves the alerts reach a human.
• Legacy API version audits ahead of the announced retirement deadlines.
• Cost tagging, log retention and budget alerts so the first month’s bill is not a surprise.

Tell me which items on this list you have evidence for, and I will tell you which of the rest matter most for your setup.

The post It Works Isn’t Ready: A Production Checklist for Salesforce AWS Integrations appeared first on John Nessime.

Nothing is broken in the way you are looking for. The records exist in Zoho. Your job did not error. What happened is that you asked for every deal modified since your last watermark, sorted, and then walked through it page by page while sales reps kept working. Records got modified during the read, changed their position in the sort order, and shifted from page 4 to page 2 after you had already read page 2. So you never saw them.

That is the failure that defines building an ETL pipeline from Zoho CRM to Amazon S3, and it does not look like a bug. It looks like flaky data. It only shows up on busy modules, it never produces an error, and re-running the job produces a different wrong answer, which is the single most confusing symptom in data engineering.

The fix is one rule, and everything else in this post follows from it. This covers that rule, which extraction API to use and when, authentication and the data centre trap that wastes an afternoon, handling deletions (Zoho is genuinely good here), shaping nested CRM JSON for S3, and staying inside your API credits.

The rule: bound both ends of the window

Almost every incremental extract is written like this:

Modified_Time > {last_run}

That query has no upper bound, which means the result set keeps growing while you read it. You are paginating a moving target. Add an upper bound and put it slightly in the past:

Modified_Time > {last_run} AND Modified_Time < {now_minus_lag}

Now the set is frozen. Records modified during your run land after the upper bound and get picked up next time. The extract becomes repeatable: run it twice, get the same rows twice. That property is worth more than any amount of retry logic, because it means a failed run costs you nothing and a suspicious number can be checked by re-running.

Two details that matter. The lag needs to comfortably exceed how long your extract takes plus any clock skew between you and Zoho; ten minutes is a sensible starting point and costs you ten minutes of freshness. And only advance the watermark after the entire window has landed in S3, never after the API call succeeds. Those are different moments, and the gap between them is where data goes missing.

#!/usr/bin/env bash
set -euo pipefail

LAG_MINUTES=10
T1=$(cat state/deals.watermark)
T2=$(date -u -d "-${LAG_MINUTES} minutes" +%Y-%m-%dT%H:%M:%S+00:00)

extract_window "$T1" "$T2"

# Watermark advances only once the data is durably in S3.
echo "$T2" > state/deals.watermark

Keep that state somewhere durable and versioned, not on the box running the job. DynamoDB, Parameter Store, or a small object in S3 all work. A watermark file on an ephemeral runner is a watermark you will lose.

Authentication, and the trap that costs an afternoon

Zoho uses OAuth 2.0. You register a client in the Zoho API Console, get a client ID and secret, exchange a grant token for a refresh token once, and then trade the refresh token for short-lived access tokens from then on. Access tokens last an hour, so cache them rather than requesting one per call.

Here is the part that catches everyone: Zoho runs multiple data centres and they are separate worlds. An account in the EU data centre authenticates against a different accounts domain and calls a different API domain than a US account. A token issued in one region is meaningless in another, and the error you get back does not say “wrong data centre”, it says the token is invalid.

So make the domain a configuration value from the first line of code, never a hardcoded string. The examples below use the US domain. If your org lives elsewhere, both the accounts domain and the API domain change together.

Two more things worth getting right at the start. Request the narrowest OAuth scopes that work, read-only on the modules you actually extract, because scopes are easy to widen later and awkward to explain in an audit. And store the refresh token in Secrets Manager rather than an environment variable, since it does not expire and is effectively a permanent key to your CRM.

Which API to extract with

Three options, and the choice is mostly about volume.

Bulk Read, for anything large

This is the right default for a lake. You POST a job description, Zoho runs the export server-side, and you poll for status or supply a callback URL. When it finishes you get a download URL for a ZIP of CSV. A single job handles up to 200,000 records, with paging beyond that, and it does not consume your standard API limits the way record-by-record calls do.

The important property for our purposes: the pagination problem disappears. Zoho assembles the export on its side, so there is no page-by-page read for records to slip between.

curl "https://www.zohoapis.com/crm/bulk/v8/read" 
  -X POST 
  -H "Authorization: Zoho-oauthtoken $ACCESS_TOKEN" 
  -H "Content-Type: application/json" 
  -d @job.json

{
  "query": {
    "module": { "api_name": "Deals" },
    "fields": ["id", "Deal_Name", "Amount", "Stage", "Modified_Time"],
    "criteria": {
      "group_operator": "and",
      "group": [
        {
          "field": { "api_name": "Modified_Time" },
          "comparator": "greater_than",
          "value": "{T1}"
        },
        {
          "field": { "api_name": "Modified_Time" },
          "comparator": "less_than",
          "value": "{T2}"
        }
      ]
    },
    "page": 1
  }
}

Two limits to design around. The download URL is only valid for about a day, so fetch and land the file promptly rather than queueing it for a later step. And downloads are rate limited per minute, so if you fan out across many modules at once you will start collecting 429s.

COQL, for moderate volumes and real filtering

Zoho’s SQL-like query API. Up to 2,000 records per call and up to 100,000 records total per unique criteria through pagination. Past that, Zoho’s own documentation tells you to use Bulk Read, which is good advice to take rather than work around.

If you do paginate COQL, paginate by key rather than by offset. Sort by id and carry the last ID you saw into the next call, so shifting records cannot cause a skip:

{
  "select_query": "select id, Deal_Name, Amount, Stage, Modified_Time from Deals where (Modified_Time > '{T1}' and Modified_Time < '{T2}') and id > {last_id} order by id asc limit 2000"
}

Note the shape: a closed time window that freezes the set, plus a keyset cursor that walks it deterministically. Offset pagination gives you neither. Zoho’s own documentation uses this pattern in its examples, which is a fair hint about what they expect.

Get Records, for small modules and metadata

The plain module endpoint returns up to 200 records a page and supports an If-Modified-Since header for incremental reads. Fine for lookup tables and small custom modules. Not what you want pointed at a Deals module with six figures of rows.

Worth saying plainly: if all of this sounds like a fortnight of work you would rather not own, a managed connector from Fivetran or Airbyte solves the extraction half and you spend your time on the modelling instead. Price it against your own hours honestly, because the build-it-yourself option is habitually costed at zero.

Deletions, which Zoho actually handles well

An incremental extract can never see a deletion, because a record that no longer exists cannot appear in a query for modified records. Most CRMs make you infer this. Zoho gives you a direct endpoint, and it is one of the nicer parts of the API.

# type: all | recycle | permanent
curl "https://www.zohoapis.com/crm/v8/Deals/deleted?type=all&per_page=200&page=1" 
  -X GET 
  -H "Authorization: Zoho-oauthtoken $ACCESS_TOKEN" 
  -H "If-Modified-Since: {T1}"

The retention windows are generous: records in the recycle bin are retrievable for up to 60 days from deletion, and permanently deleted records for up to 120 days. Compared with warehouses where the detection window is a couple of weeks and not guaranteed, that is a lot of slack. It also means there is no excuse for missing a deletion, since any sane sync interval sits comfortably inside it.

Run the deleted-records call as part of every extract cycle, using the same closed window, and mark the matching rows as deleted in your curated layer rather than removing them from raw. Soft-delete preserves the audit trail and lets you answer “when did this disappear” later.

Even with a good endpoint, run a periodic full ID reconciliation as a backstop, monthly is plenty. Pull just the ID column for the whole module, diff it against the lake, and flag anything you still believe in that Zoho does not. It catches the cases the deletion endpoint does not describe cleanly, such as records merged or moved between modules.

Shaping CRM data for S3

Land raw first, exactly as returned, in its own prefix. Then convert to Parquet in a second step. Skipping the raw layer feels efficient right up to the first time you need to reprocess a month with corrected logic.

# Partition by the window's upper bound, not by a business date.
# One bad run is then one partition to replace.
s3://acme-lake/raw/zoho_crm/deals/window_end=YYYY-MM-DDTHH/
s3://acme-lake/curated/zoho_crm/deals/window_end=YYYY-MM-DDTHH/

Zoho returns nested structures that do not map onto a flat table. Lookup fields come back as objects with an ID and a display name. Multi-select fields and tags come back as arrays. Subforms come back as arrays of objects. You have three choices per field and should make each one deliberately:

• Flatten into columns. A lookup becomes account_id and account_name. Correct for anything you filter or join on.
• Split into a child table. Subforms and line items become their own Parquet dataset keyed by parent ID. Correct when the nested rows are things people count.
• Keep as a JSON string column. Fine for rarely-queried arrays, and Athena can parse it on demand. Not fine for anything in a regular report.

Pin your types explicitly during conversion rather than letting the writer infer them from a batch. Inference is the reason a column is a string in January’s files and a double in February’s, and Athena will happily read both and quietly fail to reconcile them.

Register the result in the Glue Data Catalog and query with Athena. If you expect to apply updates and deletes in place rather than rebuilding partitions, Apache Iceberg earns its extra setup: row-level operations and schema evolution are exactly what a CRM feed generates.

On the AWS side, the orchestration is unremarkable and should stay that way: EventBridge on a schedule, Lambda or a small container for the extract, Step Functions if you need to poll a Bulk Read job and branch on the result. Resist Glue jobs for the extraction itself unless you are already deep in Glue; a Python container you can run locally is easier to debug at seven in the morning.

API credits

Zoho meters API usage as credits, allocated by edition and user count, over a rolling 24-hour window, and shared with every other integration on the org. There are also concurrency limits, so hammering the API in parallel fails differently from exceeding your daily allowance.

• Bulk Read for volume. It sidesteps the standard limits and is the whole reason the API exists.
• Ask for fewer fields. Selecting only the columns you actually land reduces payload and processing on both sides.
• Mind the COQL limit value. Credits scale with the page size you request, so grabbing 2,000 rows costs more than grabbing 200. Larger pages are still usually the better trade, but know you are making it.
• Give the pipeline its own connected app and user. Then consumption is attributable when someone asks who exhausted the credits.
• Back off properly on 429. Exponential backoff with jitter, not a fixed sleep, and treat rate limiting as an expected condition rather than an error.

Troubleshooting

Two runs of the same window return different rows

Open-ended window, offset pagination, or both. Add the upper bound, switch to keyset pagination or Bulk Read, and the symptom disappears. This is the one to check before anything else, because it makes every other investigation unreliable.

Invalid token, but the credentials are definitely right

Data centre mismatch. Confirm which region the org lives in and that your accounts domain and API domain both match it. This is far more common than an actually bad token.

Row counts drift upward over time

Deletions are not being applied. Run the deleted-records endpoint for the last 60 days and see how much comes back, then run a full ID reconciliation to catch the rest.

Bulk Read job finished but the file is gone

The download URL expires after roughly a day. Land the file as soon as the job reports complete, rather than deferring it to a downstream step that might not run until tomorrow.

Athena fails with a schema mismatch across partitions

Type inference changed between runs, usually because a nullable field was all-null in one batch and populated in the next. Define the schema explicitly at write time and reprocess the affected partitions from raw.

A new custom field never appears

You are selecting fields explicitly, which is correct, and nobody told you a field was added. Poll the module’s field metadata on a schedule and alert on changes, so schema drift is a notification rather than a discovery six weeks later.

Frequent 429s

Either concurrency or the download rate limit, depending on which call is failing. Serialise the module extracts rather than fanning them all out at once, and add jittered backoff.

Common mistakes

• An open-ended Modified_Time filter with no upper bound.
• Offset pagination over a dataset that is being modified while you read it.
• Advancing the watermark when the API call succeeds rather than when the data lands.
• Keeping watermark state on an ephemeral runner.
• Hardcoding the Zoho API domain and discovering data centres the hard way.
• Never calling the deleted-records endpoint, so counts only ever grow.
• Deferring the Bulk Read download until after the URL has expired.
• Landing only Parquet with no raw layer, so reprocessing means re-extracting.
• Letting the Parquet writer infer types per batch.
• Flattening subforms into a wide table and double-counting parent rows.
• Fanning out every module in parallel and collecting rate limits.
• Sharing the pipeline’s connected app with other integrations, so nobody can attribute credit usage.
• Storing the refresh token in an environment variable and forgetting it never expires.

Best practices

• Closed read windows with a deliberate lag, always.
• Keyset pagination when you paginate at all, Bulk Read when you can avoid it.
• Watermarks in durable storage, advanced only on confirmed landing.
• Idempotent runs, so re-running a window is always safe.
• Deleted-records endpoint every cycle, full ID reconciliation monthly.
• Raw layer untouched, curated layer derived, partitions keyed by window end.
• Explicit schemas on write, explicit field lists on read.
• Its own connected app, narrow scopes, refresh token in Secrets Manager.
• Jittered exponential backoff and serialised module extracts.
• Alerting on schema drift and on row-count delta against Zoho.
• Encryption and a retention policy on CRM data in S3 from day one, since it is personal data.

FAQ

What is the best way to export Zoho CRM data to S3?

Bulk Read for anything of size, on a schedule, with closed time windows. COQL for moderate volumes where you want real filtering. The plain records endpoint only for small modules. Land raw output first, convert to Parquet second.

Why does my extract return different results each run?

Your query has no upper time bound, so the result set changes while you paginate through it. Bound both ends of the window and put the upper bound a few minutes in the past. That single change makes runs repeatable.

How do I capture deleted records?

Call the module’s deleted-records endpoint each cycle. Recycle-bin deletions stay retrievable for 60 days and permanent deletions for 120, so a daily or hourly sync has plenty of margin. Add a monthly full ID reconciliation as a backstop.

Will this exhaust my Zoho API credits?

Not if you use Bulk Read for the heavy lifting, since it does not draw on standard API limits the way per-record calls do. The credit pool is shared across the org, so give the pipeline its own connected app and monitor consumption rather than finding out when someone else’s integration breaks.

How do I handle subforms and multi-select fields?

Decide per field. Flatten lookups into ID and name columns, split subforms and line items into their own child datasets keyed by parent ID, and keep genuinely peripheral arrays as JSON strings. Flattening a subform into the parent row is how you end up double-counting deals.

Build it or buy a connector?

Buy it if Zoho CRM is one source among several and you have no unusual requirements; managed connectors handle pagination, deletes and schema drift, which is most of the work described here. Build it when you need control over the shape of the output, want to avoid per-row pricing, or already run the surrounding infrastructure.

How fresh can the data realistically be?

Hourly is comfortable and suits nearly all reporting. Every fifteen minutes is achievable with COQL on modest modules. Anything closer to real time means webhooks or notification subscriptions rather than polling, which is a considerably larger commitment for a benefit most dashboards do not use.

The one thing to remember

An extract that returns a different answer each time it runs is not a data quality problem, it is a read consistency problem, and no amount of downstream validation will fix it. Bound both ends of the window, put the upper bound in the past, paginate by key rather than offset, and advance the watermark only when the data is safely in S3.

Get that right and the rest of the pipeline becomes ordinary engineering: land it, convert it, catalogue it, query it. Get it wrong and you will spend months chasing numbers that move every time you look at them.

Want this built or reviewed?

Most CRM pipelines I get handed work fine on a quiet module and lose rows on the busy one, which is the hardest version to notice. Work I take on:

• Building a Zoho CRM to S3 pipeline end to end: OAuth, Bulk Read extraction, Parquet conversion, Glue catalog, Athena query layer.
• Auditing an existing pipeline for read consistency and telling you whether it is silently dropping records.
• Reworking incremental logic into closed windows, keyset pagination and durable watermarks.
• Deletion handling and reconciliation jobs so row counts stay equal to the CRM.
• Modelling nested CRM data into a schema that does not double-count.
• Orchestration and alerting on AWS: EventBridge, Lambda or containers, Step Functions, and monitoring that catches drift early.

Send me your extract query and how you paginate, and I will tell you whether it can lose rows.

The post Run It Twice, Get Two Answers: Building an ETL Pipeline From Zoho CRM to Amazon S3 appeared first on John Nessime.