That is the shape of most API pipeline incidents. Not a crash. A success that isn’t one.

This post covers running Apache Airflow on AWS specifically for SaaS and API workloads: pulling from HubSpot, Salesforce, Stripe, Zendesk, Shopify, an internal partner API, whatever. It is organised by failure family rather than by feature, because the Airflow documentation already explains what an operator is and does a poor job of explaining which of these things will page you at 3am. I will cover choosing a deployment model, the concurrency trap that catches almost everyone, retry design, incremental state, secrets, and where the money actually goes.

Why SaaS and API sources break differently

When your source is a database you control, failure is loud: connection refused, deadlock, disk full. When it is somebody else’s SaaS API, three things change.

• You are a guest. The vendor decides your rate limit, and they can change it without telling you. Your pipeline’s correctness now depends on a number in someone else’s config file.
• Errors arrive as valid HTTP. A 429, a 200 with a truncated page, a 200 with an error object in the body. Your HTTP client is happy. Your data is not.
• Tasks spend most of their life waiting. API extraction is I/O bound almost end to end. That sounds harmless and is the root of the most expensive mistakes.

Pick the deployment model before you write a DAG

This decision constrains everything after it and is harder to reverse than people expect. Three realistic options.

Amazon MWAA, provisioned

AWS runs the scheduler, web server, workers, triggerer and metadata database on Fargate; you drop DAGs into an S3 bucket and they get picked up.

Where it wins: real Airflow, custom providers, custom plugins, full control over environment configuration. If your DAGs need arbitrary Python libraries, this option will not fight you.

Where it doesn’t: the environment bills by the hour whether or not anything is running. There is no scale to zero on the base environment. If you sync six APIs once a day and each run takes twenty minutes, you are paying for a mostly idle cluster around the clock. The mw1.micro class exists precisely for the small case, but it collapses the scheduler and worker into a single Fargate task and caps worker autoscale low, so treat it as a dev or isolation tier rather than a cheap production tier.

Amazon MWAA Serverless

You submit workflow definitions and AWS runs each task in its own Fargate container, billing per task duration with a one-minute minimum rather than per environment hour.

Where it wins: spiky or infrequent schedules. If the workload is “six syncs a day, nothing overnight,” the cost profile beats a permanently running environment by a wide margin. Each workflow also gets its own IAM execution role, which is a real security improvement over one shared role per environment.

Where it doesn’t: it leans on declarative YAML workflow definitions based on the DAG Factory format and a curated set of AWS operators. That is a deliberate trade: because the definition is declarative, the service can schedule tasks without executing your DAG code. It also means custom operators, exotic third-party providers and clever Python at parse time are not the sweet spot. It is also available in fewer regions than provisioned MWAA, so check your region before you design around it.

Self-managed on ECS, EKS or a VPS

On EKS with the Kubernetes executor you get per-task pods and tight cost control. On a single VPS from a provider like InterServer or Hetzner, a Docker Compose stack with a Postgres metadata database will run a modest set of API syncs for a fraction of any managed price.

Where it wins: cost at both extremes, and total control. Where it doesn’t: you now own metadata database upgrades, major version migrations, log retention and the 2am scheduler restart. Managed Airflow is a bet that your time is worth more than the hourly premium. For a solo engineer with three pipelines that bet often loses; for a data team of eight it usually wins. Astronomer is the main non-AWS managed option worth pricing alongside these.

Failure family one: the throttle that silently stops throttling

You start with a normal setup: an Airflow pool named crm_api with four slots, and every task that touches the vendor assigned to it. Four concurrent requests, comfortably under the vendor’s limit. This works.

Then you notice those tasks spend nearly all their runtime waiting on HTTP, burning worker slots to sit still. So you switch them to deferrable operators. A deferrable task suspends itself while waiting, releases its worker slot, and hands the waiting to the triggerer, which polls asynchronously. Worker pressure drops. Everything looks better.

And your rate limiting quietly stops working.

By default, a pool does not count tasks in the deferred state as occupying slots. That was deliberate, and the logic is sound in the abstract: a deferred task is not consuming a worker. But if you were using the pool to protect an external API rather than your own workers, it has just stopped doing the job you gave it. Every task can defer at once, and the vendor sees the full fan-out.

The fix is a per-pool flag, include_deferred, which tells the scheduler to count deferred tasks against the slot budget. It is off by default. You can set it when editing the pool in the Airflow UI, or through the API.

The failure signature is what makes this nasty. Nothing errors. Your DAG gets faster. The vendor starts returning 429s that your retry logic absorbs, and the only symptom is that runs take a little longer and occasionally a page goes missing. Weeks can pass. Two related traps in the same family:

• max_active_tasks at the DAG level has the same blind spot with deferred tasks, and there is no equivalent opt-in flag. If you need a hard external concurrency cap, use a pool with include_deferred enabled, not DAG-level concurrency.
• On MWAA, the triggerer runs alongside the scheduler on the same Fargate task, so scheduler count and triggerer capacity are linked. If you go heavily deferrable and your deferred tasks start stalling, scheduler capacity is the thing to look at.

Failure family two: retries that make the outage worse

The default instinct is to set retries high and move on. Against a rate-limited API, a fixed retry delay across many parallel tasks is just a slower version of the same stampede.

What you want is exponential backoff with a ceiling. The shape:

from datetime import timedelta

from airflow.sdk import dag, task

@dag(
    schedule="0 5 * * *",
    catchup=False,
    max_active_runs=1,          # never let two runs of this DAG overlap
    default_args={
        "retries": 5,
        "retry_delay": timedelta(seconds=30),
        "retry_exponential_backoff": True,   # 30s, 60s, 120s, 240s...
        "max_retry_delay": timedelta(minutes=15),  # stop doubling here
        "pool": "crm_api",      # shared budget across every task touching this vendor
    },
    tags=["crm", "extract"],
)
def crm_extract():

    @task(max_active_tis_per_dag=4)
    def fetch_page(page_token: str) -> str:
        ...

crm_extract()

The lines that matter:

• retry_exponential_backoff turns retry_delay into a base rather than a constant, so repeated failures spread out instead of hammering in lockstep.
• max_retry_delay caps the doubling. Without it, a task that fails five times can sit idle for hours and blow past the window you actually cared about.
• max_active_runs=1 is the one people skip. If a run overruns its schedule, the next one starts anyway, and now two runs are fetching the same pages from the same vendor with the same credentials. This is a common way to trigger a rate limit you have never hit before.
• max_active_tis_per_dag limits how many instances of that specific task run concurrently across DAG runs, which is the right knob for dynamically mapped extraction tasks.

One thing Airflow will not do for you: honour a Retry-After header. Airflow’s retry timing is computed from your config, not from the vendor’s response. If the API tells you exactly how long to wait, you have to catch that in your own code and sleep or reschedule accordingly. Ignoring a header the vendor bothered to send is a good way to get your API key throttled harder.

Failure family three: pagination, cursors and the empty page

Back to the Monday message. The specific bug behind most “the numbers are wrong but nothing failed” incidents is a loop that treats any non-error response as a terminating condition. Three rules prevent it:

1. Never infer “done” from an empty result. Terminate on the explicit signal the API gives you: a null next_cursor, a missing Link header, a page count. An empty array with a valid cursor still has more data behind it.
2. Assert the response shape before you use it. Check the status code explicitly and validate that the fields you depend on exist. A 200 carrying {"error": "..."} should raise, not return zero rows.
3. Land raw, transform later. Write the untouched API response to S3 first, then parse from S3. When the vendor changes a field type, you can replay from raw instead of re-extracting from an API that no longer serves that window.

Where to keep incremental state

The tempting pattern is to store the last-seen timestamp in an Airflow Variable and update it at the end of a run. Do not make that your source of truth. If a run dies midway, the Variable is in an undefined state, and clearing and re-running the DAG will not restore it. Airflow’s retry and backfill machinery has no idea it exists.

Better: make each run’s window a function of the run itself, and write output to a deterministic, run-scoped location such as s3://bucket/source=crm/dt=<logical-date>/. Re-running the same interval overwrites the same prefix. That is what makes a task idempotent, and idempotency is the difference between “clear the task and let it rerun” and a two-hour manual repair.

Then overlap your windows deliberately. Many SaaS APIs order results by modified time with eventual consistency, so a record edited at the boundary can appear after you have already moved on. Query a window slightly wider than your schedule interval and rely on an idempotent upsert downstream to absorb the duplicates. Late-arriving data is not an edge case with SaaS sources. It is the normal case.

Airflow’s asset-based scheduling is the clean way to trigger downstream DAGs from this: the extract DAG produces an asset, and the transform DAG runs when the asset updates, rather than being scheduled at a time you hope is late enough.

Failure family four: credentials

API tokens rotate, sometimes on the vendor’s schedule rather than yours. Storing an API key in an Airflow Connection through the UI works, and is the wrong long-term answer: the value lives in the metadata database and there is no rotation story. On AWS, point Airflow’s secrets backend at AWS Secrets Manager. On MWAA that is an environment configuration option:

secrets.backend
  airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend

secrets.backend_kwargs
  {"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables"}

With that in place, a connection lookup for crm_default resolves against the secret at airflow/connections/crm_default. Rotating the credential is a Secrets Manager operation with no Airflow deployment involved.

Two things to know before you turn it on. First, every connection and variable lookup becomes a Secrets Manager API call, and lookups fall through to the backend before hitting the metadata database, so a DAG that reads a Variable at parse time will generate a call on every parse cycle. Move those reads inside tasks. Second, the environment’s execution role needs explicit read permission on the relevant secret ARNs, and if you use a customer-managed KMS key, decrypt permission on that key too.

Worth knowing if you are on Airflow 3: task code can no longer reach the metadata database directly. All runtime interaction goes through the Task Execution API. If you inherited custom operators that open a session and query Airflow’s own tables, that is a migration blocker, not a warning.

Failure family five: the bill

Nobody is surprised by the environment line item. They are surprised by the other four.

• Idle time. A provisioned MWAA environment bills continuously. Compute the ratio of hours billed to hours doing work. If it is bad, that is the argument for MWAA Serverless or for consolidating several thin pipelines into one environment.
• NAT Gateway. This is the classic one. Private-subnet workers calling public SaaS APIs route through a NAT Gateway, which charges hourly and per gigabyte processed. A high-volume extraction pipeline can spend more on NAT than on Airflow. VPC endpoints remove that cost for AWS service traffic, but they do nothing for calls to a third-party API, which is exactly the traffic an API pipeline generates.
• CloudWatch Logs. Task logs go to CloudWatch, and ingestion is billed per gigabyte. Set the Airflow log level per component rather than globally at DEBUG, and set a retention policy on the log groups. The default is to keep logs forever.
• S3 requests. Landing raw API responses one small object per page generates a lot of PUTs. Batch pages into larger objects where you can.

Rates and dimensions change, so model your own workload against the current pricing page rather than trusting a number from a blog post. The point is knowing which four lines to look at.

Troubleshooting Apache Airflow on AWS when API pipelines misbehave

Tasks sit in “queued” and never start

Usually a slot problem, not a broken scheduler. Check, in order: is the pool full; has DAG-level max_active_tasks been hit; is worker autoscaling at its configured maximum. On MWAA, the container and queue utilisation metrics published to CloudWatch tell you which of the three it is far faster than reading scheduler logs.

DAG file changes don’t appear

On MWAA, DAGs sync from S3 on an interval; it is not instant. If a file has been there for several minutes and still hasn’t appeared, it almost always failed to parse. Check the DAG processing logs in CloudWatch rather than the scheduler logs, because a broken import raises there and never reaches the scheduler.

A new provider package won’t install

MWAA installs from your requirements.txt in the DAGs bucket, and from Airflow 2.7.2 onward that file must include a constraint line. Without one, MWAA picks a constraint for you, and pip is free to resolve a provider version that conflicts with the Airflow build in the image.

--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<AIRFLOW_VERSION>/constraints-<PYTHON_VERSION>.txt"

apache-airflow-providers-http
apache-airflow-providers-salesforce

Substitute the literal Airflow version your environment runs and the Python version bundled with it. MWAA does not expand shell variables in that file. Test the requirements file against a local Airflow image before you upload it, because a failed install on MWAA surfaces as a partially working environment rather than a clean error.

A backfill is stuck and you need to clear it

You do not need a web login token for this. MWAA exposes the Airflow REST API through a signed AWS API call, so you can drive it from CI or a runbook with normal IAM credentials:

aws mwaa invoke-rest-api 
  --name MyMWAAEnvironment 
  --path "/dags/crm_extract/clearTaskInstances" 
  --method POST 
  --body '{"dry_run": true}'

Start with dry_run set to true so the response tells you which task instances would be cleared before you actually clear them. Note that the resource paths differ between Airflow 2 and Airflow 3 environments, so confirm against the API version your environment exposes.

Deferred tasks stall forever

If deferred tasks stop resuming while the environment reports healthy, suspect the triggerer rather than your DAG. A triggerer that has lost its ability to process triggers can keep heartbeating normally, so the scheduler sees nothing wrong while every deferred task drifts toward timeout. This class of bug has been fixed and re-fixed upstream, so check your Airflow version’s release notes before assuming it is your code.

Common mistakes

• Switching to deferrable operators without enabling include_deferred on the pools that were protecting the API.
• Leaving max_active_runs unset, so a slow run and the next scheduled run compete for the same rate limit budget.
• Treating an empty response page as the end of pagination.
• Storing the incremental watermark in an Airflow Variable and updating it mid-run.
• Calling an API or reading a Variable at DAG parse time, which executes on every parse cycle rather than once per run.
• Transforming during extraction, so a vendor schema change means re-pulling data the API may no longer serve.
• Sizing the environment for peak concurrency when the actual constraint is the vendor’s rate limit.

Best practices

• One pool per vendor, sized to their published limit with headroom, and include_deferred enabled on every one of them.
• Land raw responses to S3 before parsing. Extraction and transformation are separate tasks with separate failure modes.
• Make every task idempotent and window-scoped, so “clear and rerun” is always a safe repair.
• Overlap extraction windows and deduplicate downstream rather than trusting a vendor’s timestamps to be exact.
• Secrets Manager for credentials, with the execution role scoped to specific secret ARNs.
• Alert on row counts and freshness, not just task state. A green DAG that produced 40% of yesterday’s rows is the failure you actually care about. Shipping Airflow’s StatsD metrics into Prometheus, Grafana Cloud or Datadog makes that a dashboard rather than a discovery.
• Define the environment in Terraform or OpenTofu. Recreating an MWAA environment by hand after a bad configuration change is a bad afternoon.

Frequently asked questions

Is MWAA worth it compared to self-hosting Airflow on EC2?

It depends almost entirely on how many people share the platform. MWAA’s premium buys you managed metadata database upgrades, patched images and version migration support. If one engineer maintains three DAGs, self-hosting on a modest VPS is cheaper and the operational load is real but small. Once several teams depend on the scheduler being up, the premium is easy to justify.

Should I use Step Functions instead of Airflow for API pipelines?

Step Functions is genuinely better for event-driven, AWS-service-centric orchestration with modest branching, and it scales to zero. Airflow wins when you need scheduled batch semantics, backfills over historical windows, dependencies between many pipelines, and a UI that non-platform engineers can use to see why last Tuesday failed. Backfill is usually the deciding feature.

Do deferrable operators reduce my AWS bill?

On provisioned MWAA, they reduce worker pressure, which reduces autoscaling into additional worker instances. The base environment cost is unchanged. On a Kubernetes executor setup where each task is a pod, the saving is more direct. Either way, do not adopt them purely for cost without revisiting your pool configuration first.

How do I handle a vendor with no documented rate limit?

Start conservative, one or two concurrent requests, and instrument the response status codes. Raise the pool size gradually and watch for 429s or rising latency. Latency creeping up under load is often the earlier signal, because some vendors throttle by slowing you down before they start rejecting.

Can Airflow read a Retry-After header automatically?

No. Airflow computes retry timing from retry_delay and the backoff settings on the task. If a vendor sends Retry-After, you need to handle it in your own request code or in a custom operator.

What breaks when upgrading to Airflow 3?

The big one for API pipelines is that task code can no longer access the metadata database directly; everything goes through the Task Execution API. Imports also move to the airflow.sdk namespace, and several core operators now live in the standard provider package. Audit custom operators first, since that is where direct database access hides. MWAA requires you to be on the latest Airflow 2 minor version before a major upgrade, so plan two steps.

How many DAGs can one MWAA environment handle?

The binding constraint is usually the metadata database and scheduler CPU, not DAG count. Watch metadata database memory and scheduler CPU utilisation; when either saturates, you either move up an environment class or split into multiple environments. Splitting also gives you blast-radius isolation, which matters more than people expect.

Wrapping up

Running Apache Airflow on AWS for SaaS and API pipelines is mostly not an Airflow problem. The scheduler works. The operators work. What bites is the gap between “the task succeeded” and “the data is correct,” and that gap lives in concurrency settings, pagination logic and retry design rather than anywhere Airflow will warn you about.

If you take one thing away: a green DAG is not a signal that your data is complete. Enable include_deferred on the pools protecting your vendors, terminate pagination on an explicit signal instead of an empty page, make every task idempotent, and alert on row counts. Those four things prevent most of the incidents that never show up as a failed task.

Need help with your Airflow pipelines on AWS?

I work with teams running data and API pipelines on AWS, usually somewhere between “it works but nobody trusts it” and “we need to move off cron.” Things I can help with:

• Reviewing existing DAGs for silent data loss: pagination logic, retry behaviour, pool and concurrency configuration.
• Choosing between MWAA provisioned, MWAA Serverless and self-managed Airflow, with a cost model for your actual schedule rather than a generic comparison.
• Building SaaS extraction pipelines that are idempotent and safely re-runnable, landing raw to S3 with incremental windows that survive failure.
• Cutting MWAA cost: environment right-sizing, NAT Gateway traffic, CloudWatch log volume and dependency install time.
• Airflow 2 to 3 migration audits, focused on custom operators and direct metadata database access.
• Data freshness and volume alerting in Grafana or CloudWatch, so you learn about a partial sync before the business does.

If something specific is broken, send me the DAG file, the task log, or the CloudWatch metrics for the run that went wrong. It is usually faster to look at the real thing than to describe it.

The post Apache Airflow on AWS: Building SaaS and API Pipelines That Don’t Lie to You appeared first on John Nessime.

That is usually the moment someone says “let’s just pull it into a warehouse.” A Jira analytics pipeline built on AWS Lambda, S3 and Athena is a reasonable answer to that, and it is genuinely cheap to run. It is also easy to build a version that looks correct for two weeks and then quietly reports numbers that are thirty percent wrong.

This post covers the extraction and modelling problems that actually cost you time: the search endpoint Atlassian removed, the change history that truncates without erroring, the incremental load pattern that duplicates rows across partitions, and the S3 layout decisions that decide whether Athena costs you pennies or hundreds. Code where it clarifies something, and honest notes on what I would skip.

Where Jira’s own reporting genuinely stops

Give the built-in tooling its due first. Jira’s velocity, burndown and control charts are fine for a single team inspecting its own recent work, they need no infrastructure, and they update instantly. Marketplace apps like eazyBI and Custom Charts cover a lot of ground without you writing a line of Python. If your question is “how did this sprint go,” you do not need a pipeline.

Where it stops is anything that crosses a boundary. Comparing lead time across projects that use different workflows. Joining ticket data to deploy events from your CI system or incident data from PagerDuty. Retaining a consistent view of history after someone renames a status or archives a board. Answering a question nobody anticipated when the board was configured. Those need the raw data somewhere you control, in a shape you decide.

The shape of the pipeline

The architecture is unremarkable, which is the point:

1. EventBridge Scheduler triggers a Lambda function on a schedule.
2. Lambda reads a Jira API token from Secrets Manager and pages through the Jira Cloud REST API.
3. It writes Parquet files to S3, partitioned by load date.
4. The Glue Data Catalog holds the table definitions, with partition projection so nothing has to crawl.
5. Athena queries S3 directly. Grafana, Power BI, Metabase or QuickSight sit on top of Athena.

No cluster, no always-on database, nothing to patch. The whole thing costs about as much as a small EC2 instance for a mid-sized Jira site, and most of that is S3 storage. Terraform or CloudFormation to define it, GitHub Actions to deploy it.

The complexity is not in the wiring. It is in four places, and they are worth taking in order.

Failure family 1: the search endpoint you were probably going to use is gone

Almost every Jira extraction tutorial and a good number of client libraries still reach for GET /rest/api/3/search. Atlassian removed it from Jira Cloud. It returns 410 Gone. If you are copying a script from a blog post that predates the change, this is the first thing that breaks, and at least it breaks loudly.

The replacement is /rest/api/3/search/jql, available as both GET and POST. Use POST for anything real, because JQL strings get long and you avoid URL encoding entirely. Three behavioural changes matter more than the URL:

Pagination is cursor-based, and there is no total

startAt is gone. You get a nextPageToken back and hand it to the next request. There is no total in the response, which means any progress bar, any “expected N rows” sanity check, and any loop that terminated on startAt >= total needs rewriting.

If you only need a count, there is a separate operation, POST /rest/api/3/search/approximate-count, which takes a JQL body and returns an approximate figure without paging through results. It is genuinely useful as a reconciliation check: run it before extraction, compare against the row count you actually wrote, and alert on a large gap.

There have been persistent community reports of isLast behaving unreliably on this endpoint, including tokens that chain without ever terminating. Do not trust isLast as your loop condition. Terminate on the absence of nextPageToken, and put a hard page cap in as a circuit breaker so a bad token cannot burn your entire Lambda budget in one invocation.

def search_issues(session, base_url, jql, fields, max_pages=2000):
    """Page through /search/jql. Terminates on missing nextPageToken,
    not on isLast, which has been reported as unreliable."""
    token = None
    for _ in range(max_pages):
        body = {"jql": jql, "fields": fields, "maxResults": 100}
        if token:
            body["nextPageToken"] = token
        r = session.post(f"{base_url}/rest/api/3/search/jql",
                         json=body, timeout=60)
        r.raise_for_status()
        page = r.json()
        for issue in page.get("issues", []):
            yield issue
        token = page.get("nextPageToken")
        if not token:
            return
    raise RuntimeError("page cap hit, refusing to loop further")

You have to ask for fields explicitly

The new endpoint does not hand you every field by default. Omit fields and you get essentially nothing back. This is the failure that looks like success: the pipeline runs, files land in S3, row counts look plausible, and every analytical column is null.

Be explicit and be narrow. Every field you request costs response size, and Jira sites accumulate hundreds of custom fields nobody uses. Name what you need:

FIELDS = [
    "summary", "status", "issuetype", "project", "priority",
    "assignee", "reporter", "created", "updated", "resolutiondate",
    "labels", "components", "parent",
    "customfield_10016",   # story points on this site, verify yours
]

Custom field IDs are per-site. Do not hardcode one you read in someone else’s blog post. Pull /rest/api/3/field once, find the field by name, and either store the mapping in config or resolve it at runtime and log what it resolved to. When a Jira admin rebuilds a field, an ID-based pipeline goes null and a name-resolving pipeline keeps working.

Rate limits and the fifteen-minute wall

Jira Cloud applies cost-based rate limiting and returns HTTP 429 when you exceed it, typically with a Retry-After header. Respect that header rather than inventing your own backoff. A naive retry loop that ignores it turns a brief throttle into a sustained one.

Lambda’s hard ceiling is fifteen minutes. A full historical backfill of a large Jira site will not finish in one invocation, and the ugly failure mode is a function that times out at minute fifteen having written half its data with no record of where it stopped. Two ways out:

• Shard the work. Fan out one Lambda invocation per project key, or per month of created date. Each one is small, independently retryable, and finishes well inside the limit.
• Checkpoint and continue. Persist the current nextPageToken to DynamoDB or S3 after each page. When the function is close to its deadline, stop cleanly and let Step Functions re-invoke it from the checkpoint.

Sharding is simpler and I reach for it first. Checkpointing is what you need when a single project is itself too large. Either way, watch Lambda’s ephemeral storage: the default /tmp allocation is 512 MB, and buffering a large Parquet write there will fail before your API calls do. Raise it or stream to S3 in chunks.

Failure family 2: the change history is the whole point, and it truncates silently

Current issue state answers almost none of the interesting questions. How long a ticket spent waiting for review, how many times it bounced back from QA, when it actually entered development rather than when someone remembered to drag the card, all of that lives in the changelog. Without it you have a list of tickets. With it you have a process.

Here is the trap. Requesting an issue with expand=changelog returns a capped number of history entries, commonly the first hundred, and it does not tell you it truncated. Well-worn tickets with lots of field edits blow past that easily. Your data does not error, it just quietly loses the later transitions, which are usually the ones near completion. Cycle time comes out looking better than reality.

Two correct approaches:

• GET /rest/api/3/issue/{issueIdOrKey}/changelog and page it properly. Correct, but it is one request per issue, which is brutal against rate limits on a large site.
• POST /rest/api/3/changelog/bulkfetch, which accepts issueIdsOrKeys and an optional fieldIds filter, with the same nextPageToken pagination. Far fewer round trips.

Bulk fetch is the one I would use, with a caveat: it has carried an experimental designation, so pin your expectations and keep the per-issue path available as a fallback. Filter fieldIds to status if status history is all you model. That cuts the response size enormously, because most changelog volume is description edits and label churn nobody analyses.

Store changelog as its own narrow table, one row per field change, not nested inside the issue record. Athena can handle nested structures, but flat is dramatically easier to reason about in SQL and much cheaper to scan.

Failure family 3: the incremental load that duplicates everything

This is the one that bites hardest, because nothing fails. Everything runs green for weeks and the numbers are wrong the entire time.

The obvious incremental design is a watermark: track the last successful run, then pull updated >= watermark on each run and append the results to S3. It is the standard pattern and it works for immutable event data.

Jira issues are not immutable. A ticket created in January and touched every week appears in every weekly extract. If you append each run into its own partition, that one issue now exists as a row in twenty partitions. Athena is doing exactly what you asked: SELECT count(*) FROM jira_issues WHERE status = 'Open' counts twenty things. Your open bug count is inflated, your throughput is inflated, and the inflation is proportional to how much a ticket gets edited, which correlates with how contentious it was. The busiest work is the most over-counted.

Nobody catches this from the dashboard, because the numbers move in the right direction and look approximately sane. It surfaces months later when someone reconciles against a JQL query in Jira and the two disagree.

Pick one of three fixes, deliberately

Full snapshot per load, dedupe at read time. Keep every version, partition by load_date, and always read through a view that takes the latest row per issue key. Storage is cheap, history is free, and you can answer “what did the board look like in March” without any extra machinery. The cost is that every query pays for the deduplication.

CREATE OR REPLACE VIEW jira_issues_current AS
SELECT * FROM (
  SELECT
    i.*,
    ROW_NUMBER() OVER (
      PARTITION BY issue_key
      ORDER BY load_date DESC, updated DESC
    ) AS rn
  FROM jira_issue_snapshot i
  WHERE load_date >= date_format(current_date - interval '7' day, '%Y-%m-%d')
) WHERE rn = 1;

The load_date filter inside the view matters. Without it the deduplication window scans the entire table on every query, which is the single most common way a cheap Athena setup becomes an expensive one.

Overwrite the affected partitions. Partition by something stable, usually issue created month, and rewrite whole partitions when any issue in them changes. Clean reads, no dedupe cost, but you now own read-modify-write logic in Lambda and a concurrency problem if two runs overlap.

Use an ACID table format. Apache Iceberg gives you real MERGE INTO semantics on S3, and Athena supports it natively. This is the right answer if you are already running Iceberg elsewhere or if the pipeline will grow to a dozen sources. It is not worth adopting solely to load one Jira site, because you inherit compaction and snapshot expiry as ongoing maintenance.

For a single Jira site feeding a handful of dashboards, snapshot plus a dedupe view is what I would build. It has the fewest moving parts and it gives you point-in-time history as a side effect, which you will want the first time someone asks a retrospective question.

Deletions and moves

A JQL watermark query never returns deleted issues, so they persist in your data forever. Same for issues moved out of scope or into an archived project. Periodically reconcile: pull the full set of issue keys with a minimal fields list, compare against what you hold, and mark the difference. Monthly is usually enough. Skip this and your historical counts drift upward permanently.

Failure family 4: the S3 layout that decides your Athena bill

Athena’s standard pricing model bills on bytes scanned, rounded up, with a small per-query minimum. DDL statements are free. That means your storage layout, not your SQL, is what determines cost.

Three things do almost all the work:

• Write Parquet, not JSON. Columnar storage lets Athena read only the columns your query touches. A dashboard selecting five columns from a forty-column table scans a small fraction of what the equivalent JSON would. Snappy or Zstd compression on top of that.
• Avoid tiny files. A Lambda that runs hourly and writes one small file per invocation produces thousands of objects. Athena spends more time opening files than reading them. Batch writes so files land in the low hundreds of megabytes, or run a periodic compaction job.
• Use partition projection. This is the one people skip and then wonder why queries have a fixed few-second overhead.

Partition projection lets Athena calculate partition locations from table properties instead of calling GetPartitions against the Glue Data Catalog. On a table with a couple of years of daily partitions, that lookup is real latency on every single query, and it grows as the table does. Projection removes it, and removes your need for a Glue crawler entirely, which is one less scheduled job and one less IAM role.

CREATE EXTERNAL TABLE jira_issue_snapshot (
  issue_id        string,
  issue_key       string,
  project_key     string,
  issue_type      string,
  status          string,
  status_category string,
  assignee_id     string,
  created         timestamp,
  updated         timestamp,
  resolutiondate  timestamp,
  story_points    double
)
PARTITIONED BY (load_date string)
STORED AS PARQUET
LOCATION 's3://example-jira-lake/issue_snapshot/'
TBLPROPERTIES (
  'projection.enabled'                = 'true',
  'projection.load_date.type'         = 'date',
  'projection.load_date.format'       = 'yyyy-MM-dd',
  'projection.load_date.range'        = 'NOW-3YEARS,NOW',
  'projection.load_date.interval'     = '1',
  'projection.load_date.interval.unit'= 'DAYS',
  'storage.location.template'         =
    's3://example-jira-lake/issue_snapshot/load_date=${load_date}'
);

One caution that surprises people: projection describes partitions Athena will look for, not partitions that exist. Set a range wider than your data and queries without a load_date filter will probe empty prefixes. Keep the range tight to what you actually hold.

Finally, set BytesScannedCutoffPerQuery on the Athena workgroup. It kills any query that exceeds a scan threshold. One analyst running SELECT * against three years of data in a BI tool’s preview pane is the classic surprise line item, and this stops it at the source.

Modelling time in status

Once status changelog rows are landed flat, the core metric is a window function. Each transition’s duration is the gap to the next transition on the same issue:

WITH transitions AS (
  SELECT
    issue_key,
    to_status,
    changed_at,
    LEAD(changed_at) OVER (
      PARTITION BY issue_key ORDER BY changed_at
    ) AS next_changed_at
  FROM jira_changelog
  WHERE field_id = 'status'
    AND load_date >= date_format(current_date - interval '90' day, '%Y-%m-%d')
)
SELECT
  issue_key,
  to_status,
  SUM(date_diff('second', changed_at,
                COALESCE(next_changed_at, current_timestamp))) / 3600.0
    AS hours_in_status
FROM transitions
GROUP BY issue_key, to_status;

The COALESCE handles the current status, which has no successor transition. Two modelling decisions to make consciously: whether to subtract non-working hours, and how to treat a ticket that moves backwards through the workflow. Both are business questions, not technical ones, and both should be settled in a documented view rather than reimplemented in each dashboard.

Map raw status names to a stable category early. Teams rename statuses constantly, and a metric keyed on the literal string breaks the moment someone changes “In Review” to “Peer Review”. A small mapping table joined at query time keeps history comparable.

Troubleshooting

• Every analytical column is null. You did not pass fields, or you passed a custom field ID from another site. Log the resolved field list on every run.
• Counts higher than the same JQL in Jira. Duplicate rows across partitions. Check whether your query goes through the dedupe view or straight at the base table.
• Extraction loops forever. Do not terminate on isLast. Terminate on missing nextPageToken and keep a page cap.
• HTTP 410 from the API. You are still calling the removed /rest/api/3/search. Check your client library version too, not just your own code.
• Cycle times suspiciously low. Truncated changelog. Move to the dedicated changelog endpoint or bulk fetch.
• Athena returns zero rows but the files are there. Partition projection range does not cover the partition, or the S3 prefix does not match storage.location.template. Compare a real object key against the template character by character.
• Queries slow down as the table grows. Either you have no projection and Glue lookups dominate, or you have accumulated small files.
• Authentication failures overnight with no deploy. API token expired or was revoked. Alert on the specific status code rather than on “run failed”.

Common mistakes

• Appending incremental extracts without a deduplication strategy.
• Building on current issue state and adding changelog later. Retrofitting history is far more work than including it from the start.
• Hardcoding custom field IDs copied from documentation.
• Storing the API token in a Lambda environment variable instead of Secrets Manager.
• Running an hourly schedule when the dashboard is read once a day. You pay in small files and API quota for freshness nobody uses.
• Letting BI tools query base tables directly instead of curated views.
• Never reconciling against Jira. A scheduled check comparing approximate count to your row count catches drift within a day.

Best practices for a Jira analytics pipeline that survives

• Land raw API responses to S3 before transforming. When your parsing is wrong, and it will be, you replay from raw rather than re-hammering the API.
• Make the load idempotent. Re-running for the same date should produce the same result, not a second copy.
• Define the pipeline in Terraform or CloudFormation and deploy it from CI. The IAM policy for Lambda and Athena is fiddly and you do not want to rebuild it from memory.
• Alert on a run that succeeds with zero rows, not just on runs that error. Silent empty loads are the more common failure.
• Scope the Jira token to a service account with read access to exactly the projects you need.
• Put an S3 lifecycle policy on the raw zone. It grows faster than you expect and nobody queries last year’s raw JSON.
• Expose curated views, not tables, to Grafana, Metabase, Power BI or QuickSight, and enable Athena’s query result reuse for repeated dashboard loads.

Frequently asked questions

Should I use Lambda or Glue for Jira extraction?

Lambda, for an API-driven pull. The work is mostly waiting on HTTP responses, which Spark’s distributed compute does nothing for, and you would be paying for a Glue job’s minimum billing on something that is idle. Glue earns its place downstream, if you have heavy joins across several sources. Athena CTAS often covers that too.

How often should the pipeline run?

Match the decision cadence, not the data cadence. Delivery metrics are reviewed weekly or in sprint ceremonies, so daily is almost always enough and produces far better file sizes. Reserve hourly for something with a real-time consumer, and know that you are buying that freshness with small-file overhead and API quota.

Can I use webhooks instead of polling?

You can, and for near-real-time reaction it is the right tool. For analytics it is a poor primary source, because a missed delivery leaves a permanent hole you have no way to detect. The pattern that works is webhooks for freshness plus a scheduled reconciliation pull as the source of truth. If you only build one, build the scheduled pull.

Does this work with Jira Data Center or Server?

The AWS half is identical. The extraction half is not. Cursor pagination and the /search/jql endpoint are Cloud-only changes, so self-hosted instances still use the older offset-based /rest/api/2/search. Write the extractor behind an interface if you need to support both, and expect the auth model to differ as well.

How much does an Athena-based Jira pipeline cost to run?

The mechanism matters more than any figure I could quote. You pay for S3 storage, Lambda invocation time, and Athena per byte scanned with a small per-query minimum. Jira issue data compresses extremely well as Parquet, so storage stays small. The variable is dashboard query volume, which is why partitioning and workgroup scan limits matter. Model it against current AWS rates and your own expected query count.

Why not just use eazyBI or a Jira reporting app?

Often you should. If your questions stay inside Jira, a Marketplace app gets you there in an afternoon with no infrastructure. The case for a pipeline is joining Jira to data that lives elsewhere, retaining history the app does not, or needing your data in a warehouse you already run. Build the pipeline when the app has actually failed you, not in anticipation.

Should I load into Redshift instead of querying S3?

Only if you have concurrent BI users hitting the same tables constantly and Athena’s per-query latency is a real complaint. Jira data volumes are small, the query pattern is bursty, and Athena’s zero-idle-cost model fits that far better. Redshift makes sense as a consolidation layer across many sources, not for one issue tracker.

The one thing to carry away

The hard part of a Jira analytics pipeline is not the AWS wiring. Lambda, S3, Glue and Athena will be working within a day. The hard part is that Jira issues are mutable, so an incremental load that appends is an incremental load that duplicates, and it does so without a single error in your logs.

Decide your deduplication strategy before you write the first extract, not after someone reconciles a dashboard against JQL and finds a gap. Snapshot with a read-time dedupe view, partition projection so queries stay fast as history accumulates, and a scheduled reconciliation check that alerts on drift. Get those three right and the rest is plumbing.

Need this built or fixed?

I design and run serverless data pipelines on AWS, and Jira extraction has more sharp edges than most sources. Things I can help with directly:

• Migrating an extractor off the removed /rest/api/3/search endpoint to cursor-based /search/jql without losing rows in the cutover
• Auditing an existing Jira pipeline for duplicate rows and reconciling your numbers back against JQL
• Building changelog-based cycle time and time-in-status models that survive workflow renames
• Restructuring an S3 layer with Parquet, partition projection and file compaction to cut Athena scan costs
• Packaging the whole thing as Terraform or CloudFormation with CI deployment and least-privilege IAM
• Connecting Athena to Grafana, Power BI, Metabase or QuickSight with curated views instead of raw tables

Send me your extractor code, a Glue table definition, or an Athena query that is scanning more than it should, and I will tell you what I would change.

The post Building a Jira Analytics Pipeline with AWS Lambda and Athena (Without Double-Counting Everything) appeared first on John Nessime.

Nothing is broken in the way you are looking for. The records exist in Zoho. Your job did not error. What happened is that you asked for every deal modified since your last watermark, sorted, and then walked through it page by page while sales reps kept working. Records got modified during the read, changed their position in the sort order, and shifted from page 4 to page 2 after you had already read page 2. So you never saw them.

That is the failure that defines building an ETL pipeline from Zoho CRM to Amazon S3, and it does not look like a bug. It looks like flaky data. It only shows up on busy modules, it never produces an error, and re-running the job produces a different wrong answer, which is the single most confusing symptom in data engineering.

The fix is one rule, and everything else in this post follows from it. This covers that rule, which extraction API to use and when, authentication and the data centre trap that wastes an afternoon, handling deletions (Zoho is genuinely good here), shaping nested CRM JSON for S3, and staying inside your API credits.

The rule: bound both ends of the window

Almost every incremental extract is written like this:

Modified_Time > {last_run}

That query has no upper bound, which means the result set keeps growing while you read it. You are paginating a moving target. Add an upper bound and put it slightly in the past:

Modified_Time > {last_run} AND Modified_Time < {now_minus_lag}

Now the set is frozen. Records modified during your run land after the upper bound and get picked up next time. The extract becomes repeatable: run it twice, get the same rows twice. That property is worth more than any amount of retry logic, because it means a failed run costs you nothing and a suspicious number can be checked by re-running.

Two details that matter. The lag needs to comfortably exceed how long your extract takes plus any clock skew between you and Zoho; ten minutes is a sensible starting point and costs you ten minutes of freshness. And only advance the watermark after the entire window has landed in S3, never after the API call succeeds. Those are different moments, and the gap between them is where data goes missing.

#!/usr/bin/env bash
set -euo pipefail

LAG_MINUTES=10
T1=$(cat state/deals.watermark)
T2=$(date -u -d "-${LAG_MINUTES} minutes" +%Y-%m-%dT%H:%M:%S+00:00)

extract_window "$T1" "$T2"

# Watermark advances only once the data is durably in S3.
echo "$T2" > state/deals.watermark

Keep that state somewhere durable and versioned, not on the box running the job. DynamoDB, Parameter Store, or a small object in S3 all work. A watermark file on an ephemeral runner is a watermark you will lose.

Authentication, and the trap that costs an afternoon

Zoho uses OAuth 2.0. You register a client in the Zoho API Console, get a client ID and secret, exchange a grant token for a refresh token once, and then trade the refresh token for short-lived access tokens from then on. Access tokens last an hour, so cache them rather than requesting one per call.

Here is the part that catches everyone: Zoho runs multiple data centres and they are separate worlds. An account in the EU data centre authenticates against a different accounts domain and calls a different API domain than a US account. A token issued in one region is meaningless in another, and the error you get back does not say “wrong data centre”, it says the token is invalid.

So make the domain a configuration value from the first line of code, never a hardcoded string. The examples below use the US domain. If your org lives elsewhere, both the accounts domain and the API domain change together.

Two more things worth getting right at the start. Request the narrowest OAuth scopes that work, read-only on the modules you actually extract, because scopes are easy to widen later and awkward to explain in an audit. And store the refresh token in Secrets Manager rather than an environment variable, since it does not expire and is effectively a permanent key to your CRM.

Which API to extract with

Three options, and the choice is mostly about volume.

Bulk Read, for anything large

This is the right default for a lake. You POST a job description, Zoho runs the export server-side, and you poll for status or supply a callback URL. When it finishes you get a download URL for a ZIP of CSV. A single job handles up to 200,000 records, with paging beyond that, and it does not consume your standard API limits the way record-by-record calls do.

The important property for our purposes: the pagination problem disappears. Zoho assembles the export on its side, so there is no page-by-page read for records to slip between.

curl "https://www.zohoapis.com/crm/bulk/v8/read" 
  -X POST 
  -H "Authorization: Zoho-oauthtoken $ACCESS_TOKEN" 
  -H "Content-Type: application/json" 
  -d @job.json

{
  "query": {
    "module": { "api_name": "Deals" },
    "fields": ["id", "Deal_Name", "Amount", "Stage", "Modified_Time"],
    "criteria": {
      "group_operator": "and",
      "group": [
        {
          "field": { "api_name": "Modified_Time" },
          "comparator": "greater_than",
          "value": "{T1}"
        },
        {
          "field": { "api_name": "Modified_Time" },
          "comparator": "less_than",
          "value": "{T2}"
        }
      ]
    },
    "page": 1
  }
}

Two limits to design around. The download URL is only valid for about a day, so fetch and land the file promptly rather than queueing it for a later step. And downloads are rate limited per minute, so if you fan out across many modules at once you will start collecting 429s.

COQL, for moderate volumes and real filtering

Zoho’s SQL-like query API. Up to 2,000 records per call and up to 100,000 records total per unique criteria through pagination. Past that, Zoho’s own documentation tells you to use Bulk Read, which is good advice to take rather than work around.

If you do paginate COQL, paginate by key rather than by offset. Sort by id and carry the last ID you saw into the next call, so shifting records cannot cause a skip:

{
  "select_query": "select id, Deal_Name, Amount, Stage, Modified_Time from Deals where (Modified_Time > '{T1}' and Modified_Time < '{T2}') and id > {last_id} order by id asc limit 2000"
}

Note the shape: a closed time window that freezes the set, plus a keyset cursor that walks it deterministically. Offset pagination gives you neither. Zoho’s own documentation uses this pattern in its examples, which is a fair hint about what they expect.

Get Records, for small modules and metadata

The plain module endpoint returns up to 200 records a page and supports an If-Modified-Since header for incremental reads. Fine for lookup tables and small custom modules. Not what you want pointed at a Deals module with six figures of rows.

Worth saying plainly: if all of this sounds like a fortnight of work you would rather not own, a managed connector from Fivetran or Airbyte solves the extraction half and you spend your time on the modelling instead. Price it against your own hours honestly, because the build-it-yourself option is habitually costed at zero.

Deletions, which Zoho actually handles well

An incremental extract can never see a deletion, because a record that no longer exists cannot appear in a query for modified records. Most CRMs make you infer this. Zoho gives you a direct endpoint, and it is one of the nicer parts of the API.

# type: all | recycle | permanent
curl "https://www.zohoapis.com/crm/v8/Deals/deleted?type=all&per_page=200&page=1" 
  -X GET 
  -H "Authorization: Zoho-oauthtoken $ACCESS_TOKEN" 
  -H "If-Modified-Since: {T1}"

The retention windows are generous: records in the recycle bin are retrievable for up to 60 days from deletion, and permanently deleted records for up to 120 days. Compared with warehouses where the detection window is a couple of weeks and not guaranteed, that is a lot of slack. It also means there is no excuse for missing a deletion, since any sane sync interval sits comfortably inside it.

Run the deleted-records call as part of every extract cycle, using the same closed window, and mark the matching rows as deleted in your curated layer rather than removing them from raw. Soft-delete preserves the audit trail and lets you answer “when did this disappear” later.

Even with a good endpoint, run a periodic full ID reconciliation as a backstop, monthly is plenty. Pull just the ID column for the whole module, diff it against the lake, and flag anything you still believe in that Zoho does not. It catches the cases the deletion endpoint does not describe cleanly, such as records merged or moved between modules.

Shaping CRM data for S3

Land raw first, exactly as returned, in its own prefix. Then convert to Parquet in a second step. Skipping the raw layer feels efficient right up to the first time you need to reprocess a month with corrected logic.

# Partition by the window's upper bound, not by a business date.
# One bad run is then one partition to replace.
s3://acme-lake/raw/zoho_crm/deals/window_end=YYYY-MM-DDTHH/
s3://acme-lake/curated/zoho_crm/deals/window_end=YYYY-MM-DDTHH/

Zoho returns nested structures that do not map onto a flat table. Lookup fields come back as objects with an ID and a display name. Multi-select fields and tags come back as arrays. Subforms come back as arrays of objects. You have three choices per field and should make each one deliberately:

• Flatten into columns. A lookup becomes account_id and account_name. Correct for anything you filter or join on.
• Split into a child table. Subforms and line items become their own Parquet dataset keyed by parent ID. Correct when the nested rows are things people count.
• Keep as a JSON string column. Fine for rarely-queried arrays, and Athena can parse it on demand. Not fine for anything in a regular report.

Pin your types explicitly during conversion rather than letting the writer infer them from a batch. Inference is the reason a column is a string in January’s files and a double in February’s, and Athena will happily read both and quietly fail to reconcile them.

Register the result in the Glue Data Catalog and query with Athena. If you expect to apply updates and deletes in place rather than rebuilding partitions, Apache Iceberg earns its extra setup: row-level operations and schema evolution are exactly what a CRM feed generates.

On the AWS side, the orchestration is unremarkable and should stay that way: EventBridge on a schedule, Lambda or a small container for the extract, Step Functions if you need to poll a Bulk Read job and branch on the result. Resist Glue jobs for the extraction itself unless you are already deep in Glue; a Python container you can run locally is easier to debug at seven in the morning.

API credits

Zoho meters API usage as credits, allocated by edition and user count, over a rolling 24-hour window, and shared with every other integration on the org. There are also concurrency limits, so hammering the API in parallel fails differently from exceeding your daily allowance.

• Bulk Read for volume. It sidesteps the standard limits and is the whole reason the API exists.
• Ask for fewer fields. Selecting only the columns you actually land reduces payload and processing on both sides.
• Mind the COQL limit value. Credits scale with the page size you request, so grabbing 2,000 rows costs more than grabbing 200. Larger pages are still usually the better trade, but know you are making it.
• Give the pipeline its own connected app and user. Then consumption is attributable when someone asks who exhausted the credits.
• Back off properly on 429. Exponential backoff with jitter, not a fixed sleep, and treat rate limiting as an expected condition rather than an error.

Troubleshooting

Two runs of the same window return different rows

Open-ended window, offset pagination, or both. Add the upper bound, switch to keyset pagination or Bulk Read, and the symptom disappears. This is the one to check before anything else, because it makes every other investigation unreliable.

Invalid token, but the credentials are definitely right

Data centre mismatch. Confirm which region the org lives in and that your accounts domain and API domain both match it. This is far more common than an actually bad token.

Row counts drift upward over time

Deletions are not being applied. Run the deleted-records endpoint for the last 60 days and see how much comes back, then run a full ID reconciliation to catch the rest.

Bulk Read job finished but the file is gone

The download URL expires after roughly a day. Land the file as soon as the job reports complete, rather than deferring it to a downstream step that might not run until tomorrow.

Athena fails with a schema mismatch across partitions

Type inference changed between runs, usually because a nullable field was all-null in one batch and populated in the next. Define the schema explicitly at write time and reprocess the affected partitions from raw.

A new custom field never appears

You are selecting fields explicitly, which is correct, and nobody told you a field was added. Poll the module’s field metadata on a schedule and alert on changes, so schema drift is a notification rather than a discovery six weeks later.

Frequent 429s

Either concurrency or the download rate limit, depending on which call is failing. Serialise the module extracts rather than fanning them all out at once, and add jittered backoff.

Common mistakes

• An open-ended Modified_Time filter with no upper bound.
• Offset pagination over a dataset that is being modified while you read it.
• Advancing the watermark when the API call succeeds rather than when the data lands.
• Keeping watermark state on an ephemeral runner.
• Hardcoding the Zoho API domain and discovering data centres the hard way.
• Never calling the deleted-records endpoint, so counts only ever grow.
• Deferring the Bulk Read download until after the URL has expired.
• Landing only Parquet with no raw layer, so reprocessing means re-extracting.
• Letting the Parquet writer infer types per batch.
• Flattening subforms into a wide table and double-counting parent rows.
• Fanning out every module in parallel and collecting rate limits.
• Sharing the pipeline’s connected app with other integrations, so nobody can attribute credit usage.
• Storing the refresh token in an environment variable and forgetting it never expires.

Best practices

• Closed read windows with a deliberate lag, always.
• Keyset pagination when you paginate at all, Bulk Read when you can avoid it.
• Watermarks in durable storage, advanced only on confirmed landing.
• Idempotent runs, so re-running a window is always safe.
• Deleted-records endpoint every cycle, full ID reconciliation monthly.
• Raw layer untouched, curated layer derived, partitions keyed by window end.
• Explicit schemas on write, explicit field lists on read.
• Its own connected app, narrow scopes, refresh token in Secrets Manager.
• Jittered exponential backoff and serialised module extracts.
• Alerting on schema drift and on row-count delta against Zoho.
• Encryption and a retention policy on CRM data in S3 from day one, since it is personal data.

FAQ

What is the best way to export Zoho CRM data to S3?

Bulk Read for anything of size, on a schedule, with closed time windows. COQL for moderate volumes where you want real filtering. The plain records endpoint only for small modules. Land raw output first, convert to Parquet second.

Why does my extract return different results each run?

Your query has no upper time bound, so the result set changes while you paginate through it. Bound both ends of the window and put the upper bound a few minutes in the past. That single change makes runs repeatable.

How do I capture deleted records?

Call the module’s deleted-records endpoint each cycle. Recycle-bin deletions stay retrievable for 60 days and permanent deletions for 120, so a daily or hourly sync has plenty of margin. Add a monthly full ID reconciliation as a backstop.

Will this exhaust my Zoho API credits?

Not if you use Bulk Read for the heavy lifting, since it does not draw on standard API limits the way per-record calls do. The credit pool is shared across the org, so give the pipeline its own connected app and monitor consumption rather than finding out when someone else’s integration breaks.

How do I handle subforms and multi-select fields?

Decide per field. Flatten lookups into ID and name columns, split subforms and line items into their own child datasets keyed by parent ID, and keep genuinely peripheral arrays as JSON strings. Flattening a subform into the parent row is how you end up double-counting deals.

Build it or buy a connector?

Buy it if Zoho CRM is one source among several and you have no unusual requirements; managed connectors handle pagination, deletes and schema drift, which is most of the work described here. Build it when you need control over the shape of the output, want to avoid per-row pricing, or already run the surrounding infrastructure.

How fresh can the data realistically be?

Hourly is comfortable and suits nearly all reporting. Every fifteen minutes is achievable with COQL on modest modules. Anything closer to real time means webhooks or notification subscriptions rather than polling, which is a considerably larger commitment for a benefit most dashboards do not use.

The one thing to remember

An extract that returns a different answer each time it runs is not a data quality problem, it is a read consistency problem, and no amount of downstream validation will fix it. Bound both ends of the window, put the upper bound in the past, paginate by key rather than offset, and advance the watermark only when the data is safely in S3.

Get that right and the rest of the pipeline becomes ordinary engineering: land it, convert it, catalogue it, query it. Get it wrong and you will spend months chasing numbers that move every time you look at them.

Want this built or reviewed?

Most CRM pipelines I get handed work fine on a quiet module and lose rows on the busy one, which is the hardest version to notice. Work I take on:

• Building a Zoho CRM to S3 pipeline end to end: OAuth, Bulk Read extraction, Parquet conversion, Glue catalog, Athena query layer.
• Auditing an existing pipeline for read consistency and telling you whether it is silently dropping records.
• Reworking incremental logic into closed windows, keyset pagination and durable watermarks.
• Deletion handling and reconciliation jobs so row counts stay equal to the CRM.
• Modelling nested CRM data into a schema that does not double-count.
• Orchestration and alerting on AWS: EventBridge, Lambda or containers, Step Functions, and monitoring that catches drift early.

Send me your extract query and how you paginate, and I will tell you whether it can lose rows.

The post Run It Twice, Get Two Answers: Building an ETL Pipeline From Zoho CRM to Amazon S3 appeared first on John Nessime.