A nightly sync from a partner’s GraphQL API had run for weeks without a single failed execution. Green across the board in the console. Then someone asked why one column had been empty since the previous Tuesday. Not sparse. Empty. Every row, every night, for six nights.

The API had started returning a field-level error on that one field. Status code 200. A populated data block. An errors array sitting right next to it. The fetcher parsed data, wrote the records to S3, exited zero, and the orchestrator logged a success. Nothing in the stack was wrong, exactly. Everything in the stack was checking the wrong thing.

This post covers how to build a GraphQL data ingestion pipeline on AWS that fails honestly. It is organised by failure family rather than by service, because the AWS wiring is the easy part. The hard part is that GraphQL breaks four assumptions that REST ingestion quietly relies on, and every one of them fails silently by default.

Why GraphQL ingestion breaks the habits REST taught you

Most ETL code carries three inherited reflexes from years of REST work: the status code tells you whether it worked, the response shape is the server’s business, and rate limits are counted in requests. GraphQL invalidates all three.

• The status code is not the contract. The GraphQL over HTTP specification says a server should return 200 even when field errors are raised during execution, because a partial response is still a successful execution.
• You own the response shape. The selection set is written by you, which means schema drift shows up as a query that stops being valid, not as a payload that changes underneath you.
• Cost is query-shaped, not endpoint-shaped. One endpoint can cost almost nothing or blow your entire budget depending on how deeply you nested it.

Each of those becomes a distinct failure family. Let’s take them in the order they bite.

Failure one: the 200 OK that quietly ate a column

This is the one that costs the most and shows up the latest. GraphQL splits errors into two categories, and they behave completely differently on the wire.

• Request errors happen before execution starts: syntax errors, validation failures, bad variable coercion. No data key is returned at all. Nothing executed.
• Field errors happen during execution: a resolver threw, a downstream service timed out, a permission check failed. Execution continues, the failed field goes null, and you get data and errors together.

Field errors are the dangerous ones for ingestion, because they look exactly like success to anything that only inspects the transport layer. Worse, GraphQL’s null propagation rules mean a single failing non-null field bubbles up to the nearest nullable ancestor, so one broken resolver can hollow out an entire nested object while the surrounding rows land normally.

The fix is to make the response body the contract. Classify before you commit anything:

import json, urllib.request

class RequestError(Exception): pass
class FieldError(Exception): pass

def graphql(endpoint, token, query, variables):
    body = json.dumps({"query": query, "variables": variables}).encode()
    req = urllib.request.Request(
        endpoint,
        data=body,
        headers={
            "Content-Type": "application/json",
            "Accept": "application/graphql-response+json, application/json",
            "Authorization": f"Bearer {token}",
        },
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        payload = json.loads(resp.read())

    # The status code told us nothing. The body is the contract.
    if "data" not in payload:
        # Nothing executed. Query or auth problem. Do not retry blindly.
        raise RequestError(payload.get("errors"))

    if payload.get("errors"):
        # Partial response. data is populated but incomplete.
        raise FieldError(payload["errors"])

    return payload["data"]

Two things worth explaining. The Accept header asks for application/graphql-response+json, the media type the GraphQL over HTTP spec recommends, which lets a compliant server return a real 4xx for pre-execution failures instead of burying them in a 200. Servers that don’t support it fall back to application/json, so including both is free.

The second is the split between the two exception types. They need different handling. A request error is a bug in your query or a credential problem, and retrying it just burns budget on the same failure. A field error is transient often enough to be worth retrying, but never worth committing.

Deciding what “partial” means for your data

Failing the whole page on any error is the safe default and the one I reach for first. But it’s not always right. If you are pulling a hundred fields and one optional enrichment field is flaky, refusing every page forever is its own outage.

The workable middle ground is to classify by path. Every error object carries a path array pointing at the exact field that failed. Maintain a small allowlist of paths that are permitted to be null, fail the page on anything outside it, and emit the error count as a CloudWatch metric so a rising trend is visible before someone in finance notices.

Failure two: rate limits that never send a 429

Your retry logic almost certainly triggers on 429 and 5xx. Several major GraphQL APIs will throttle you without ever sending one.

GitHub’s GraphQL API is explicit about this: exceeding the primary rate limit returns a 200 response with an error message in the body and x-ratelimit-remaining set to zero. Your Step Functions retrier sees a success. Your Lambda sees no data. The run completes, having ingested nothing, and reports green.

The other half of the problem is that request counting is the wrong unit. Shopify’s GraphQL Admin API and several others price queries by calculated cost, using a leaky bucket. The server statically analyses your query before executing it, charges the bucket the worst-case requested cost, executes, then refunds the difference between requested and actual cost. That has a consequence people miss: a query asking for the maximum connection size gets charged as if every page were full, even when the last page returns three records.

The good news is that these APIs tell you the state of the bucket on every response, under the extensions key:

{
  "extensions": {
    "cost": {
      "requestedQueryCost": 72,
      "actualQueryCost": 38,
      "throttleStatus": {
        "maximumAvailable": 1000.0,
        "currentlyAvailable": 962.0,
        "restoreRate": 100.0
      }
    }
  }
}

Read those numbers, never hardcode them. Bucket size and restore rate vary by plan tier, and a query that works fine against a large merchant’s store will fail instantly against a small one. The pattern that holds up is a pre-flight check: before dispatching, compare your estimated cost against currentlyAvailable, and if you’re short, sleep for the shortfall divided by restoreRate.

On APIs that expose their own budget as a queryable field, ask for it in the same round trip rather than spending a second call to find out:

query BudgetCheck {
  rateLimit {
    limit
    cost
    remaining
    resetAt
  }
}

Two more things about concurrency. First, if you fan out across Lambda invocations, each worker’s local view of the bucket is wrong, because the bucket is shared. Multiple workers independently deciding they have headroom is the classic way to get throttled while every individual worker believes it’s behaving. Either centralise the token accounting or cap concurrency low enough that the arithmetic can’t go wrong.

Second, if the source offers a bulk or async export operation, use it for backfills. Paginating fifty thousand records through the interactive endpoint burns the same budget your incremental syncs need, and it competes with itself.

Failure three: cursors that don’t survive a restart

Most production GraphQL schemas follow the Relay cursor connections convention: first and after arguments, a pageInfo object with hasNextPage and endCursor, and edges carrying opaque per-item cursors.

query Orders($cursor: String) {
  orders(first: 100, after: $cursor) {
    pageInfo {
      hasNextPage
      endCursor
    }
    edges {
      cursor
      node {
        id
        updatedAt
      }
    }
  }
}

The filter argument for incremental sync is not standardised. Some schemas take a query-string filter, some take a typed timestamp argument, some expose nothing at all. Run introspection against the schema and find out rather than guessing, because a wrong argument name is a request error and your run will fail before it starts.

The failure here is subtle. Cursors are opaque and their validity is entirely up to the server. Some are stable positional pointers. Some encode a sort key and a snapshot identifier and expire. If a page-47 fetch fails and you retry an hour later with the stored cursor, you may get an error, or you may get results from a shifted window with a hole in the middle. The second outcome is worse, because it looks like it worked.

Three habits make restarts safe:

1. Checkpoint after the write, not after the fetch. A cursor recorded before the page lands in S3 is a promise you can’t keep.
2. Store a high watermark alongside the cursor. The cursor is your fast path. The watermark, taken from the newest record you actually committed, is your fallback when the cursor is rejected.
3. Overlap the watermark deliberately. Restart a few minutes before the last committed timestamp and deduplicate on the primary key downstream. Re-reading a handful of rows is cheap. Missing rows that arrived during a clock skew is not.

A DynamoDB item per source stream is enough state for all of this:

{
  "source": "orders",
  "run_id": "01hq7k...",
  "cursor": "eyJsYXN0X2lkIjoxMjM0fQ==",
  "high_watermark": "<updatedAt of newest committed record>",
  "pages_committed": 47,
  "status": "IN_PROGRESS"
}

Overlap only works if downstream deduplication is real. Write raw pages with an ingest-time partition and a run identifier in the object key, then let the curated layer resolve duplicates by primary key and latest updatedAt. If you write straight into a merged table with no dedupe step, overlapping restarts will double-count and you’ll have traded silent loss for silent inflation.

Failure four: the schema moved and nobody told you

GraphQL’s deprecation story is genuinely better than REST’s. Fields carry a @deprecated directive with a reason string, and introspection exposes it. The problem is that nothing forces you to look.

The failure mode is a field that gets deprecated, stops being populated, then eventually gets removed. Stage one is invisible: your query still validates, the field returns null, and your warehouse fills with nulls. Stage two is loud but late: the query becomes invalid and the whole ingest breaks at 3am.

Put an introspection diff in CI. Fetch the schema on a schedule, store it as an artifact, and diff against the last known good version. Fail the build on removals or type changes to fields your queries actually select, and warn on new deprecations. Tools in the Apollo and Hasura ecosystems ship schema-diff tooling for this, and rolling your own against the introspection result is not much work either. Either way, the point is that the check runs on a schedule rather than when the pager goes off.

The complementary habit is to keep queries narrow. Every field you select is a field that can break you, a field that costs budget, and a field that widens your Glue table. Selecting a whole object because it was convenient during development is a liability you carry indefinitely.

Wiring the GraphQL data ingestion pipeline on AWS

With the failure families understood, the AWS assembly is mostly a set of trade-offs about where the fetch loop lives.

Where the fetch loop runs

Step Functions can call an HTTPS endpoint directly with an HTTP Task, no Lambda involved. It uses an EventBridge connection to hold the credentials, which means your token lives in a Secrets Manager secret managed by EventBridge rather than in an environment variable:

"FetchPage": {
  "Type": "Task",
  "Resource": "arn:aws:states:::http:invoke",
  "Parameters": {
    "ApiEndpoint": "https://api.example.com/graphql",
    "Method": "POST",
    "InvocationConfig": {
      "ConnectionArn": "arn:aws:events:region:account-id:connection/name/id"
    },
    "Headers": { "Content-Type": "application/json" },
    "RequestBody": {
      "query.$": "$.query",
      "variables.$": "$.variables"
    }
  },
  "Retry": [{
    "ErrorEquals": [
      "States.Http.StatusCode.429",
      "States.Http.StatusCode.502",
      "States.Http.StatusCode.503",
      "States.Http.StatusCode.504"
    ],
    "IntervalSeconds": 1,
    "BackoffRate": 2,
    "MaxAttempts": 3,
    "JitterStrategy": "FULL"
  }],
  "Next": "InspectBody"
}

The execution role needs four permissions: states:InvokeHTTPEndpoint, events:RetrieveConnectionCredentials, and both secretsmanager:GetSecretValue and secretsmanager:DescribeSecret scoped to the EventBridge-managed secret.

It’s a clean pattern, and I like it for small, high-value syncs. Two constraints decide whether it fits. HTTP Task requests time out after 60 seconds, which a deeply nested GraphQL query can exceed without being pathological. And Step Functions reads the response within its standard state payload ceiling of roughly 256 KB, so a page of a hundred rich records will not fit. Also note the retrier above: it fires on HTTP status codes, so it will not catch a throttle that arrives as a 200 with an error in the body. That check has to happen in the state after it.

The alternative is a Lambda driver that paginates internally and writes pages straight to S3, returning only the cursor and a count to the state machine. This dodges both limits, gives you a real HTTP client with real backoff, and keeps large payloads out of the workflow’s state. It costs you per-page retry granularity, which you buy back with the checkpoint record.

Distributed Map is worth mentioning for the fan-out case, such as one child execution per tenant or per shard. It gives you high concurrency and a per-item execution history, which is genuinely useful for debugging. Be aware of two things before committing: express child workflows cap at five minutes rather than Lambda’s fifteen, and practitioners have published benchmarks showing throughput well below what the headline concurrency numbers suggest for small, fast items. For a lot of ingestion work, SQS plus a DynamoDB job table remains the simpler answer.

One case genuinely falls outside serverless: sources that require a long-lived session, a persistent websocket subscription, or a synchronisation window longer than fifteen minutes with no resumable cursor. A small always-on worker on a modest VPS from a provider like InterServer or Hetzner, feeding the same S3 landing zone, is less clever and more honest than contorting Lambda around the constraint.

The landing zone

Land raw responses first, unmodified, including the errors and extensions keys. Storage is the cheapest thing in this architecture and the raw layer is what lets you reprocess after you discover a parsing bug six weeks later.

• Partition by ingest time, not event time. Event-time partitioning requires reading the record before you know where to put it, which means you can’t write until you’ve parsed. Ingest-time keys let the raw write be dumb and fast.
• Put the run identifier in the key. When you need to unwind one bad run, being able to delete or ignore by prefix beats writing a surgical delete.
• Define Glue tables explicitly rather than crawling. You wrote the selection set, so you already know the schema. A crawler inferring types from JSON will surprise you the first time an optional field is absent from an entire page.
• Convert to columnar in the curated layer. Athena over raw JSON works and is fine for debugging. Parquet is what you want for anything anyone queries repeatedly.

If your source pushes rather than being polled, such as webhook deliveries carrying GraphQL payloads, Amazon Data Firehose (renamed from Kinesis Data Firehose, with APIs and IAM policies unchanged) will buffer, convert to Parquet and partition on the way to S3 without you running a consumer. For scheduled pull-based syncs it adds a hop you don’t need.

Troubleshooting

• Runs succeed but a column is all null. Field errors. Pull a raw response from the landing zone and look at the errors array. The path tells you exactly which field.
• Runs succeed but ingest zero rows. Likely a throttle returned as a 200. Log the response body and the rate-limit headers on empty results, always.
• Intermittent failures partway through pagination. Cursor expiry. Fall back to the watermark and check whether the source documents a cursor lifetime.
• Query fails immediately with no data key. Request error. Validate against current introspection; something in the schema moved.
• Throttled despite staying under the documented limit. Either concurrent workers are sharing a bucket they each think they own, or you’re being charged requested cost on near-empty pages.
• HTTP Task fails on large pages. The response exceeds the state payload ceiling. Reduce page size or move the fetch into Lambda.
• Row counts drift upward over time. Overlap without deduplication. Check that the curated layer actually resolves on the primary key.

Common mistakes

• Treating a 200 as success and never reading the body.
• Retrying request errors. They are deterministic; retrying just spends budget to fail identically.
• Hardcoding bucket sizes and restore rates that differ per plan tier and change without notice.
• Checkpointing the cursor before the data is durable.
• Requesting the maximum page size on every call while being charged the worst-case cost for it.
• Selecting entire objects because it was easier during development.
• Alerting only on execution failure, so a job that succeeds while ingesting nothing never pages anyone.
• Overlapping restart windows with no downstream deduplication.

Best practices

• Classify every response into clean, partial, or request-failed before anything downstream sees it.
• Persist raw responses with errors and extensions intact, then parse from the raw layer.
• Alarm on row-count deltas and partial-error rate, not just on execution status.
• Read the throttle state the API hands you and back off before you hit the wall, not after.
• Keep both a cursor and a watermark, and know which one is authoritative on restart.
• Diff the schema on a schedule so deprecations reach you before they reach your data.
• Hold credentials in Secrets Manager or an EventBridge connection, never in a Lambda environment variable.
• Trace the pipeline end to end. CloudWatch covers the basics; Grafana or Datadog earn their keep once you’re correlating child executions across a fan-out.

Frequently asked questions

Can Step Functions call a GraphQL API without a Lambda function?

Yes. An HTTP Task using the arn:aws:states:::http:invoke resource posts your query directly, with an EventBridge connection handling authentication. It works well for small pages. The 60-second request timeout and the state payload size ceiling are what push most real ingestion work back into Lambda.

How do I detect GraphQL errors when the status code is always 200?

Inspect the body. If there’s no data key, nothing executed and you have a request error. If data is present alongside a non-empty errors array, you have a partial response and the data is incomplete. Only a response with data and no errors is unambiguously clean.

Does AWS AppSync fit into this?

Only if you’re on the serving side. AppSync is a managed GraphQL API for exposing your own data, so it’s what you’d reach for if you wanted clients to push events in via mutations. For pulling data out of somebody else’s GraphQL API, AppSync isn’t part of the picture.

Should I store raw GraphQL responses or parsed records in S3?

Both, in separate layers. Raw responses are your audit trail and your reprocessing path, and they’re the only way to answer “was this field null in the source, or did my parser drop it?” weeks after the fact. Parsed and columnar output goes in a curated prefix that analysts query.

How do I run an incremental sync from a GraphQL API?

Find the schema’s filter argument for a modification timestamp using introspection, pass your stored watermark into it, and page forward with cursors. Overlap the window slightly on each run and deduplicate on the primary key downstream. If the schema exposes no such filter, you’re limited to full pulls or webhook-driven change capture.

Why am I throttled when I’m making very few requests?

Cost-based APIs charge for query complexity, not call count. A single deeply nested query pulling a connection inside a connection can consume a large fraction of the bucket by itself. Read the cost figures the API returns and flatten the query if the requested cost is consistently much higher than the actual cost.

Is GraphQL a worse ingestion source than REST?

Different, not worse. You control the payload, which cuts transfer and lets you avoid the N+1 fan-out REST forces on nested data. Deprecation metadata is machine-readable, which REST rarely offers. The cost is that error signalling and rate limiting both moved into the body, so naive clients fail silently instead of loudly.

The one thing worth remembering

A GraphQL data ingestion pipeline on AWS fails differently from a REST one, and the difference is almost entirely about where the truth lives. In REST, the transport tells you what happened. In GraphQL, the transport tells you the request arrived, and the body tells you what happened. Every failure family in this post traces back to that: partial errors hidden behind a 200, throttles hidden behind a 200, cursors that fail by returning the wrong rows rather than an error.

So build the pipeline so that a green run means something. Parse before you commit, alarm on row deltas rather than exit codes, and keep the raw responses so you can prove what the source actually said. Everything else is ordinary AWS plumbing.

Need help with a GraphQL ingestion pipeline?

This is the kind of work I take on regularly. Specifically:

• Auditing an existing GraphQL sync for silent partial-error loss and telling you which columns are already affected
• Building the fetch, checkpoint and land loop in Step Functions and Lambda with cursors that survive restarts
• Fitting a cost-aware throttle to whatever budget model your source actually uses, including multi-worker coordination
• Designing the raw and curated S3 layout, Glue table definitions and Athena partitioning so queries stay cheap
• Adding introspection diffing to CI so schema deprecations surface in a pull request, not in a dashboard
• Wiring CloudWatch metrics and alarms on row-count deltas and partial-error rates, so a green run is actually green

Send me a run log, a state machine definition, or a raw response with an errors array in it, and I’ll tell you what it’s costing you.

The post Building a GraphQL Data Ingestion Pipeline on AWS That Doesn’t Lie to You appeared first on John Nessime.

That divergence is not a bug. It is documented behaviour, sitting in one table in the AWS docs that almost everyone scrolls past. This post covers what Zendesk data integration with AWS Glue zero-ETL actually replicates, the entities where deletes silently never propagate, the configuration choices you cannot undo after creation, how to wire the three IAM roles involved, and how to monitor the thing so drift shows up as an alarm instead of a meeting.

What Zendesk AWS Glue zero-ETL actually gives you

Zero-ETL is AWS’s name for managed replication. You create a Glue connection to Zendesk, pick the entities you want, pick a target, and Glue handles the initial snapshot, the schema mapping, and ongoing change data capture. No Spark job. No bookmark logic. No pagination code hammering the Zendesk API and no retry handling for it.

Supported targets are a general purpose Amazon S3 bucket through the lakehouse architecture of Amazon SageMaker, S3 Tables through the same lakehouse, Redshift Managed Storage, or an Amazon Redshift data warehouse directly. Data lands as Apache Iceberg, which is what makes row-level updates and deletes viable on object storage at all.

What you give up is control. No transform step, no server-side filter, no column pruning at ingest. You get the entity as the connector sees it and you shape it downstream. For most support analytics that trade is fine. It stops being fine the moment your compliance team asks why deleted records are still queryable.

The delete gap: four of seven entities never remove a row

AWS Glue zero-ETL supports seven Zendesk entities. Only three of them replicate deletes.

• tickets – create, update and delete all propagate
• users – create, update and delete all propagate
• organizations – create, update and delete all propagate
• satisfaction-rating – create and update only, no delete
• articles – create and update only, no delete
• calls – create and update only, no delete
• legs (call legs) – create and update only, no delete

When a satisfaction rating disappears in Zendesk, the row stays in your lakehouse. Permanently. Nothing errors, IngestionSucceeded fires as usual, and the row count in that table only ever goes up.

Now think about what your CSAT query does. It averages a score column across a set of rows. If removed ratings never leave the target, your average drifts further from the source every cycle, and the skew is rarely random because retracted or disputed ratings are exactly the ones most likely to be removed. The same mechanism inflates knowledge base article counts, call volume, and any call leg analysis you build. It stays invisible until two dashboards get compared side by side.

How to design around it

You cannot make the connector track deletes it does not track. So stop trying, and split the problem instead.

1. Run two integrations, not one. Put tickets, users and organizations in a continuously synced integration. Put the four append-only entities in a second integration you can tear down and rebuild on a cadence that matches how much drift you can tolerate.
2. Treat the append-only tables as an event log, not as state. Query them through a view that reconciles against a periodic authoritative count, rather than trusting row presence to mean the record still exists.
3. Prefer status fields over row existence. Where an entity exposes a field describing its state, filter on that field downstream. Check which fields your entities actually return before you build on this, because coverage varies by entity.
4. Reconcile on a schedule. Pull counts from the Zendesk API for the four entities and compare against the target. Alert on divergence, not on an absolute number.

Splitting the integration costs you a second set of IAM wiring and a second thing to monitor. It buys you the ability to rebuild the drifting half without touching the half that is correct. That is a good trade.

Entity coverage is narrower than the Zendesk API

Seven entities is not the Zendesk API. Ticket comments, ticket metrics, ticket audits, groups, views, macros and SLA policy definitions are not part of the zero-ETL entity set. If your reporting needs first response time, the audit trail behind a status change, or custom field definitions, zero-ETL alone will not get you there.

The useful part is that the Glue connection you create is reusable. The same connection backs a normal Glue ETL job, so you can run zero-ETL for the bulk entities and a scheduled Spark job for the ones it does not cover:

# Read a Zendesk entity through the same connection
# used by the zero-ETL integration. ENTITY_NAME takes
# the entity name, not the label, e.g. "tickets".
zendesk_read = glueContext.create_dynamic_frame.from_options(
    connection_type="Zendesk",
    connection_options={
        "connectionName": "my-zendesk-connection",
        "ENTITY_NAME": "tickets",
        "API_VERSION": "v2"
    }
)

One schema detail worth knowing before you write a single downstream query: the connector converts struct and list types to strings. Zendesk custom fields arrive as arrays of objects, so they land as serialized text, not as a nested column you can address with dot notation. Partitioning is also not supported on the Zendesk source. Plan for a parse step in your silver layer rather than discovering it in a broken dashboard.

Two decisions you cannot change after creation

Most Glue settings are editable. These two are not, and getting them wrong means deleting the integration and starting over.

Continuous sync versus on-demand snapshot

By default the integration syncs continuously. Enable the on-demand snapshot setting and it does a single one-time replication with no ongoing CDC. That setting is locked once the integration exists. If you enabled it for a proof of concept and then decided you wanted CDC, you are recreating the integration and re-running the full load.

Refresh interval, and why your target choice locks it

The refresh interval controls how often CDC pulls run. It accepts anything from 15 minutes up to 8640 minutes, which is six days. Here is the part that catches people:

• If the target is Amazon Redshift, the refresh interval cannot be modified after creation.
• For other targets, including the SageMaker lakehouse on S3, you can change it later.

So the target you pick is not only an architecture decision, it is a flexibility decision. If you are not certain what freshness the business actually needs, land in the lakehouse first and query Redshift over it. You keep the ability to tune the interval once you have real usage data instead of a guess made in week one.

On cost: AWS does not charge separately for the integration itself. You pay for everything around it, target storage, Redshift or Athena query compute, S3 requests, Data Catalog usage and CloudWatch logs. A 15 minute interval on seven entities produces far more small-file churn and log volume than a 6 hour one. Set the interval from the freshness a human will actually act on, not from the smallest number the field accepts.

Wiring the three roles

There are three separate permission layers here and they fail in different ways, which is why “check IAM” is useless advice on its own.

• The Zendesk OAuth credential. The connector uses the authorization code grant, so you get redirected to Zendesk to log in and approve. You can rely on the Glue-managed client application and supply only your instance URL, or register your own OAuth app in the Zendesk admin center and provide your own client ID and secret. The resulting access token does not expire, which is convenient and also means a revoked app in Zendesk is something you will only discover through failed ingestions.
• The source role. This is what lets the integration read through the connection. It is a prerequisite for SaaS sources and it is the step most people miss, because creating the connection successfully does not mean the integration can use it.
• The target role and catalog policy. This is what lets the integration write. A misconfigured catalog resource policy puts the integration into NEEDS_ATTENTION, not into a clear error at creation time.

The source role policy looks like this. Note glue:RefreshOAuth2Tokens, which is the one people leave out and then spend an afternoon debugging:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "GlueConnections",
            "Effect": "Allow",
            "Action": [
                "glue:GetConnections",
                "glue:GetConnection"
            ],
            "Resource": [
                "arn:aws:glue:*:111122223333:catalog",
                "arn:aws:glue:us-east-1:111122223333:connection/*"
            ]
        },
        {
            "Sid": "GlueActionBasedPermissions",
            "Effect": "Allow",
            "Action": [
                "glue:ListEntities",
                "glue:RefreshOAuth2Tokens"
            ],
            "Resource": ["*"]
        },
        {
            "Sid": "CloudWatchLogging",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": ["*"]
        }
    ]
}

The trust policy allows glue.amazonaws.com to assume it. Then you attach the role to the connection, which is a CLI-only step and does not appear in the console flow:

aws glue create-integration-resource-property 
  --resource-arn arn:aws:glue:us-east-1:123456789012:connection/my-zendesk-connection 
  --source-processing-properties "{"RoleArn" : "arn:aws:iam::123456789012:role/zendesk-zetl-source"}" 
  --region us-east-1

If you skip this, the connection tests fine and the integration still cannot read. That mismatch is the single most common reason a first attempt sits in NEEDS_ATTENTION with no obvious cause.

Setting up the integration

1. If you are using your own OAuth app, register it in the Zendesk admin center under the API settings and note the client ID and secret. Store the secret in AWS Secrets Manager, one secret per Glue connection.
2. In Glue Studio, create a connection under Data Connections. Choose Zendesk as the connection type, supply your instance URL and environment, and complete the OAuth redirect.
3. Create the source role with the policy above and attach it to the connection using create-integration-resource-property.
4. Prepare the target: the Glue database or S3 Table bucket, the target role, the catalog resource policy, and any Lake Formation grants your account’s permission model requires.
5. Create the integration. Select entities, set the refresh interval, and decide continuous sync versus on-demand snapshot. Both of those are locked from here.
6. Watch the first full load complete before you point anything at the tables. Full load and CDC are separate load types in the metrics, and the numbers look very different.

Once it works, move it into code. Glue zero-ETL integrations are supported by CloudFormation and the AWS CDK, which is the difference between a thing one person built in the console and a thing your team can redeploy into another account.

Monitoring: the metrics that expose drift

Zero-ETL publishes to the AWS/Glue/ZeroETL CloudWatch namespace, with dimensions for integrationArn, loadType and tableName. The metrics are InsertCount, UpdateCount, DeleteCount, IngestionSucceeded, IngestionFailed and LastSyncTimestamp.

The obvious alarm is on failure:

aws cloudwatch put-metric-alarm 
  --alarm-name zendesk-zetl-ingestion-failed 
  --namespace AWS/Glue/ZeroETL 
  --metric-name IngestionFailed 
  --dimensions Name=integrationArn,Value=<your-integration-arn> 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 1 
  --threshold 1 
  --comparison-operator GreaterThanOrEqualToThreshold 
  --treat-missing-data notBreaching

That one catches loud failures. It does not catch the quiet one, which is an integration that stops running entirely. For that, alarm on IngestionSucceeded with --treat-missing-data breaching over a window longer than your refresh interval. Absence of success is the signal, not presence of failure.

And here is the one specific to this post. Because DeleteCount carries a tableName dimension, you can see the delete gap directly in a graph. Plot DeleteCount per table. Tickets, users and organizations will show a nonzero line. Satisfaction ratings, articles, calls and legs will sit flat at zero forever. That flat line is not a broken metric, it is the behaviour, and having it on a dashboard is the cheapest way to keep the whole team aware of it.

Glue also writes a system table into the target database recording the outcome of each full load and CDC run, with per-run record, insert and delete counts. Query that when CloudWatch is not granular enough. If you already run Grafana Cloud or Datadog, pulling the AWS/Glue/ZeroETL namespace in alongside your existing infrastructure dashboards puts pipeline health next to everything else on call already watches.

Troubleshooting

Integrations move through Creating, Active, Modifying, Syncing, Needs attention, Failed and Deleting. What each one means in practice:

• Syncing means it hit a recoverable error and is re-seeding data. Not an emergency. Let it finish before you touch anything.
• Needs attention means you have to fix something, usually connection credentials, the source role, the target role, or the catalog resource policy. Once you fix it, there is no manual recovery action. Glue retries automatically on an exponential backoff schedule, so the gap between attempts grows over time. If you fixed a policy and nothing happened after two minutes, that is expected. Wait it out.
• Failed is terminal. Delete the integration and recreate it. There is no repair path.

For anything else, the CloudWatch logs emitted after each full load or CDC run carry the actual root cause. The console status is a summary, and a fairly lossy one. Read the logs first.

Common mistakes

• Building CSAT, article count or call volume metrics on raw row counts from the four entities that never delete.
• Creating the connection and assuming the integration can use it. The source role attachment is a separate CLI step.
• Choosing Redshift as the direct target during a proof of concept and locking a refresh interval you picked arbitrarily.
• Setting a 15 minute interval because it was available, then being surprised by small-file churn and CloudWatch log volume.
• Writing downstream queries that address Zendesk custom fields as nested columns. They arrive as strings.
• Alarming only on IngestionFailed, so a stalled integration produces silence and silence looks like health.
• Reaching for zero-ETL when the requirement is ticket comments or ticket metrics, which are not in the entity set at all.

Best practices

• Split delete-tracking and append-only entities into separate integrations so you can rebuild one without disturbing the other.
• Land in the SageMaker lakehouse rather than straight into Redshift unless you are certain about freshness. It keeps the refresh interval editable.
• Put DeleteCount per table on a dashboard. It makes an invisible behaviour visible to everyone, not just whoever read the docs.
• Define the integration in CloudFormation or CDK once it works, so the next environment is a deploy rather than a repeat of the console clicking.
• Keep a silver layer between the replicated tables and your BI tool. That is where you parse the stringified structs and apply the reconciliation logic, and it means schema surprises break one view instead of every dashboard.
• Document which entities do not track deletes somewhere your analysts will actually read, next to the tables, not in a wiki page nobody opens.

When to use something else

Zero-ETL is a good fit when your target is already AWS, your entities are in the supported set, and you want AWS to own the pipeline. It is the wrong fit if you need entities outside that list, need transformation at ingest, or need the same Zendesk data landing somewhere that is not AWS. Managed connector platforms like Fivetran or Airbyte cover a wider slice of the Zendesk API and will replicate to non-AWS destinations, at the cost of another vendor relationship, another bill, and data leaving your account boundary on the way through. If the seven entities cover your reporting, zero-ETL is cheaper and simpler. If they do not, forcing it is worse than paying for a tool that fits.

Frequently asked questions

Which Zendesk entities does AWS Glue zero-ETL support?

Tickets, users, organizations, satisfaction ratings, articles, calls and call legs. Tickets, users and organizations replicate creates, updates and deletes. The other four replicate creates and updates only.

Can I get ticket comments or ticket metrics through zero-ETL?

No. They are not in the entity set. Use a Glue ETL job against the same Zendesk connection, or a connector platform with broader API coverage, and join the result to your replicated tables downstream.

How fresh can the data be?

The refresh interval goes down to 15 minutes and up to six days. This is near real time replication, not streaming. If you need sub-minute latency on Zendesk events, zero-ETL is the wrong mechanism and you want webhooks into an event pipeline instead.

Can I change the refresh interval later?

Only if the target is not Redshift. With Redshift as the target the interval is fixed at creation. With the SageMaker lakehouse and other targets you can modify it afterwards.

Why is my integration stuck in NEEDS_ATTENTION after I fixed the permissions?

Because recovery is automatic but not immediate. Glue retries on exponential backoff, so if the integration has been unhealthy for a while the next retry may be some way off. There is no manual recovery command. Confirm the fix is correct, then wait.

Does zero-ETL cost extra on top of Glue?

AWS does not bill a separate charge for the integration itself. You pay for the surrounding services: target storage, Redshift or Athena query compute, S3 requests, Data Catalog usage and CloudWatch logs. Refresh interval is the main lever, since it drives how many small commits and log entries you generate.

Why do my Zendesk custom fields look like text?

The connector converts struct and list types to strings. Custom fields come through as serialized values rather than nested columns, so parse them in a downstream layer instead of querying them directly.

Conclusion

Zendesk AWS Glue zero-ETL removes a genuinely tedious pipeline from your plate. The setup is fiddly in the IAM layer and then it mostly runs itself, which is exactly the point of a managed integration.

The one thing worth carrying out of this post: four of the seven supported entities never replicate a delete. Satisfaction ratings, articles, calls and call legs only grow. Nothing warns you, every metric stays green, and your numbers drift a little further from Zendesk with every cycle. Design for that on day one, put DeleteCount per table on a dashboard, and you will not be the person explaining the discrepancy in a meeting six months from now.

Need help with your Zendesk to AWS data pipeline?

I work with teams building and fixing replication pipelines on AWS. On this kind of setup, that usually means:

• Auditing an existing Zendesk zero-ETL integration and quantifying how far the target has drifted from the source
• Designing the split between continuously synced and append-only entities, including the rebuild cadence and reconciliation queries
• Untangling the source role, target role and catalog policy layers when an integration sits in NEEDS_ATTENTION with no clear cause
• Building the silver layer that parses stringified Zendesk custom fields into something your BI tool can actually use
• Setting up CloudWatch alarms and dashboards that catch a stalled integration, not just a failed one
• Moving a console-built integration into CloudFormation or CDK so it can be deployed across accounts

Send me the integration status, a CloudWatch log excerpt, or the query that is giving you the wrong number, and I will tell you what I think is going on.

The post Zendesk Data Integration with AWS Glue Zero-ETL: The Delete Gap That Skews Your Numbers appeared first on John Nessime.