That is the real cost of picking the wrong data connection mode, and it is why the choice between Zoho Analytics import vs Live Connect deserves more than five minutes during setup. Both modes work. Both are documented. But they fail in completely different ways, and one of those failure modes is silent.

This post covers what each mode actually does under the hood, where each one genuinely wins, the four levers that should drive the decision, a short procedure for running that decision yourself, and the arguments people repeat that don’t hold up once you have a workspace in production.

What the two modes actually do

Strip away the marketing language and the difference is where the data lives when someone opens a report.

Data Import copies rows out of your source and stores them inside Zoho Analytics. Reports query that stored copy. You schedule refreshes to pull new data on an interval, and between refreshes your dashboard is a photograph, not a window.

Live Connect does not copy anything. Zoho pulls metadata about your tables and columns, then generates SQL against your source database every single time a report or dashboard is opened. Your database does the work. Zoho does the rendering.

That one architectural difference cascades into everything else: what you get billed for, which features you can use, who owns schema changes, and what happens when a query gets slow.

Import mode: where it wins and where it hurts

Where import mode wins

Import mode is the default for a reason, and the reasons are good ones.

• Report speed is yours to control. Because the data sits in Zoho’s storage, load time doesn’t depend on how busy your production database is at 9am on a Monday.
• You get the full feature surface. Query Tables, custom import queries, data type changes on imported columns, and blending several unrelated sources into one workspace all work in import mode.
• Your production database is insulated. An analyst who builds an accidental many-to-many join hits a copy, not the system taking customer orders.
• Incremental fetch keeps sync cheap. Rather than re-pulling a whole table on every schedule, you nominate a column that identifies new rows and pull only what changed.
• It works on every plan. There is no tier gate on importing data, which matters if you are prototyping before anyone has approved a budget.

Where import mode hurts

The first problem is the one from the opening: staleness is invisible. A Zoho Analytics dashboard built on imported data looks exactly the same whether it refreshed two minutes ago or two days ago. There is no built-in “as of” banner on the chart. If your refresh schedule is daily and someone is making a call at 4pm, they are working from this morning’s world.

The second is billing shape. Zoho Analytics prices on stored rows, counted as the sum of all rows across every table in every workspace in the account. Import mode is therefore the mode that grows your bill. Every table you sync “just in case” is a line item. Rates change, so check the current pricing page rather than trusting any figure you read in a blog post, but understand the mechanism: rows stored equals money.

Third, the copy drifts and nobody owns it. Six months in, a table has a scheduled import that has been silently failing, or someone changed a transformation in the source and the imported copy still reflects the old logic. Import mode creates a second version of the truth, and second versions need maintenance.

Fourth, refresh frequency is plan-gated. How often you can schedule a sync depends on your subscription tier. If your reporting requirement is “within the hour” and your plan permits fewer refreshes per day than that implies, import mode cannot meet the requirement no matter how you configure it.

Live Connect: where it wins and where it hurts

Where Live Connect wins

• There is exactly one copy of the data. No reconciliation conversation, because there is nothing to reconcile. What the report shows is what the database holds at that moment.
• Sensitive data stays where it already is. For teams with a data residency position, or a security team that objects to bulk-copying customer records into a BI vendor’s storage, this is often the deciding argument on its own.
• Foreign keys carry over. Tables linked by foreign keys in the source get lookup relationships created automatically in Zoho, which is a genuine time saver compared to wiring lookup columns by hand.
• Row storage isn’t the cost driver. You’re not paying to warehouse a duplicate of a table you already pay to host on Amazon RDS, Snowflake, or a Postgres instance on your own VPS.

Where Live Connect hurts

Here is the one that catches people, and it is worth reading twice: Live Connect does not automatically track schema changes. Add a column, rename a column, drop a table, change a data type in the source, and Zoho will not follow along. Reports built on the affected objects land in a Mismatch state and you have to reconcile them manually through the Sync Design option in the data source settings.

This is the inverse of what most people assume. “Live” sounds like “always current in every respect.” It means live data, not live structure. A routine migration that renames customer_id to account_id will break dashboards that nobody looked at that week, and you’ll find out from a user rather than from a monitor.

The rest of the trade-offs:

• Query timeout. Zoho documents a default query timeout of 2.5 minutes on Live Connect. Anything slower fails. On a source that hasn’t been tuned for analytical queries, that ceiling arrives faster than you expect.
• No Custom Query at import time. You can’t hand Zoho a filtered SELECT to narrow what it sees. The documented workaround is to create a view in the source database and connect to that instead.
• No Query Tables. The SQL-over-your-workspace feature depends on data being stored locally. Zoho notes Query Table support for selected databases is enabled on request, so it is worth asking, but do not plan around it.
• No data type changes in Zoho. If a column is text in the source and should be a date, you fix it in the source or in a view. There is no in-Zoho override.
• Lookups only within the connection. You can define lookup rules between tables in the same live connection, not across to other data sources. Reports from different sources can still be combined on a dashboard, but that is a presentation-layer join, not a data-layer one.
• It is gated to higher plans. Zoho’s documentation places Live Connect on the Premium and Enterprise tiers. If you’re on a lower plan, the decision has already been made for you.

The four levers behind Zoho Analytics import vs Live Connect

Feature checklists make this look like a close call. It usually isn’t, once you work out which of these four you actually care about.

1. Your freshness contract. Not “how fresh would be nice.” What decision gets made from this dashboard, and how wrong can the data be before that decision goes bad? A weekly pipeline review tolerates a nightly sync comfortably. A dispatcher deciding which order to fulfil next does not. Write the tolerance down as a number of minutes before you look at any feature table.

2. Where the cost lands. Import mode moves cost onto your Zoho row count. Live Connect moves it onto your database, in query load and in whatever you pay for the compute that serves it. Neither is free. A warehouse that bills per query scanned is a different economic proposition under Live Connect than a fixed-price MySQL instance on Contabo or InterServer, where extra query load costs nothing until you outgrow the box.

3. Where transformation happens. If your model needs joins, unions, derived columns, and cleanup, import mode gives you Query Tables to do that inside Zoho. Live Connect pushes all of it down into the source, which means you need database write access and someone comfortable maintaining views. That’s a team-capability question as much as a technical one.

4. Who owns schema change. Under Live Connect, every source migration is potentially a reporting incident. That is fine if the same team owns both, and painful if the application team ships migrations without telling anyone. Import mode is more forgiving here, since Zoho can automatically pick up column additions and deletions on synced tables.

A decision procedure you can run in ten minutes

1. Check the plan gate first. If you’re not on a tier that includes Live Connect, stop. Import mode, tuned well, is the answer.
2. Write down the freshness tolerance in minutes. If the number is larger than your plan’s shortest available sync interval, import mode satisfies it. If it’s smaller, Live Connect is the only mode that can meet it.
3. Time the heaviest query. Take the ugliest aggregation your dashboard will need and run it against the source with realistic filters. If it takes anywhere near 2.5 minutes, Live Connect will fail under real usage. Tune the source or choose import.
4. List the transformations you need. If any of them require a Query Table, joining across unrelated sources, or changing a column’s data type in Zoho, import mode is doing that work.
5. Ask who ships schema migrations. If it isn’t the people who own the dashboards, budget for Sync Design work every release, or use import mode.
6. Check the security position last, because it can override everything. If the answer to “may customer records be stored in a third-party BI platform” is no, Live Connect is the only option and you build around its constraints.

In practice, mixed estates are common and reasonable. Live Connect for the operational workspace where freshness is the point, import mode for the historical and blended reporting where transformation and speed matter more.

Arguments that don’t survive contact

“Live Connect is real-time, so it’s more accurate.” It’s more current. Accuracy is a modelling problem, not a transport problem. A live connection to a table with duplicate lookup keys produces wrong numbers instantly instead of wrong numbers on a schedule.

“Live Connect is cheaper because you don’t pay for rows.” Sometimes. You’ve moved the cost, not removed it. Every dashboard open is a set of queries against your source. On per-query or per-compute-second billing, an enthusiastic team with auto-refreshing dashboards can generate a bill that makes row storage look like a bargain.

“Just enable the cache and Live Connect gets fast.” Zoho does offer report caching on Live Connect workspaces, with a refresh interval you set in minutes. It genuinely helps. But note what you’ve just built: a copy of the data with a staleness window, which is the thing you chose Live Connect to avoid. Caching is a legitimate tuning tool, not a free upgrade.

“We’ll switch modes later if it doesn’t work out.” Switching is not a toggle. Zoho does not permit importing data into an existing Live Connect workspace, and the two modes support different feature sets, so a migration means rebuilding the workspace and its reports. Choose deliberately the first time.

If you go Live Connect, tune the source first

Live Connect performance is your database’s performance. Zoho’s own guidance is blunt about this, and it matches what you’d do for any reporting workload.

Index the columns that lookups and joins run through. Without them, every dashboard open is a table scan:

-- Index the join key that Zoho will generate queries against
CREATE INDEX idx_orders_customer_id ON orders (customer_id);

-- Confirm the planner actually uses it (PostgreSQL)
EXPLAIN ANALYZE
SELECT customer_id, SUM(total)
FROM orders
WHERE order_date >= CURRENT_DATE - INTERVAL '90 days'
GROUP BY customer_id;

Because Custom Query isn’t available in Live Connect, a view is how you narrow and shape what Zoho sees. Connect to the view rather than the raw table:

CREATE VIEW reporting_orders AS
SELECT
    o.id,
    o.customer_id,
    o.order_date,
    o.total,
    CASE WHEN o.status = 'complete' THEN 1 ELSE 0 END AS is_complete
FROM orders o
WHERE o.order_date >= CURRENT_DATE - INTERVAL '2 years';

For calculations that get repeated across many reports, a materialized view precomputes the work so the 2.5 minute ceiling stops being a threat. You trade a refresh schedule for query speed, which is a much smaller staleness window than a full import.

The rest of the list is short and worth following:

• Keep formula columns simple, and prefer CASE WHEN over deeply nested IF statements.
• Make sure lookup columns hold unique values. Duplicate keys degrade query performance badly and quietly inflate aggregates.
• Avoid many-to-many joins and circular join paths in your model.
• Give Zoho a database user with SELECT plus metadata read on exactly the tables it needs, and nothing more. Missing metadata permission is a common cause of a table that connects but never loads.
• Decide your connectivity path deliberately: allowlisting Zoho’s published IP ranges for a cloud database, or Zoho Databridge as an agent when the database sits in a private network. SSL and SSH tunneling are both supported on the connection.
• Put a change-notification step in your migration checklist. If a release renames or drops a column, someone should run Sync Design before users find the Mismatch.

Frequently asked questions

Can I switch a workspace from import mode to Live Connect?

Not as an in-place setting change. Zoho does not permit importing data into an existing Live Connect workspace, and the feature sets differ enough that reports built one way won’t simply carry over. Treat a mode change as a rebuild and plan the reporting downtime accordingly.

Does Live Connect count against my row limit?

Zoho’s row-based pricing counts rows stored across your workspace tables, and Live Connect doesn’t store your source data in Zoho. The practical implication is that Live Connect shifts your constraint from storage to source query capacity. Confirm the exact treatment for your account with Zoho before you build a cost model on it, since add-ons and plan definitions change.

Why do my Live Connect reports suddenly show a view-cannot-be-accessed error?

Almost always because a table or column the report depends on was renamed or dropped in the source. Live Connect doesn’t track structural changes automatically. Open the data source settings, check the Mismatch tab, and use Sync Design to remap. Insufficient SELECT or metadata permissions on the source produce a similar symptom.

What is incremental fetch, and does it apply to both modes?

Incremental fetch is an import-mode feature. Instead of re-pulling every row on each schedule, you nominate a column that identifies new records, such as an auto-incrementing ID or a timestamp, and choose whether new rows are appended or matched and updated. It cuts sync time dramatically on large transactional tables. It has no meaning under Live Connect, where nothing is fetched on a schedule in the first place.

Can I combine Live Connect tables with data from other sources?

Not at the data layer. Lookup rules only work between tables within the same live connection. You can place reports from a Live Connect workspace and reports from other sources side by side on a dashboard, but you cannot join them into a single dataset the way you can with imported tables in one workspace.

Is Live Connect available for on-premise and private databases?

Yes, through Zoho Databridge, the agent that bridges a database in a private network to Zoho’s servers. You choose the connection type during setup, so the same agent supports either mode. If your database is publicly reachable instead, you allowlist Zoho’s published IP ranges and connect directly.

Which mode is better for a Zoho CRM or Zoho Desk workspace?

Neither, in the sense that the question doesn’t apply. Zoho’s own application connectors sync data into Analytics on a schedule, which behaves like import mode. Live Connect is for external databases and data warehouses. If you need faster refreshes on Zoho app data, the lever is the connector’s sync schedule and your plan tier, not the connection mode.

The one thing worth remembering

The Zoho Analytics import vs Live Connect decision is not a performance question. It’s a question about which failure you’d rather have.

Import mode fails quietly. Numbers stay confidently wrong until someone reconciles them by hand, and the bill grows with every table you sync out of habit. Live Connect fails loudly. Someone renames a column and a dashboard breaks in front of a user, or a heavy query hits the timeout during the Monday review.

Loud failures are usually easier to live with, because you find out about them. But they only work if you have the source database, the access to tune it, and a change process that includes reporting. If you don’t have those three, import mode with an honest refresh schedule and a documented staleness window will serve you far better than a live connection you can’t maintain.

Need a second opinion on your Zoho Analytics setup?

Most of the Zoho Analytics work I get called into is somebody discovering the mode choice was made in five minutes eighteen months ago. Things I can help with:

• Auditing an existing workspace and telling you honestly whether the connection mode still fits what you’re using it for
• Profiling and tuning the source database behind a Live Connect workspace so dashboards stop hitting the query timeout
• Designing reporting views and materialized views that push transformation into the database where Live Connect can use it
• Setting up Zoho Databridge for databases in private networks, including SSL and SSH tunneling and the firewall rules around it
• Configuring incremental fetch on import-mode tables to cut sync times and stop row counts drifting upward
• Building a schema-change process so source migrations don’t turn into reporting incidents

If you want a concrete starting point, send me the slow query, the Mismatch screenshot, or the EXPLAIN output from the aggregation that’s timing out. That’s usually enough to tell where the actual problem is.

The post Zoho Analytics Import vs Live Connect: The Trade-Offs Nobody Mentions appeared first on John Nessime.

Those submissions went to the old server. It was still answering for a slice of the internet twenty hours after the DNS change, and by then I had already stopped syncing it.

That is the failure that separates a bulk WordPress migration from moving a single site. When you move one site you watch it. When you move twenty in a weekend, you are running a pipeline, and the things that go wrong are the ones nobody is watching: writes landing on a server you have mentally decommissioned, cron firing on two boxes at once, a certificate that never issued because a DNS record was still pointing at the old IP when the challenge ran.

This is the process I use for fleet moves onto a single VPS: the manifest that makes the weekend repeatable, the two-pass sync, the database step people rush, the certificate limits that bite specifically at fleet scale, and a cutover order that keeps rollback cheap. It assumes SSH on both boxes and some comfort on the command line.

The failure that actually costs you: writes landing on the old server

Every migration guide tells you to lower your TTL. Almost none of them tell you what happens in the gap afterwards, and the gap is where the money goes.

Here is the mechanism. You change the A record. Most resolvers pick up the new IP inside the TTL window. Some do not: corporate resolvers that clamp minimum TTLs, ISP resolvers that ignore short values, browsers holding a connection open, a mail server that resolved the host an hour ago. For those clients, the old server is still the site. It still has a working database. It still accepts comments, form entries, orders, user registrations and password resets.

Nothing errors. Nothing logs. The write succeeds on a database you are about to throw away.

There are only two honest answers to this, and picking one per site is the first real decision of the weekend.

• Accept the loss. Fine for brochure sites, documentation, most blogs. A missed comment is not a business problem.
• Make the old server read-only the moment DNS changes. Mandatory for anything with a checkout, a booking flow, a members area or a lead form that feeds a sales process.

Read-only in practice means maintenance mode on the old copy, a static holding page on the old vhost, or pointing the old install’s wp-config.php at a database user with SELECT only. Any of those turns silent data loss into a visible error the visitor can react to. Someone who sees “temporarily unavailable” comes back. Someone whose order vanished does not.

Do not skip this because the old server “will be off soon”. The window between DNS change and old server shutdown is the longest, least supervised part of the whole weekend.

Build the fleet manifest before you touch a server

Twenty sites is the point where memory stops working. You will not remember which one runs an old PHP branch, which one has a hardcoded absolute path in a theme, which one has MX records pointing somewhere else entirely.

So the first artifact is a table. One row per site, filled in before anything moves.

• Document root and database name
• Current siteurl and home values, exactly as stored
• PHP version the site currently runs under
• Disk size of the document root and dump size of the database
• Where DNS is authoritative, and who holds the login
• Whether mail for that domain is on this server or elsewhere
• Does it take writes from the public: comments, forms, checkout, registration
• Cache and object cache in use
• Cutover batch number and rollback owner

Most of that you can pull off the old server rather than typing it. This walks a DirectAdmin-style layout and prints a tab-separated line per WordPress install:

for d in /home/*/domains/*/public_html; do
  [ -f "$d/wp-config.php" ] || continue
  printf '%st%st%sn' 
    "$d" 
    "$(wp --path="$d" option get siteurl --skip-plugins --skip-themes 2>/dev/null)" 
    "$(du -sh "$d" | cut -f1)"
done

The --skip-plugins --skip-themes flags matter here. Without them WP-CLI boots the full site, and one broken plugin on one install will stop your inventory halfway through with a fatal error.

The DNS column is the one people fill in last and regret. Check where each zone actually lives before the weekend, not during it. A domain whose nameservers point at Cloudflare while you are editing records in the old host’s DNS panel will not move no matter how many times you refresh.

Rebuild the stack, don’t clone the disk

There is a tempting shortcut where you image the old server and restore it onto the new one. It usually works and it is usually the wrong call, because you have just carried across every accumulated fix, every orphaned config, and every reason the old box got slow enough to need replacing.

Build the destination clean. Whatever the provider is, whether that is Contabo, InterServer, Hetzner or a machine you already have, the sequence is the same: base OS, web server, PHP with the version and extensions the manifest says you need, database server, control panel if you use one, firewall, then a hardening pass.

Two things to get right before any files land.

PHP versions. If three sites need an older branch and seventeen are happy on current, install both and make the per-site assignment explicit in the manifest. Discovering the mismatch after cutover means a white screen on a live domain instead of a note in a spreadsheet. Check the extension list too, not just the version number, because imagick, soap and intl are the usual absentees.

Access during the weekend. If you lock SSH and the control panel behind an IP allowlist, decide beforehand what your source address will be. A rotating home IP is how people lock themselves out of their own migration at two in the morning. A dedicated-IP option from a VPN provider such as NordVPN or Surfshark gives you one stable address to allowlist; a second admin path on a non-standard port works too. Deciding at the moment you are locked out does not.

Two-pass rsync beats one big copy

The mistake at fleet scale is treating the file copy as a cutover-day task. Copy everything on cutover day and your maintenance window is however long twenty document roots take, which for a media-heavy fleet is hours you do not have.

Split it. Do a warm sync days in advance while both sites are live, then a delta sync at cutover that moves only what changed.

Warm sync, run from the old server, no deletes:

rsync -aHAX --numeric-ids --info=progress2 
  -e "ssh -p 22" 
  /home/user/domains/example.com/public_html/ 
  root@NEW_SERVER_IP:/home/user/domains/example.com/public_html/

What those flags buy you: -a preserves permissions, times, symlinks and ownership; -H keeps hard links, which some backup plugins and dedup schemes rely on; -A and -X carry ACLs and extended attributes; --numeric-ids stops rsync remapping ownership through name lookups when the UIDs differ between boxes, which is exactly what happens when the new server creates users in a different order.

Delta sync at cutover, same command with one addition:

rsync -aHAX --numeric-ids --delete --info=progress2 
  -e "ssh -p 22" 
  /home/user/domains/example.com/public_html/ 
  root@NEW_SERVER_IP:/home/user/domains/example.com/public_html/

--delete removes files on the destination that no longer exist on the source. It is what makes the second pass an accurate mirror rather than a merge of two states. It is also the flag that will destroy your afternoon if you get the paths wrong, so check the trailing slashes: a source path ending in a slash copies the contents of the directory, one without it copies the directory itself, one level deeper than you meant.

Run the delta pass once with --dry-run before running it for real. And on SELinux systems, relabel the tree afterwards: files arriving over rsync carry contexts the web server may not accept, and that produces a 403 which has nothing to do with permission bits.

Databases last, and never with sed

Files can be stale for a few days. Databases cannot. Dump each database as late as you reasonably can, immediately before that site’s cutover, not on Friday for a Sunday move.

# on the old server, per site
wp --path=/home/user/domains/example.com/public_html db export 
  /root/migration/example.com.sql --skip-plugins --skip-themes

Import on the new box, then handle URLs. If the domain is not changing, you may have nothing to replace. If the site is moving from HTTP to HTTPS, or from a staging hostname, or from a temporary preview domain you used for testing, you do.

The rule that matters: do not run a find-and-replace over the SQL file with sed. WordPress stores widget settings, theme options, page builder layouts and plugin config as PHP serialized arrays, and every serialized string carries a byte-length prefix. Change the string without recalculating the prefix and the array fails to unserialize. The site does not error loudly. It loses its widgets, or its theme options revert to defaults, or the page builder renders an empty page.

WP-CLI unserializes, replaces and reserializes, which is why it is the only tool I use for this.

wp search-replace 'http://example.com' 'https://example.com' 
  --all-tables-with-prefix 
  --precise 
  --skip-columns=guid 
  --report-changed-only 
  --dry-run

Read the dry-run output before you commit. Each flag earns its place: --all-tables-with-prefix catches plugin tables that share your prefix and would otherwise be skipped, --precise forces a full unserialize pass on every column instead of the faster string-level path, and --skip-columns=guid leaves the GUID column alone because feed readers use it as an identifier, not as a URL to fetch. Drop --dry-run to apply.

Flush the object cache afterwards if the site uses one. A stale Redis or Memcached entry will happily serve you the old values and convince you the replacement failed.

TLS at fleet scale: where Let’s Encrypt rate limits actually bite

Twenty certificates in one weekend is where a limit you have never noticed becomes the thing that blocks you.

Let’s Encrypt publishes its limits and they are worth reading properly before a fleet move. Two of them are the ones that catch migrations.

• New certificates per registered domain. Up to 50 certificates per registered domain every 7 days, counted globally across all accounts. If your twenty sites are twenty separate registered domains, you are nowhere near it. If they are subdomains of one or two parent domains, which is common for an agency fleet, you can burn through it.
• Authorization failures per identifier per account. Up to 5 failures per hostname per account per hour, refilling at one every 12 minutes. This is the one that actually bites, because it is triggered by the exact thing you are doing: requesting a certificate for a hostname whose DNS still points at the old server, so the HTTP-01 challenge is served by the wrong machine and fails.

Order matters as a result. If you issue before DNS moves, HTTP-01 fails. If you issue after DNS moves, visitors hit a certificate error in the gap. Three ways out, in the order I reach for them:

1. DNS-01 validation. Proves control through a TXT record instead of an HTTP request, so it does not care where the A record points. You can issue every certificate before touching DNS. This is the clean answer if your DNS provider has an API your ACME client supports.
2. Cut DNS first, issue immediately after. Simple, and the exposure is short if you are watching. Batch it so you are never waiting on twenty challenges at once.
3. Copy the existing certificates and keys across. Buys you a bridge until the new server can renew on its own. Only viable if you control the old server’s filesystem and the certificates have real time left.

Whichever you choose, test the plumbing against the staging environment first. Most ACME clients expose this as a dry-run mode, and it exists precisely so you can fail as many times as you like without spending production budget.

One more thing that catches people: if any site has HSTS enabled with a long max-age, browsers that have seen it will refuse to fall back to HTTP. There is no grace period and no way to talk them out of it. Those sites need a valid certificate on the new server from the first request, which pushes them firmly toward DNS-01 or a copied certificate.

Planning the cutover order for a bulk WordPress migration

Do not move twenty sites at once. Batch them, and batch by risk rather than alphabetically or by size.

1. Batch one: the throwaways. Two or three low-traffic sites you own or that nobody would notice. This batch exists to prove the stack, not to make progress.
2. Batch two: the ordinary majority. Standard brochure and content sites, no checkout, no logins that matter.
3. Batch three: the ones with writes. Stores, membership sites, anything with a form feeding a sales process.
4. Batch four: the awkward ones. Custom stacks, odd PHP requirements, third-party integrations that whitelist your server IP.

Stop between batches. Look at the error logs on the new server, not just the homepages. Twenty sites failing the same way is one problem; catching it after three is a fifteen-minute fix, catching it after twenty is a rollback.

The per-site sequence inside a batch:

1. Drop the TTL on that domain’s records to a low value, at least a day ahead.
2. Run the delta rsync.
3. Dump, transfer and import the database.
4. Run search-replace if the URL is changing at all.
5. Test through a hosts-file entry or a resolved request, before DNS moves.
6. Change the A and AAAA records.
7. Put the old copy read-only.
8. Verify from outside, then move to the next site.

Step five is the one people skip and should not. You can send a request to the new server as if DNS had already changed, without changing anything:

curl -sSI --resolve example.com:443:NEW_SERVER_IP https://example.com/
curl -sS --resolve example.com:443:NEW_SERVER_IP https://example.com/wp-login.php | head -n 20

That gives you real status codes, real redirects and a real TLS handshake against the new box while the live site is untouched. If that returns a 301 loop or a 500, you have found it before any visitor did.

Cron, email, and the things that run twice

During the overlap window you have two live copies of every site. Both have a database. Both have a scheduler.

WordPress schedules through wp-cron.php, fired by visitor traffic. The old server still gets traffic, so it still fires cron, so it still sends the newsletter, still processes the subscription renewal, still runs the backup plugin. Subscribers get things twice. Worse, the site’s system crontab came across in your file sync and is now also running on the new box.

Handle it explicitly. On the old copy, disable the loopback scheduler when you make it read-only:

define( 'DISABLE_WP_CRON', true );

And on the new server, set up a real system cron per site rather than leaving it to traffic, because a site with no traffic yet has no scheduler at all:

wp --path=/home/user/domains/example.com/public_html cron event run --due-now

Wrap that in a cron entry or a systemd timer at whatever interval the site needs, and check wp cron event list before assuming nothing important is queued.

Email is the parallel problem and it fails quietly in the other direction. Your sites now send from a new IP. If SPF still authorizes only the old server’s address, and DKIM keys did not come across, and DMARC is set to anything stricter than none, your transactional mail starts landing in spam. Nothing bounces visibly. Password resets just stop arriving and you hear about it in support tickets a week later.

Three records to check per domain. MX is the one not to touch casually: if mail lives elsewhere and you replace a zone wholesale, the mailboxes go with it.

• MX: unchanged unless mail is genuinely moving with you.
• SPF: add the new server’s IP before cutover, remove the old one after the overlap window closes, not during.
• DKIM: keys are per-server. Either copy them or regenerate and publish the new public key.

The more resilient answer is to stop sending through the server at all. Routing WordPress mail through an authenticated SMTP relay means outbound deliverability stops depending on which machine the site happens to live on, which makes the next migration a non-event.

Troubleshooting the weekend

Redirect loop on the new server. Usually siteurl and home disagreeing with what the vhost or reverse proxy is doing about HTTPS. Check both values with wp option get siteurl and wp option get home. If the site sits behind a proxy or CDN that terminates TLS, WordPress may see HTTP on the back end and redirect forever.

Site loads, media does not. Either uploads did not finish syncing, ownership is wrong, or database paths still reference the old host. Compare a file listing on both sides before assuming it is a database problem.

Widgets or theme options empty after import. This is the serialization damage described earlier. It is not recoverable by re-running the replacement. Restore the database from the dump and redo the URL change with WP-CLI.

Certificate issuance failing repeatedly. Confirm what the challenge is actually hitting before retrying, because each failure spends part of your hourly authorization budget.

dig +noall +answer example.com A
dig +noall +answer @1.1.1.1 example.com A
openssl s_client -connect NEW_SERVER_IP:443 -servername example.com < /dev/null 2>/dev/null | openssl x509 -noout -subject -dates

The first query uses your local resolver, the second asks a public one directly, and the difference between them tells you whether you are looking at a real propagation state or your own DNS cache.

Site fine, admin slow. Often a plugin trying to reach an external service that whitelisted the old IP. Look for API integrations, license checks and payment gateways in the manifest’s awkward-ones column.

Common mistakes

• Lowering TTL on cutover day. The old value is still cached; the change has no effect for as long as the old TTL says.
• Cancelling the old server as soon as the sites load. Keep it running, read-only, for the whole overlap plus a margin. It is your rollback.
• Replacing an entire DNS zone instead of editing the A record, and taking MX, SPF and verification TXT records with it.
• Running search-replace before importing, or on the SQL file instead of the live database.
• Testing only the homepage. Homepages are static and cached. Test a login, a form submission, an admin page and an uploaded image.
• Leaving both schedulers live and finding out through duplicate customer emails.
• Assuming the new server’s PHP extension set matches the old one because the version number does.
• Treating rollback as a fleet-level decision. Per-site rollback is what keeps one bad site from stalling nineteen good ones.

Best practices for a fleet move

• Write the manifest first and update it as you go. It is the migration record and the handover document.
• Take a restorable backup of each site to somewhere off both servers before you start, and test restoring one. A backup you have not restored is a hypothesis.
• Warm sync early, delta sync at cutover, dump databases last. Batch by risk, and prove the stack on sites nobody will miss.
• Verify every site from outside with a resolved request before its DNS moves.
• Point uptime monitoring at the new IP before cutover, not after. UptimeRobot or Better Stack will tell you about a broken site faster than the client will.
• Make the old copy read-only at the moment of DNS change, not at the end of the weekend.
• Keep a rollback note per site: which records changed, what they were before, who can revert them.
• Re-check cache configuration after the move rather than assuming it carried over, whether that is WP Rocket, WP Fastest Cache or a server-level layer.
• Retire the old server on a scheduled date with a final archived backup, not by forgetting to renew it.

Frequently asked questions

Is a bulk WordPress migration of 20 sites realistic in one weekend?

Yes, if the preparation happens in the week before. The weekend itself should be delta syncs, database moves and DNS changes. If you are still installing PHP extensions on Saturday morning, the weekend is not long enough.

How low should I set the TTL, and how far ahead?

A low value like 300 seconds, set at least 24 to 48 hours before cutover so the previous TTL has fully expired everywhere. Going much lower than 300 buys little and increases query load. Raise it back after the migration settles.

Will moving hosts hurt my search rankings?

Changing the server behind an unchanged domain is not itself a ranking event. What hurts is what goes wrong during it: 5xx responses while crawlers visit, failed certificates, redirect loops, a staging copy left indexable. Keep URLs identical, keep the site responding, and check Search Console afterwards.

Can I use a migration plugin instead of the command line for this?

For one site, plugins are a reasonable choice and handle serialized data correctly. At twenty the arithmetic changes: twenty manual export and import cycles through a browser, with no dry run and no delta sync. The command line is repeatable and scriptable, which is the point at fleet scale.

How long should I keep the old VPS running?

At least a week after the last site cuts over, in read-only mode. Some problems only appear over a full traffic cycle: weekly scheduled jobs, integrations that call in periodically, mail arriving at the old MX. Keep an archived backup after you destroy it.

What if one site breaks after cutover?

Revert that site’s A record and lift the read-only restriction on the old copy. That is why the old server stays up and why rollback is per site. If you sync the database back afterwards, be careful not to overwrite writes that landed on the new server in the meantime.

Do I need to move email at the same time?

No, and separating them is usually wiser. Move web first, confirm it is stable, then plan mailbox migration as its own change with its own window. Combining them means a single mistake in one DNS zone takes down both.

The one thing worth remembering

A bulk WordPress migration is not twenty small migrations. It is one process run twenty times, and the difference is that at scale the visible failures get caught while the invisible ones do not.

The copy is the easy part. rsync and WP-CLI are reliable, well documented and boring. What costs you is the overlap window: two servers both answering, both writing, both scheduling, for hours after you have mentally finished. Decide before the weekend what happens to writes on the old server, make that decision per site, and act on it at the moment DNS changes rather than at the end.

Do that, batch by risk, and keep a rollback that works at the level of a single site, and twenty sites in a weekend is an ordinary piece of work rather than a gamble.

Need help moving a fleet of sites?

Migrations are a good thing to have a second pair of hands on, because the expensive mistakes are the quiet ones. This is work I do regularly:

• Auditing an existing fleet and producing the migration manifest, batch order and rollback plan before anything moves
• Provisioning and hardening the destination VPS, including multiple PHP versions, control panel setup and firewall rules
• Running the sync and database cutover with WP-CLI, including URL replacement on serialized data
• Planning TLS issuance around ACME rate limits, including DNS-01 validation and automated renewal
• Sorting out post-migration email: SPF, DKIM, DMARC and moving transactional mail onto an authenticated relay
• Sitting in as the on-call engineer for the cutover window itself, or reviewing a plan you have already written

If you are planning a move and something in it worries you, send me the concrete thing: the site list, the current vhost config, the rsync output, the certbot error. I would rather look at the actual output than talk in general terms.

The post Bulk WordPress Migration: Moving 20 Sites to a New VPS in One Weekend appeared first on John Nessime.

That’s the shape of the problem. When you backfill historical API data into S3, the failure to worry about isn’t the one that throws. It’s the one that writes fewer records than it should and reports success anyway. Bytes transferred look plausible. Nothing goes red. The data is quietly wrong, and you find out when someone downstream trusts it.

This post covers the failure families that produce that outcome and the design decisions that make them impossible rather than merely unlikely: pagination that drifts under a mutating source, retries that write the same page twice under different keys, a prefix layout chosen while thinking about writes instead of reads, the seam where the backfill meets the live pipeline, and the storage class trap that makes a cheap-looking archive expensive to fix.

Why a backfill that finishes cleanly can still be wrong

A live pipeline has a natural error signal. Data stops arriving, a lag metric climbs, someone notices. A backfill has none of that. It runs once, against a range nobody is watching in real time, and then it’s over. The only evidence it leaves behind is a set of objects in a bucket.

So the question you need to answer months later isn’t “did the job succeed?” It’s “is this window complete, and how do I know?” If your design can’t answer that from artifacts on disk, you don’t have a backfill. You have a large amount of data of unknown provenance.

Failure family 1: the source moved while you paginated

This one bites hardest and shows up least. Most REST APIs paginate historical queries with an offset or page number. You ask for page 1, page 2, page 3. Meanwhile the underlying table is still accepting writes.

If a record is inserted into the middle of the result set between your page 4 and page 5 requests, everything after it shifts down by one. The row that would have started page 5 is now the last row of page 4, which you already fetched. You request page 5 and that record is gone. Not errored. Gone. Insertions cause skips, deletions cause duplicates, and the job still exits zero.

What to do instead

• Prefer keyset or cursor pagination. If the API offers an opaque cursor or a “records after ID X” parameter, use it. The read position is anchored to a value rather than a count, so inserts elsewhere in the set can’t shift it.
• Sort by an immutable column. If you’re stuck with offsets, sort by something that never changes after insert, usually the primary key or the creation timestamp. Sorting by a last-modified column while records are being modified is the worst case: rows physically move within the ordering as you walk it.
• Bound every request on both ends. Don’t ask for “everything before now.” Ask for a closed interval with an upper bound already in the past. A closed window can’t grow while you read it.
• Know which timestamp the filter applies to. Filtering on created-at gives you records that came into existence in that window. Filtering on updated-at gives you records touched in it, which for a mutable source is a different set entirely.

That last point causes arguments. Backfill on created-at, then run incremental syncs on updated-at, and your historical range never picks up edits made to old records. That may be correct for your use case. It is not correct by default, and it’s worth writing down which one you chose.

Failure family 2: the retry that wrote the same page twice

Backfills retry. They have to, because you’re hammering an API for hours and something will time out. The problem is what the retry writes.

If your object key contains a UUID, a run identifier or a wall-clock timestamp, a retried page produces a second object with a different key and the same records inside. You now have duplicates that listing will never reveal, because both objects are legitimately present and neither is obviously a copy. Derive the key from the request instead of the run, so the same window plus the same cursor always produces the same key and a retry overwrites rather than accumulates.

import hashlib
import boto3
from botocore.config import Config
from botocore.exceptions import ClientError

s3 = boto3.client(
    "s3",
    config=Config(
        retries={"total_max_attempts": 8, "mode": "standard"},
        max_pool_connections=32,
    ),
)

def page_key(dataset, window, cursor):
    # Deterministic: the same page always maps to the same key.
    token = hashlib.sha256(cursor.encode()).hexdigest()[:16]
    return f"raw/{dataset}/dt={window}/page-{token}.json.gz"

def put_once(bucket, key, body):
    try:
        s3.put_object(Bucket=bucket, Key=key, Body=body, IfNoneMatch="*")
        return "written"
    except ClientError as err:
        if err.response["Error"]["Code"] == "PreconditionFailed":
            return "already_present"
        raise

Two things in there are worth explaining rather than copying.

IfNoneMatch="*" is a conditional write. S3 accepts the PUT only if no object exists at that key and returns a 412 PreconditionFailed otherwise. It’s supported on PutObject and CompleteMultipartUpload. That turns “did I already write this page?” into a single request with no read-before-write race. If two workers hit the same key concurrently, S3 can instead return a 409 ConditionalRequestConflict, which AWS documents as retryable, so handle the two codes differently: retry on 409, skip on 412.

The same thing from the CLI, useful for testing the behaviour by hand:

aws s3api put-object 
  --bucket your-bucket 
  --key raw/orders/dt=WINDOW/page-abc123.json.gz 
  --body page.json.gz 
  --if-none-match "*"

The retries block matters for a reason that trips people up. In botocore, max_attempts means different things depending on where you set it: in the AWS config file or the environment variable it counts total requests including the first, but in a Config object it counts retries on top of the first. total_max_attempts exists only in a Config object and always means total, so using it removes the ambiguity. There’s also an adaptive mode that layers client-side rate limiting on top of standard; AWS still labels it experimental, so I reach for standard plus my own concurrency cap first.

Failure family 3: a layout designed for writing, not reading

The prefix structure you pick during the backfill is the one you’re stuck with. Reorganising later means rewriting every object, and copy requests aren’t free.

Partition on event time, not ingestion time

The tempting layout is the date you ran the job. It’s easy, it’s monotonic, and it’s useless. Every query anyone writes against historical data filters on when the thing happened, not when you happened to fetch it. If your partitions are ingestion dates, every one of those queries becomes a full scan. Use Hive-style keys so query engines prune without extra configuration:

s3://your-bucket/raw/orders/dt=YYYY-MM-DD/page-<token>.json.gz
s3://your-bucket/curated/orders/dt=YYYY-MM-DD/part-<n>.parquet

Keep the raw landing zone and the curated zone separate. Raw is whatever the API gave you, unmodified, so you can re-derive everything if your parsing logic turns out to be wrong. Curated is columnar and typed. Merging them saves nothing and costs you the ability to reprocess.

Don’t let the write path throttle itself

S3 documents at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix, with no limit on the number of prefixes. The word doing the work is partitioned. That rate applies once S3 has split the keyspace out for that prefix, and AWS is explicit that the scaling is gradual rather than instant. During the ramp you can see 503 SlowDown responses even when your sustained rate looks well inside the documented ceiling.

For a backfill this is mostly good news, since date-partitioned keys spread writes across many prefixes as you work through the range. It becomes a problem when you parallelise within a single day and drive thousands of writes per second into one dt= prefix. Two mitigations: back off with jitter instead of retrying 503s immediately, and shard workers across windows rather than within them.

Register partitions without a scan

A backfill can create thousands of partitions at once. MSCK REPAIR TABLE handles that badly. AWS documents that it only adds partitions and never removes them, that it can take a long time on large tables, and that it can time out leaving the catalog partially updated. Partial success with no error is exactly the failure mode this whole post is about.

Partition projection sidesteps the catalog. Athena computes the partition list from table properties instead of looking it up, so a backfilled range is queryable the moment the objects exist.

TBLPROPERTIES (
  'projection.enabled'          = 'true',
  'projection.dt.type'          = 'date',
  'projection.dt.format'        = 'yyyy-MM-dd',
  'projection.dt.range'         = 'START_DATE,NOW',
  'projection.dt.interval'      = '1',
  'projection.dt.interval.unit' = 'DAYS',
  'storage.location.template'   = 's3://your-bucket/curated/orders/dt=${dt}/'
)

Replace START_DATE with the earliest date in your backfill. The trade-off is worth stating plainly: projection generates every partition in the range whether or not data exists there, so an over-wide range on a sparse dataset wastes planning effort. And if the S3 layout ever stops matching storage.location.template, queries return nothing rather than erroring. Set the range to your actual data, not to a comfortable margin.

Failure family 4: the seam between backfill and live pipeline

Two processes write to the same table: a backfill walking backwards through history, and a live ingest walking forwards. They meet somewhere, and that meeting point is where records go missing.

The instinct is to make the handoff exact. Backfill up to timestamp T, start streaming from T. Don’t. Clock skew between your runner and the API, records that arrive late at the source, and rounding in the API’s own filtering will all conspire to open a hole at exactly T.

Overlap deliberately instead. Run the backfill past the point where live ingest started, by a margin generously larger than any plausible source lag, then deduplicate on a natural key during curation. Duplicates you can remove with a query. Gaps you can only fix by going back to the API, and by then the retention window may have closed.

Failure family 5: the storage class you can’t undo

Historical data feels like archive data, so the instinct is to write it straight into a cold class or transition it immediately. Read the mechanics first, because the billing rules for cold classes are structural and pro-rated against you.

• S3 Standard-IA and One Zone-IA carry a 30-day minimum storage duration and a 128 KB minimum billable object size.
• Glacier Instant Retrieval and Glacier Flexible Retrieval carry a 90-day minimum duration; Deep Archive carries 180 days. Delete, overwrite or transition earlier and you’re charged the remainder.
• Objects in Glacier Flexible Retrieval and Deep Archive are billed for 40 KB of extra per-object metadata on top of the object itself.

Read those together and the conclusion is uncomfortable. A backfill that writes millions of small JSON pages into an archive class pays a size floor on every one of them, plus metadata overhead, plus a minimum duration that makes fixing the mistake expensive. Those pages are also the objects you’re most likely to want to reprocess, which is exactly the overwrite that triggers the early-deletion charge.

What I’d actually do: land raw pages in Standard, compact them into larger Parquet files in the curated zone, verify completeness, and only then apply a lifecycle rule to the raw prefix. Compaction is what makes cold storage sane, because it turns a size floor per tiny object into a rounding error on a large one. It also fixes the small-file problem in Athena, which is a separate cost line.

One more cost note: if your backfill runs from EC2 in a private subnet, every byte pulled from the API and pushed to S3 may route through a NAT gateway with per-GB processing charges. A gateway VPC endpoint for S3 removes the S3 half of that. It’s a five-minute change people usually discover after the bill.

Proving the backfill is complete

Here’s the part most backfills skip, and the part that makes everything above verifiable rather than hopeful. Write a manifest object per window, alongside the data, describing what that window contains.

{
  "dataset": "orders",
  "window": "<dt value>",
  "filter_field": "created_at",
  "range_start_inclusive": "<iso8601>",
  "range_end_exclusive": "<iso8601>",
  "pages_written": "<int>",
  "records_written": "<int>",
  "source_reported_total": "<int or null>",
  "last_cursor": "<opaque token>",
  "schema_fingerprint": "<sha256 of sorted field names>"
}

The manifest turns completeness into something you can query. A window with no manifest was never finished. A window whose records_written disagrees with source_reported_total is suspect. A window whose schema_fingerprint differs from its neighbours tells you the API changed shape partway through your range, which is otherwise close to undetectable.

Write the manifest last, after every page for that window is confirmed in S3. A manifest written optimistically at the start is worse than no manifest at all, because it asserts a completeness nobody checked.

Troubleshooting a backfill that already ran

1. Count objects per window and look at the shape. Use S3 Inventory rather than a live ListObjectsV2 walk on a large bucket. Plot objects per dt= partition. Real data has a rhythm, weekday peaks and weekend troughs. A window that breaks the rhythm without a business reason is your gap.
2. Reconcile record counts against the source. Many APIs return a total count on the first page of a filtered query. Pull that for a sample of windows and compare with what’s in S3. If you kept manifests, this is one query.
3. Check for duplicate natural keys within a partition. A GROUP BY on the record ID with HAVING COUNT(*) > 1 finds retry duplicates immediately. Duplicates clustered at window boundaries mean pagination drift; duplicates spread evenly mean your keys weren’t deterministic.
4. Look at boundary hours, not whole days. Cursor drift and clock skew concentrate at the edges of each request window. Aggregate to the hour and the missing slice usually becomes obvious even when daily totals look fine.
5. Verify the layout still matches the table definition. If you’re using partition projection and a query returns zero rows, the first suspect is storage.location.template not matching the actual prefix, trailing slashes included.

Common mistakes

• Treating an exit code of zero as evidence of completeness.
• Putting a run ID or timestamp in the object key, which makes retries additive instead of idempotent.
• Partitioning by ingestion date because it’s what the job naturally knows.
• Making the backfill and live-ingest boundary exact instead of overlapping.
• Writing millions of small objects straight into an archive class and meeting the minimum object size and duration rules afterwards.
• Running MSCK REPAIR TABLE over thousands of new partitions and trusting that it completed.
• Backfilling on a last-modified field while assuming you captured creation events.
• Sharding workers within a single day, concentrating writes on one prefix and inviting 503s.

Best practices when you backfill historical API data into S3

• Make every window independently re-runnable. If re-running one day is routine rather than scary, you’ll actually fix problems when you find them.
• Derive object keys deterministically from window plus cursor, and use conditional writes so a retry is a no-op.
• Bound every API request on both ends, and never let the upper bound be “now”.
• Write raw responses untouched before parsing anything. Parsing bugs are common; re-fetching a closed API window often isn’t possible.
• Emit a manifest per window, written last, and reconcile it against source counts.
• Compact to Parquet before applying any lifecycle transition.
• Run the job somewhere it can survive for hours. A long backfill on a laptop over hotel wifi is a bad plan; a small dedicated VPS from a provider like Contabo or InterServer, or a spot-priced EC2 instance, costs little and won’t sleep halfway through.
• Watch progress somewhere other than the terminal. Pushing records-per-window into CloudWatch or Grafana gives you a curve you can eyeball, and an unusually flat stretch is a gap you can catch while the source data is still retrievable.

FAQ

Should I write raw JSON or convert to Parquet during the backfill?

Both, in two zones. Write the raw response body first so the fetch step has no dependency on your schema assumptions, then convert as a separate job. Converting inline couples an expensive, rate-limited API walk to a parsing step that will eventually throw on an unexpected field, and you don’t want to re-fetch years of data because of a type error.

How large should each object be?

In the raw zone, one object per API page is fine and keeps idempotency simple. In the curated zone, aim for files in the tens to low hundreds of megabytes. Query engines pay a fixed overhead per file, so thousands of tiny Parquet files scan slower and cost more than a few large ones holding identical data.

Can I run the backfill in parallel across date windows?

Yes, and that’s the right axis, because each window is independent and lands on its own prefix. The binding constraint is almost always the source API’s rate limit rather than S3. Add a global concurrency cap and honour any documented limit or Retry-After header. Getting your credentials throttled or suspended mid-backfill is a worse outcome than finishing slowly.

Do I need AWS Glue or Airflow for this?

Not to start. A single script that takes a window as an argument, writes pages and a manifest, and is safe to re-run covers most backfills. Orchestration earns its place when you need retries across hundreds of windows, dependency ordering, or a visible run history. Glue and Step Functions fit AWS-native stacks; Airflow gives you more control and more to operate. Pick after the script works, not before.

What if the API caps how far back I can query?

Then the retention boundary is your real deadline and it should drive priority. Fetch the oldest reachable data first, not the most recent, because the newest data is still available tomorrow and the oldest may not be. If the vendor offers a bulk export or reporting endpoint separate from the standard API, it’s usually faster and less rate-limited than paginating.

Does any of this change with S3-compatible storage instead of AWS?

The layout, idempotency and manifest patterns carry over unchanged to Cloudflare R2, Backblaze B2, MinIO and similar. What varies is API-level feature support. Conditional writes with If-None-Match, request-rate behaviour and storage class semantics differ by provider, so verify the specific behaviour you’re depending on against that provider’s documentation rather than assuming parity with S3.

How do I know the backfill is actually finished?

When every expected window has a manifest, every manifest’s record count reconciles against the source, and a duplicate check on natural keys returns nothing. Not when the script exits.

The one thing worth remembering

A backfill isn’t finished when the job exits. It’s finished when you can prove the range is complete without re-reading the source. Deterministic keys, conditional writes, closed windows, manifests and deliberate overlap all exist to make that proof cheap.

If you’re about to backfill historical API data into S3, spend the extra hour on the key derivation and the manifest before you start the run. The alternative is learning about a gap from a chart, months later, with the source retention window already closed behind you.

Need a hand with a backfill?

This is a large part of what I do. If you’re staring at a historical range and an API with a rate limit, I can help with:

• Designing an idempotent, re-runnable backfill job with deterministic keys and per-window manifests, so a failed day is a one-command fix.
• Choosing a partition layout that serves your actual query patterns, and setting up Athena or Glue tables with partition projection so thousands of backfilled partitions become queryable without catalog scans.
• Auditing a backfill that already ran, reconciling record counts against the source and finding the gaps and duplicates before your dashboards do.
• Handling awkward source APIs: offset-only pagination, aggressive rate limits, undocumented result caps, schema drift partway through a historical range.
• Compaction and lifecycle design, so cold storage actually saves money instead of hitting minimum object size and minimum duration charges.
• Wiring the seam between the backfill and your live pipeline so there’s no gap at the handover, plus the monitoring that would have caught it.

Send me something concrete: a sample API response, your current key layout, or the object counts per partition. That’s usually enough to spot where the gap is.

The post Backfilling Historical API Data into S3 Without Silent Gaps appeared first on John Nessime.

Then someone points out that this particular support lead has never had access to Opportunity records. Not through her profile, not through the role hierarchy, not through a sharing rule. And the assistant just read one out to her.

That is the failure that kills most CRM assistant projects, and it almost never shows up in testing, because you test as an admin. Everything you ask works. Everything you ask is supposed to work. The gap only opens when a real user with a real, narrow profile starts asking questions.

This post walks through building a CRM knowledge assistant on Salesforce and Amazon Bedrock Knowledge Bases: which of the two knowledge base paths you are actually allowed to use, how the Salesforce connector authenticates, what it does and does not carry across from your CRM, and the two architectures that survive a real permission model. The wiring is the easy part. The scoping is the work.

The three moving parts

Strip away the marketing and a CRM knowledge assistant is three things bolted together:

• Ingestion — something pulls Cases, Knowledge articles, Opportunities and Accounts out of Salesforce on a schedule, chunks them, embeds them, and writes the vectors somewhere.
• Retrieval — a query comes in, gets embedded, and the closest chunks come back. This is where filtering lives, and it is the security surface.
• Generation — a foundation model reads those chunks and writes an answer with citations.

Generation is the part everyone stares at and the part you should worry about least. It is a model choice and a prompt. Retrieval is where the assistant becomes either trustworthy or a liability.

Decision one: which knowledge base you are allowed to use

Bedrock now has two shapes of knowledge base, and this decision gets made for you the moment you say “Salesforce”.

Bedrock Managed Knowledge Base is the newer, opinionated one. AWS manages the vector store, picks the embedding and reranking models by default, handles parsing, and exposes agentic multi-hop retrieval. It ships with a small set of first-party connectors: Amazon S3, SharePoint, Confluence, Google Drive, OneDrive, Web Crawler, and a custom ingestion path. Salesforce is not on that list.

Customer-managed knowledge bases are the older shape. You bring the vector store, you pick the embedding model, you own more of the pipeline. This is where the Salesforce connector lives, and AWS still flags that connector as a preview release subject to change. It also has a hard constraint worth reading twice: with a Salesforce data source, Amazon OpenSearch Serverless is currently the only supported vector store. If your organisation has standardised on Aurora pgvector or something else, that standard does not apply here.

So the fork is real. Either you use the preview Salesforce connector on the customer-managed path, or you land Salesforce data in S3 yourself and use the managed path. I will come back to why the second option is often the right call, because it is not just a workaround.

One piece of context that matters if you are inheriting an older design: Amazon Kendra entered maintenance mode on 30 June 2026 and stopped accepting new customers on 30 July 2026. Kendra was the standard answer to “search my Salesforce org” for years, and AWS now points people at Bedrock Knowledge Bases instead. Existing indexes keep running and no end-of-service date has been set, but Kendra will not gain features. If a proposal on your desk starts with a Kendra index, that is worth raising before anyone writes code.

The permission model is the whole problem

Salesforce spends enormous effort on who can see what. Profiles, permission sets, the role hierarchy, org-wide defaults, criteria-based sharing rules, field-level security, restriction rules. A given Opportunity is visible to a specific, computed set of users, and that computation happens on every query.

None of that survives ingestion.

The AWS documentation is blunt about it, and this is the single most important sentence in the whole feature: everything you sync from your data source becomes available to anyone holding bedrock:Retrieve on that knowledge base, including content that had controlled permissions at the source. Three layers of Salesforce scoping collapse into one IAM action.

Worse, the connector authenticates as a single identity. You configure a Connected App using the OAuth 2.0 client credentials flow, and that flow runs as one named user. Whatever that user can see, the crawler ingests. Pick a System Administrator to “make sure the sync works” and you have just copied the entire org into a flat index that your whole application tier can read.

Why this is invisible until it isn’t

Nothing errors. There is no warning in the console, no failed sync, no denied API call. Retrieval returns chunks with good similarity scores and the model writes a fluent, cited answer. The only signal that something is wrong is a human recognising a record they should not be looking at, and that human is usually a customer-facing employee, not you.

Treat the knowledge base as a published dataset, not as a view over Salesforce. Ask the question you would ask about an S3 bucket: if every row of this landed in front of every user of the app, who gets hurt?

The two architectures that actually hold up

Option A: ingest only what everyone is allowed to see. Run the Connected App as a deliberately low-privilege integration user whose profile grants read access to Knowledge articles, published solutions, product data and price books, and nothing else. Exclude Contact, Lead, Case and Opportunity entirely. You get a genuinely useful support and enablement assistant, the permission question disappears, and you can ship it without a security review turning into a project.

This is the one I reach for first. Most of the value people want from a CRM knowledge assistant is “what do we know about this product, this policy, this recurring problem”, and none of that needs record-level data.

Option B: export to S3 with your own access metadata. If you genuinely need per-record answers, stop using the connector. Pull the objects you need through the Salesforce API or a scheduled extract, write each record to S3 as a document, and write a sidecar metadata file next to it carrying the access attribute you will filter on. Then point a knowledge base at the bucket and apply an explicit metadata filter on every retrieval, derived from the caller’s identity rather than from the query text.

The cost is honest: you now own an extraction pipeline, a metadata schema, and the job of keeping the access attribute correct when a record changes owner. That is real work. The benefit is that you can point at exactly where the boundary is enforced and test it.

What you cannot do is enforce the boundary in the prompt. “Only answer about accounts the user owns” is a suggestion to a language model, not an access control. Retrieval already happened by then.

Wiring the Salesforce connector

Assuming you have chosen Option A and want the connector, here is the shape of it.

On the Salesforce side

1. Create a Connected App in your org and enable the OAuth 2.0 client credentials flow.
2. Under the client credentials flow settings, set the Run As field to your integration user. This is the step people skip, and the sync fails without it. It is also the step that decides your entire blast radius, so pick that user before you pick anything else.
3. Copy the consumer key and consumer secret from the app’s OAuth settings.
4. Note your instance URL.

On the AWS side

Store the credentials in AWS Secrets Manager. The connector looks for three specific keys, and it will not fall back to alternatives if you name them differently:

{
  "consumerKey": "your Connected App client ID",
  "consumerSecret": "your Connected App client secret",
  "authenticationUrl": "https://yourcompany.my.salesforce.com/"
}

The secret has to live in the same AWS region as the knowledge base. Cross-region reads are not supported here, and the resulting error is generic enough to waste an hour. Do not reuse this secret for any other data source; rotate it on the same schedule as your other integration credentials.

Creating the data source

The data source configuration is a JSON document. The interesting part is patternObjectFilter, which is how you keep the crawler away from objects you never wanted:

{
  "salesforceConfiguration": {
    "sourceConfiguration": {
      "hostUrl": "https://yourcompany.my.salesforce.com/",
      "authType": "OAUTH2_CLIENT_CREDENTIALS",
      "credentialsSecretArn": "arn:aws:secretsmanager:REGION:ACCOUNT:secret:NAME"
    },
    "crawlerConfiguration": {
      "filterConfiguration": {
        "type": "PATTERN",
        "patternObjectFilter": {
          "filters": [
            {
              "objectType": "Knowledge__kav",
              "inclusionFilters": [".*"]
            },
            {
              "objectType": "Case",
              "exclusionFilters": [".*"]
            }
          ]
        }
      }
    }
  },
  "type": "SALESFORCE"
}

Two things to internalise about these filters. First, they are regular expressions matched against a per-object identifier, not against record contents: for Knowledge__kav that is the article title, for Case the case number, for Attachment the file name. Second, when an inclusion and an exclusion filter both match the same document, exclusion wins. That precedence is what makes an allow-list pattern safe to write.

Then create the data source and attach it to an existing knowledge base:

aws bedrock create-data-source 
  --name "salesforce-knowledge" 
  --knowledge-base-id "YOUR_KB_ID" 
  --data-source-configuration file://salesforce-connector-config.json 
  --data-deletion-policy "DELETE" 
  --vector-ingestion-configuration '{"chunkingConfiguration":[{"chunkingStrategy":"FIXED_SIZE","fixedSizeChunkingConfiguration":[{"maxTokens":"300","overlapPercentage":"10"}]}]}'

A note on the command namespace: the agent-plane operations have historically lived under a separate CLI command group from the model-inference ones, and this has shifted between CLI versions. Run aws bedrock help against your installed version before assuming. The JSON payload is the stable part.

--data-deletion-policy DELETE means the vectors get removed from the store when you delete the data source. The alternative retains them. Choose DELETE unless you have a specific reason not to, because orphaned vectors in an OpenSearch Serverless collection are both a cost line and a surprise waiting for whoever inherits this.

After that, kick off a sync with StartIngestionJob, or hit Sync in the console. The connector does incremental syncs after the first run: added, updated and deleted content only. The first run crawls everything.

Chunking is a one-way door

You cannot change the chunking strategy after connecting the data source. Not tune it, not swap it. You delete the data source and start over, which means re-crawling and re-embedding everything.

So spend twenty minutes here rather than five. CRM content is not uniform. Knowledge articles are structured prose and behave like documentation. Case comment threads are short, conversational, and lose their meaning when split mid-exchange. Opportunity descriptions are often two sentences. A fixed-size chunk that suits one of these will mangle another.

The practical move is to stop trying to find one strategy that fits everything. Create separate data sources for materially different content shapes, each with its own chunking configuration, all pointing at the same knowledge base. It costs you a bit of configuration and saves you a full re-index later.

One more constraint: the Salesforce connector does not handle multimodal content. Tables, charts and diagrams embedded in your Knowledge articles are not going to be understood. If your best troubleshooting articles are mostly screenshots, this connector will quietly ingest the surrounding text and nothing else, and your retrieval quality will be worse than the article count suggests.

Retrieval, filtering, and one trap

Bedrock gives you two kinds of metadata filter at query time, and confusing them is a security bug.

Explicit filters are the ones you construct in code and pass in the retrieval configuration. Your application decides them from the authenticated caller. They are deterministic.

{
  "retrievalQuery": { "text": "renewal value on the Northwind account" },
  "retrievalConfiguration": {
    "vectorSearchConfiguration": {
      "numberOfResults": 8,
      "filter": {
        "andAll": [
          { "equals": { "key": "visibility_group", "value": "emea-enterprise" } },
          { "equals": { "key": "record_class", "value": "internal" } }
        ]
      }
    }
  }
}

Implicit filters are the other kind. You declare which metadata attributes exist and Bedrock uses a model to infer filters from the natural-language query. Someone asks about last quarter, it infers a date filter. This is a genuinely nice relevance feature and it is not an access control. A model inferring a filter from user-supplied text is a model that can be talked out of it. Use implicit filters to improve answers; use explicit filters to enforce boundaries.

The trap: filters on the Salesforce connector operate on auto-detected fields from your CRM records. You cannot inject your own visibility_group attribute at ingestion. That capability comes from sidecar metadata files, which is an S3 data source feature. This is the concrete reason Option B routes through a bucket. If per-user scoping is a requirement, the connector cannot give it to you no matter how carefully you configure it.

Where the money goes

Rates change, so learn the meters instead of memorising numbers.

• Embedding at ingestion — billed per token processed. A full re-crawl re-embeds everything, which is exactly why the chunking decision is expensive to reverse.
• Vector storage — on the customer-managed path this is your OpenSearch Serverless collection, billed on provisioned compute units and storage, with a floor whether or not anyone queries it. This is the line item that surprises people on a proof of concept that nobody uses. Managed Knowledge Base bills on indexed data size and retrieval count instead, with no idle collection sitting underneath.
• Generation — input and output tokens on whichever foundation model you chose. Retrieved chunks are input tokens, so numberOfResults is a direct cost lever and worth tuning down until quality actually degrades.
• Everything around it — the API layer, the chat frontend, and the observability stack. A small VPS from a provider like InterServer or a Lambda behind API Gateway both work; the choice barely matters next to the Bedrock line. Push retrieval latency and citation counts into Grafana Cloud or Datadog early, because “the assistant feels worse this week” is otherwise unfalsifiable.

Troubleshooting

• Sync fails immediately with an authentication error. Check the Run As user on the Connected App’s client credentials flow first. An empty Run As field is the most common cause. Then check that the secret keys are spelled exactly consumerKey, consumerSecret, authenticationUrl, and that the secret is in the knowledge base’s region.
• Sync succeeds but the document count is far lower than expected. Usually the integration user’s profile, not the filters. Query the object as that user in Salesforce and count. Also check the per-knowledge-base file count and file size quotas, which are separate limits and are silently enforced.
• Retrieval returns nothing for queries you know are covered. Test with no filter at all. If results appear, your filter keys do not match the auto-detected field names. If they do not, the content probably never made it past chunking; check whether those articles are mostly images.
• Deleted Salesforce records still turn up in answers. Incremental sync handles deletions, but only on the next sync. If you deleted something sensitive, run a sync immediately rather than waiting for the schedule.
• Answers are confidently wrong about numbers. Fixed-size chunking split a record away from its label. Retrieve the raw chunks for that query and read them. Nine times out of ten the problem is visible in the chunk, not in the model.

Common mistakes

• Running the Connected App as a System Administrator “just to get the sync working”. That decision is almost never revisited.
• Assuming sharing rules and field-level security follow the data into the index. They do not.
• Enforcing access in the system prompt. Retrieval has already happened by the time the model reads your instructions.
• Ingesting Contact and Lead because they were there. Both carry personal data and both are usually irrelevant to the questions people actually ask.
• Accepting default chunking, then discovering it is immutable.
• Building on a preview connector without telling stakeholders it is preview. The configuration surface can change under you.
• Testing only as an admin. Every acceptance test should run as the narrowest real profile you have.

Best practices

• Decide the integration user’s profile before you touch AWS. It defines the security boundary more than any IAM policy will.
• Start with Knowledge articles only. Ship that, get real queries, then argue about whether Opportunity data is worth the architecture it requires.
• Write filters as allow-lists. Exclusion beats inclusion, so an allow-list fails closed.
• Split content shapes across separate data sources so chunking suits each one.
• Keep a fixed evaluation set of about fifty real questions with known-good answers, and re-run it after every configuration change.
• Log the retrieved chunk IDs alongside every generated answer. When someone reports a bad answer months from now, that log is the only thing that makes it debuggable.
• Treat bedrock:Retrieve as a privileged permission and scope it to the specific knowledge base ARN, not to a wildcard.

Frequently asked questions

Does the Bedrock Salesforce connector respect sharing rules?

No. The crawler authenticates as one user and everything it can read becomes retrievable by anyone with bedrock:Retrieve on the knowledge base. Per-user scoping has to be built at the retrieval layer with explicit metadata filters, which in practice means routing through S3 rather than using the connector.

Can I use Aurora or pgvector instead of OpenSearch Serverless?

Not with the Salesforce data source. AWS currently restricts that connector to Amazon OpenSearch Serverless. If your vector store is non-negotiable, extract to S3 and use a data source that supports your chosen store.

Should I use Agentforce instead of building this on Bedrock?

If your questions are answerable entirely from Salesforce data and your users already live in Salesforce, Agentforce is the shorter path and it operates inside the platform’s existing permission model, which removes the hardest problem in this post. Bedrock earns its complexity when the assistant has to reason across CRM data plus sources that are not in Salesforce, or when you need a specific foundation model, or when the data must stay inside your AWS account. Cost profiles differ substantially and are worth modelling on your real query volume before committing.

How current will the assistant’s answers be?

As current as your last sync. Ingestion is scheduled, not live. For rapidly changing fields such as case status or pipeline stage, do not answer from the index at all: have the assistant call the Salesforce API for the live value and use the knowledge base only for the surrounding context.

What happens to the connector when it leaves preview?

Unknown, which is the point of the label. Preview means the configuration surface, supported objects and behaviour can change without the compatibility guarantees you would expect from a GA service. Keep your data source configuration in version control so a re-create is a rerun rather than an archaeology exercise.

Is a CRM knowledge assistant worth building if we already have Salesforce search?

It depends on the question shape. Search is better when people know what record they want. A CRM knowledge assistant wins on synthesis: “what usually causes this error for enterprise customers” pulls from twenty articles and cases at once, which no search result page does well. If your users mostly navigate to known records, you will not get much return.

Do I need Bedrock AgentCore for this?

Not for retrieval and generation. AgentCore matters when you want the assistant to take actions, chain tools, or be exposed to agent frameworks over MCP. A read-only question-answering assistant works fine against the knowledge base APIs directly, and skipping AgentCore removes a layer you would otherwise have to operate.

The one thing to remember

Connecting Salesforce to Amazon Bedrock takes an afternoon. The connector works, the syncs run, the answers are good. That speed is exactly what makes this dangerous, because nothing in the setup path forces you to think about who the answers are for.

A CRM knowledge assistant is a publishing decision wearing an engineering costume. Every record you ingest, you are publishing to everyone who can call retrieve. Decide what that audience is allowed to see, encode it in the integration user’s profile and in your inclusion filters, and test every question as your narrowest real user. Do that first and the rest of the build is routine.

Need help getting this into production?

I work with teams building retrieval systems on AWS, usually at the point where a working prototype has to survive a security review. Things I can help with on a CRM knowledge assistant:

• Reviewing an existing Bedrock knowledge base for records that should never have been ingested, and scoping the integration user’s Salesforce profile to match.
• Designing the S3 extraction path with sidecar access metadata when per-user filtering is a hard requirement.
• Choosing between the managed and customer-managed knowledge base paths, with the OpenSearch Serverless cost floor modelled against your actual query volume.
• Building the chunking and data source split before it becomes an immutable mistake, plus a repeatable evaluation set to measure changes against.
• Terraform or OpenTofu for the whole stack: knowledge base, data sources, Secrets Manager, IAM, and the vector store.
• Retrieval observability so you can tell whether answer quality moved, and why.
• Migration planning for teams currently on a Kendra index.

If you have a data source configuration, a failing sync, or a retrieval result that looks wrong, send it over and we can work out what it is doing.

The post Build a CRM Knowledge Assistant on Amazon Bedrock Without Leaking Salesforce Records appeared first on John Nessime.

A nightly sync from a partner’s GraphQL API had run for weeks without a single failed execution. Green across the board in the console. Then someone asked why one column had been empty since the previous Tuesday. Not sparse. Empty. Every row, every night, for six nights.

The API had started returning a field-level error on that one field. Status code 200. A populated data block. An errors array sitting right next to it. The fetcher parsed data, wrote the records to S3, exited zero, and the orchestrator logged a success. Nothing in the stack was wrong, exactly. Everything in the stack was checking the wrong thing.

This post covers how to build a GraphQL data ingestion pipeline on AWS that fails honestly. It is organised by failure family rather than by service, because the AWS wiring is the easy part. The hard part is that GraphQL breaks four assumptions that REST ingestion quietly relies on, and every one of them fails silently by default.

Why GraphQL ingestion breaks the habits REST taught you

Most ETL code carries three inherited reflexes from years of REST work: the status code tells you whether it worked, the response shape is the server’s business, and rate limits are counted in requests. GraphQL invalidates all three.

• The status code is not the contract. The GraphQL over HTTP specification says a server should return 200 even when field errors are raised during execution, because a partial response is still a successful execution.
• You own the response shape. The selection set is written by you, which means schema drift shows up as a query that stops being valid, not as a payload that changes underneath you.
• Cost is query-shaped, not endpoint-shaped. One endpoint can cost almost nothing or blow your entire budget depending on how deeply you nested it.

Each of those becomes a distinct failure family. Let’s take them in the order they bite.

Failure one: the 200 OK that quietly ate a column

This is the one that costs the most and shows up the latest. GraphQL splits errors into two categories, and they behave completely differently on the wire.

• Request errors happen before execution starts: syntax errors, validation failures, bad variable coercion. No data key is returned at all. Nothing executed.
• Field errors happen during execution: a resolver threw, a downstream service timed out, a permission check failed. Execution continues, the failed field goes null, and you get data and errors together.

Field errors are the dangerous ones for ingestion, because they look exactly like success to anything that only inspects the transport layer. Worse, GraphQL’s null propagation rules mean a single failing non-null field bubbles up to the nearest nullable ancestor, so one broken resolver can hollow out an entire nested object while the surrounding rows land normally.

The fix is to make the response body the contract. Classify before you commit anything:

import json, urllib.request

class RequestError(Exception): pass
class FieldError(Exception): pass

def graphql(endpoint, token, query, variables):
    body = json.dumps({"query": query, "variables": variables}).encode()
    req = urllib.request.Request(
        endpoint,
        data=body,
        headers={
            "Content-Type": "application/json",
            "Accept": "application/graphql-response+json, application/json",
            "Authorization": f"Bearer {token}",
        },
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        payload = json.loads(resp.read())

    # The status code told us nothing. The body is the contract.
    if "data" not in payload:
        # Nothing executed. Query or auth problem. Do not retry blindly.
        raise RequestError(payload.get("errors"))

    if payload.get("errors"):
        # Partial response. data is populated but incomplete.
        raise FieldError(payload["errors"])

    return payload["data"]

Two things worth explaining. The Accept header asks for application/graphql-response+json, the media type the GraphQL over HTTP spec recommends, which lets a compliant server return a real 4xx for pre-execution failures instead of burying them in a 200. Servers that don’t support it fall back to application/json, so including both is free.

The second is the split between the two exception types. They need different handling. A request error is a bug in your query or a credential problem, and retrying it just burns budget on the same failure. A field error is transient often enough to be worth retrying, but never worth committing.

Deciding what “partial” means for your data

Failing the whole page on any error is the safe default and the one I reach for first. But it’s not always right. If you are pulling a hundred fields and one optional enrichment field is flaky, refusing every page forever is its own outage.

The workable middle ground is to classify by path. Every error object carries a path array pointing at the exact field that failed. Maintain a small allowlist of paths that are permitted to be null, fail the page on anything outside it, and emit the error count as a CloudWatch metric so a rising trend is visible before someone in finance notices.

Failure two: rate limits that never send a 429

Your retry logic almost certainly triggers on 429 and 5xx. Several major GraphQL APIs will throttle you without ever sending one.

GitHub’s GraphQL API is explicit about this: exceeding the primary rate limit returns a 200 response with an error message in the body and x-ratelimit-remaining set to zero. Your Step Functions retrier sees a success. Your Lambda sees no data. The run completes, having ingested nothing, and reports green.

The other half of the problem is that request counting is the wrong unit. Shopify’s GraphQL Admin API and several others price queries by calculated cost, using a leaky bucket. The server statically analyses your query before executing it, charges the bucket the worst-case requested cost, executes, then refunds the difference between requested and actual cost. That has a consequence people miss: a query asking for the maximum connection size gets charged as if every page were full, even when the last page returns three records.

The good news is that these APIs tell you the state of the bucket on every response, under the extensions key:

{
  "extensions": {
    "cost": {
      "requestedQueryCost": 72,
      "actualQueryCost": 38,
      "throttleStatus": {
        "maximumAvailable": 1000.0,
        "currentlyAvailable": 962.0,
        "restoreRate": 100.0
      }
    }
  }
}

Read those numbers, never hardcode them. Bucket size and restore rate vary by plan tier, and a query that works fine against a large merchant’s store will fail instantly against a small one. The pattern that holds up is a pre-flight check: before dispatching, compare your estimated cost against currentlyAvailable, and if you’re short, sleep for the shortfall divided by restoreRate.

On APIs that expose their own budget as a queryable field, ask for it in the same round trip rather than spending a second call to find out:

query BudgetCheck {
  rateLimit {
    limit
    cost
    remaining
    resetAt
  }
}

Two more things about concurrency. First, if you fan out across Lambda invocations, each worker’s local view of the bucket is wrong, because the bucket is shared. Multiple workers independently deciding they have headroom is the classic way to get throttled while every individual worker believes it’s behaving. Either centralise the token accounting or cap concurrency low enough that the arithmetic can’t go wrong.

Second, if the source offers a bulk or async export operation, use it for backfills. Paginating fifty thousand records through the interactive endpoint burns the same budget your incremental syncs need, and it competes with itself.

Failure three: cursors that don’t survive a restart

Most production GraphQL schemas follow the Relay cursor connections convention: first and after arguments, a pageInfo object with hasNextPage and endCursor, and edges carrying opaque per-item cursors.

query Orders($cursor: String) {
  orders(first: 100, after: $cursor) {
    pageInfo {
      hasNextPage
      endCursor
    }
    edges {
      cursor
      node {
        id
        updatedAt
      }
    }
  }
}

The filter argument for incremental sync is not standardised. Some schemas take a query-string filter, some take a typed timestamp argument, some expose nothing at all. Run introspection against the schema and find out rather than guessing, because a wrong argument name is a request error and your run will fail before it starts.

The failure here is subtle. Cursors are opaque and their validity is entirely up to the server. Some are stable positional pointers. Some encode a sort key and a snapshot identifier and expire. If a page-47 fetch fails and you retry an hour later with the stored cursor, you may get an error, or you may get results from a shifted window with a hole in the middle. The second outcome is worse, because it looks like it worked.

Three habits make restarts safe:

1. Checkpoint after the write, not after the fetch. A cursor recorded before the page lands in S3 is a promise you can’t keep.
2. Store a high watermark alongside the cursor. The cursor is your fast path. The watermark, taken from the newest record you actually committed, is your fallback when the cursor is rejected.
3. Overlap the watermark deliberately. Restart a few minutes before the last committed timestamp and deduplicate on the primary key downstream. Re-reading a handful of rows is cheap. Missing rows that arrived during a clock skew is not.

A DynamoDB item per source stream is enough state for all of this:

{
  "source": "orders",
  "run_id": "01hq7k...",
  "cursor": "eyJsYXN0X2lkIjoxMjM0fQ==",
  "high_watermark": "<updatedAt of newest committed record>",
  "pages_committed": 47,
  "status": "IN_PROGRESS"
}

Overlap only works if downstream deduplication is real. Write raw pages with an ingest-time partition and a run identifier in the object key, then let the curated layer resolve duplicates by primary key and latest updatedAt. If you write straight into a merged table with no dedupe step, overlapping restarts will double-count and you’ll have traded silent loss for silent inflation.

Failure four: the schema moved and nobody told you

GraphQL’s deprecation story is genuinely better than REST’s. Fields carry a @deprecated directive with a reason string, and introspection exposes it. The problem is that nothing forces you to look.

The failure mode is a field that gets deprecated, stops being populated, then eventually gets removed. Stage one is invisible: your query still validates, the field returns null, and your warehouse fills with nulls. Stage two is loud but late: the query becomes invalid and the whole ingest breaks at 3am.

Put an introspection diff in CI. Fetch the schema on a schedule, store it as an artifact, and diff against the last known good version. Fail the build on removals or type changes to fields your queries actually select, and warn on new deprecations. Tools in the Apollo and Hasura ecosystems ship schema-diff tooling for this, and rolling your own against the introspection result is not much work either. Either way, the point is that the check runs on a schedule rather than when the pager goes off.

The complementary habit is to keep queries narrow. Every field you select is a field that can break you, a field that costs budget, and a field that widens your Glue table. Selecting a whole object because it was convenient during development is a liability you carry indefinitely.

Wiring the GraphQL data ingestion pipeline on AWS

With the failure families understood, the AWS assembly is mostly a set of trade-offs about where the fetch loop lives.

Where the fetch loop runs

Step Functions can call an HTTPS endpoint directly with an HTTP Task, no Lambda involved. It uses an EventBridge connection to hold the credentials, which means your token lives in a Secrets Manager secret managed by EventBridge rather than in an environment variable:

"FetchPage": {
  "Type": "Task",
  "Resource": "arn:aws:states:::http:invoke",
  "Parameters": {
    "ApiEndpoint": "https://api.example.com/graphql",
    "Method": "POST",
    "InvocationConfig": {
      "ConnectionArn": "arn:aws:events:region:account-id:connection/name/id"
    },
    "Headers": { "Content-Type": "application/json" },
    "RequestBody": {
      "query.$": "$.query",
      "variables.$": "$.variables"
    }
  },
  "Retry": [{
    "ErrorEquals": [
      "States.Http.StatusCode.429",
      "States.Http.StatusCode.502",
      "States.Http.StatusCode.503",
      "States.Http.StatusCode.504"
    ],
    "IntervalSeconds": 1,
    "BackoffRate": 2,
    "MaxAttempts": 3,
    "JitterStrategy": "FULL"
  }],
  "Next": "InspectBody"
}

The execution role needs four permissions: states:InvokeHTTPEndpoint, events:RetrieveConnectionCredentials, and both secretsmanager:GetSecretValue and secretsmanager:DescribeSecret scoped to the EventBridge-managed secret.

It’s a clean pattern, and I like it for small, high-value syncs. Two constraints decide whether it fits. HTTP Task requests time out after 60 seconds, which a deeply nested GraphQL query can exceed without being pathological. And Step Functions reads the response within its standard state payload ceiling of roughly 256 KB, so a page of a hundred rich records will not fit. Also note the retrier above: it fires on HTTP status codes, so it will not catch a throttle that arrives as a 200 with an error in the body. That check has to happen in the state after it.

The alternative is a Lambda driver that paginates internally and writes pages straight to S3, returning only the cursor and a count to the state machine. This dodges both limits, gives you a real HTTP client with real backoff, and keeps large payloads out of the workflow’s state. It costs you per-page retry granularity, which you buy back with the checkpoint record.

Distributed Map is worth mentioning for the fan-out case, such as one child execution per tenant or per shard. It gives you high concurrency and a per-item execution history, which is genuinely useful for debugging. Be aware of two things before committing: express child workflows cap at five minutes rather than Lambda’s fifteen, and practitioners have published benchmarks showing throughput well below what the headline concurrency numbers suggest for small, fast items. For a lot of ingestion work, SQS plus a DynamoDB job table remains the simpler answer.

One case genuinely falls outside serverless: sources that require a long-lived session, a persistent websocket subscription, or a synchronisation window longer than fifteen minutes with no resumable cursor. A small always-on worker on a modest VPS from a provider like InterServer or Hetzner, feeding the same S3 landing zone, is less clever and more honest than contorting Lambda around the constraint.

The landing zone

Land raw responses first, unmodified, including the errors and extensions keys. Storage is the cheapest thing in this architecture and the raw layer is what lets you reprocess after you discover a parsing bug six weeks later.

• Partition by ingest time, not event time. Event-time partitioning requires reading the record before you know where to put it, which means you can’t write until you’ve parsed. Ingest-time keys let the raw write be dumb and fast.
• Put the run identifier in the key. When you need to unwind one bad run, being able to delete or ignore by prefix beats writing a surgical delete.
• Define Glue tables explicitly rather than crawling. You wrote the selection set, so you already know the schema. A crawler inferring types from JSON will surprise you the first time an optional field is absent from an entire page.
• Convert to columnar in the curated layer. Athena over raw JSON works and is fine for debugging. Parquet is what you want for anything anyone queries repeatedly.

If your source pushes rather than being polled, such as webhook deliveries carrying GraphQL payloads, Amazon Data Firehose (renamed from Kinesis Data Firehose, with APIs and IAM policies unchanged) will buffer, convert to Parquet and partition on the way to S3 without you running a consumer. For scheduled pull-based syncs it adds a hop you don’t need.

Troubleshooting

• Runs succeed but a column is all null. Field errors. Pull a raw response from the landing zone and look at the errors array. The path tells you exactly which field.
• Runs succeed but ingest zero rows. Likely a throttle returned as a 200. Log the response body and the rate-limit headers on empty results, always.
• Intermittent failures partway through pagination. Cursor expiry. Fall back to the watermark and check whether the source documents a cursor lifetime.
• Query fails immediately with no data key. Request error. Validate against current introspection; something in the schema moved.
• Throttled despite staying under the documented limit. Either concurrent workers are sharing a bucket they each think they own, or you’re being charged requested cost on near-empty pages.
• HTTP Task fails on large pages. The response exceeds the state payload ceiling. Reduce page size or move the fetch into Lambda.
• Row counts drift upward over time. Overlap without deduplication. Check that the curated layer actually resolves on the primary key.

Common mistakes

• Treating a 200 as success and never reading the body.
• Retrying request errors. They are deterministic; retrying just spends budget to fail identically.
• Hardcoding bucket sizes and restore rates that differ per plan tier and change without notice.
• Checkpointing the cursor before the data is durable.
• Requesting the maximum page size on every call while being charged the worst-case cost for it.
• Selecting entire objects because it was easier during development.
• Alerting only on execution failure, so a job that succeeds while ingesting nothing never pages anyone.
• Overlapping restart windows with no downstream deduplication.

Best practices

• Classify every response into clean, partial, or request-failed before anything downstream sees it.
• Persist raw responses with errors and extensions intact, then parse from the raw layer.
• Alarm on row-count deltas and partial-error rate, not just on execution status.
• Read the throttle state the API hands you and back off before you hit the wall, not after.
• Keep both a cursor and a watermark, and know which one is authoritative on restart.
• Diff the schema on a schedule so deprecations reach you before they reach your data.
• Hold credentials in Secrets Manager or an EventBridge connection, never in a Lambda environment variable.
• Trace the pipeline end to end. CloudWatch covers the basics; Grafana or Datadog earn their keep once you’re correlating child executions across a fan-out.

Frequently asked questions

Can Step Functions call a GraphQL API without a Lambda function?

Yes. An HTTP Task using the arn:aws:states:::http:invoke resource posts your query directly, with an EventBridge connection handling authentication. It works well for small pages. The 60-second request timeout and the state payload size ceiling are what push most real ingestion work back into Lambda.

How do I detect GraphQL errors when the status code is always 200?

Inspect the body. If there’s no data key, nothing executed and you have a request error. If data is present alongside a non-empty errors array, you have a partial response and the data is incomplete. Only a response with data and no errors is unambiguously clean.

Does AWS AppSync fit into this?

Only if you’re on the serving side. AppSync is a managed GraphQL API for exposing your own data, so it’s what you’d reach for if you wanted clients to push events in via mutations. For pulling data out of somebody else’s GraphQL API, AppSync isn’t part of the picture.

Should I store raw GraphQL responses or parsed records in S3?

Both, in separate layers. Raw responses are your audit trail and your reprocessing path, and they’re the only way to answer “was this field null in the source, or did my parser drop it?” weeks after the fact. Parsed and columnar output goes in a curated prefix that analysts query.

How do I run an incremental sync from a GraphQL API?

Find the schema’s filter argument for a modification timestamp using introspection, pass your stored watermark into it, and page forward with cursors. Overlap the window slightly on each run and deduplicate on the primary key downstream. If the schema exposes no such filter, you’re limited to full pulls or webhook-driven change capture.

Why am I throttled when I’m making very few requests?

Cost-based APIs charge for query complexity, not call count. A single deeply nested query pulling a connection inside a connection can consume a large fraction of the bucket by itself. Read the cost figures the API returns and flatten the query if the requested cost is consistently much higher than the actual cost.

Is GraphQL a worse ingestion source than REST?

Different, not worse. You control the payload, which cuts transfer and lets you avoid the N+1 fan-out REST forces on nested data. Deprecation metadata is machine-readable, which REST rarely offers. The cost is that error signalling and rate limiting both moved into the body, so naive clients fail silently instead of loudly.

The one thing worth remembering

A GraphQL data ingestion pipeline on AWS fails differently from a REST one, and the difference is almost entirely about where the truth lives. In REST, the transport tells you what happened. In GraphQL, the transport tells you the request arrived, and the body tells you what happened. Every failure family in this post traces back to that: partial errors hidden behind a 200, throttles hidden behind a 200, cursors that fail by returning the wrong rows rather than an error.

So build the pipeline so that a green run means something. Parse before you commit, alarm on row deltas rather than exit codes, and keep the raw responses so you can prove what the source actually said. Everything else is ordinary AWS plumbing.

Need help with a GraphQL ingestion pipeline?

This is the kind of work I take on regularly. Specifically:

• Auditing an existing GraphQL sync for silent partial-error loss and telling you which columns are already affected
• Building the fetch, checkpoint and land loop in Step Functions and Lambda with cursors that survive restarts
• Fitting a cost-aware throttle to whatever budget model your source actually uses, including multi-worker coordination
• Designing the raw and curated S3 layout, Glue table definitions and Athena partitioning so queries stay cheap
• Adding introspection diffing to CI so schema deprecations surface in a pull request, not in a dashboard
• Wiring CloudWatch metrics and alarms on row-count deltas and partial-error rates, so a green run is actually green

Send me a run log, a state machine definition, or a raw response with an errors array in it, and I’ll tell you what it’s costing you.

The post Building a GraphQL Data Ingestion Pipeline on AWS That Doesn’t Lie to You appeared first on John Nessime.

That divergence is not a bug. It is documented behaviour, sitting in one table in the AWS docs that almost everyone scrolls past. This post covers what Zendesk data integration with AWS Glue zero-ETL actually replicates, the entities where deletes silently never propagate, the configuration choices you cannot undo after creation, how to wire the three IAM roles involved, and how to monitor the thing so drift shows up as an alarm instead of a meeting.

What Zendesk AWS Glue zero-ETL actually gives you

Zero-ETL is AWS’s name for managed replication. You create a Glue connection to Zendesk, pick the entities you want, pick a target, and Glue handles the initial snapshot, the schema mapping, and ongoing change data capture. No Spark job. No bookmark logic. No pagination code hammering the Zendesk API and no retry handling for it.

Supported targets are a general purpose Amazon S3 bucket through the lakehouse architecture of Amazon SageMaker, S3 Tables through the same lakehouse, Redshift Managed Storage, or an Amazon Redshift data warehouse directly. Data lands as Apache Iceberg, which is what makes row-level updates and deletes viable on object storage at all.

What you give up is control. No transform step, no server-side filter, no column pruning at ingest. You get the entity as the connector sees it and you shape it downstream. For most support analytics that trade is fine. It stops being fine the moment your compliance team asks why deleted records are still queryable.

The delete gap: four of seven entities never remove a row

AWS Glue zero-ETL supports seven Zendesk entities. Only three of them replicate deletes.

• tickets – create, update and delete all propagate
• users – create, update and delete all propagate
• organizations – create, update and delete all propagate
• satisfaction-rating – create and update only, no delete
• articles – create and update only, no delete
• calls – create and update only, no delete
• legs (call legs) – create and update only, no delete

When a satisfaction rating disappears in Zendesk, the row stays in your lakehouse. Permanently. Nothing errors, IngestionSucceeded fires as usual, and the row count in that table only ever goes up.

Now think about what your CSAT query does. It averages a score column across a set of rows. If removed ratings never leave the target, your average drifts further from the source every cycle, and the skew is rarely random because retracted or disputed ratings are exactly the ones most likely to be removed. The same mechanism inflates knowledge base article counts, call volume, and any call leg analysis you build. It stays invisible until two dashboards get compared side by side.

How to design around it

You cannot make the connector track deletes it does not track. So stop trying, and split the problem instead.

1. Run two integrations, not one. Put tickets, users and organizations in a continuously synced integration. Put the four append-only entities in a second integration you can tear down and rebuild on a cadence that matches how much drift you can tolerate.
2. Treat the append-only tables as an event log, not as state. Query them through a view that reconciles against a periodic authoritative count, rather than trusting row presence to mean the record still exists.
3. Prefer status fields over row existence. Where an entity exposes a field describing its state, filter on that field downstream. Check which fields your entities actually return before you build on this, because coverage varies by entity.
4. Reconcile on a schedule. Pull counts from the Zendesk API for the four entities and compare against the target. Alert on divergence, not on an absolute number.

Splitting the integration costs you a second set of IAM wiring and a second thing to monitor. It buys you the ability to rebuild the drifting half without touching the half that is correct. That is a good trade.

Entity coverage is narrower than the Zendesk API

Seven entities is not the Zendesk API. Ticket comments, ticket metrics, ticket audits, groups, views, macros and SLA policy definitions are not part of the zero-ETL entity set. If your reporting needs first response time, the audit trail behind a status change, or custom field definitions, zero-ETL alone will not get you there.

The useful part is that the Glue connection you create is reusable. The same connection backs a normal Glue ETL job, so you can run zero-ETL for the bulk entities and a scheduled Spark job for the ones it does not cover:

# Read a Zendesk entity through the same connection
# used by the zero-ETL integration. ENTITY_NAME takes
# the entity name, not the label, e.g. "tickets".
zendesk_read = glueContext.create_dynamic_frame.from_options(
    connection_type="Zendesk",
    connection_options={
        "connectionName": "my-zendesk-connection",
        "ENTITY_NAME": "tickets",
        "API_VERSION": "v2"
    }
)

One schema detail worth knowing before you write a single downstream query: the connector converts struct and list types to strings. Zendesk custom fields arrive as arrays of objects, so they land as serialized text, not as a nested column you can address with dot notation. Partitioning is also not supported on the Zendesk source. Plan for a parse step in your silver layer rather than discovering it in a broken dashboard.

Two decisions you cannot change after creation

Most Glue settings are editable. These two are not, and getting them wrong means deleting the integration and starting over.

Continuous sync versus on-demand snapshot

By default the integration syncs continuously. Enable the on-demand snapshot setting and it does a single one-time replication with no ongoing CDC. That setting is locked once the integration exists. If you enabled it for a proof of concept and then decided you wanted CDC, you are recreating the integration and re-running the full load.

Refresh interval, and why your target choice locks it

The refresh interval controls how often CDC pulls run. It accepts anything from 15 minutes up to 8640 minutes, which is six days. Here is the part that catches people:

• If the target is Amazon Redshift, the refresh interval cannot be modified after creation.
• For other targets, including the SageMaker lakehouse on S3, you can change it later.

So the target you pick is not only an architecture decision, it is a flexibility decision. If you are not certain what freshness the business actually needs, land in the lakehouse first and query Redshift over it. You keep the ability to tune the interval once you have real usage data instead of a guess made in week one.

On cost: AWS does not charge separately for the integration itself. You pay for everything around it, target storage, Redshift or Athena query compute, S3 requests, Data Catalog usage and CloudWatch logs. A 15 minute interval on seven entities produces far more small-file churn and log volume than a 6 hour one. Set the interval from the freshness a human will actually act on, not from the smallest number the field accepts.

Wiring the three roles

There are three separate permission layers here and they fail in different ways, which is why “check IAM” is useless advice on its own.

• The Zendesk OAuth credential. The connector uses the authorization code grant, so you get redirected to Zendesk to log in and approve. You can rely on the Glue-managed client application and supply only your instance URL, or register your own OAuth app in the Zendesk admin center and provide your own client ID and secret. The resulting access token does not expire, which is convenient and also means a revoked app in Zendesk is something you will only discover through failed ingestions.
• The source role. This is what lets the integration read through the connection. It is a prerequisite for SaaS sources and it is the step most people miss, because creating the connection successfully does not mean the integration can use it.
• The target role and catalog policy. This is what lets the integration write. A misconfigured catalog resource policy puts the integration into NEEDS_ATTENTION, not into a clear error at creation time.

The source role policy looks like this. Note glue:RefreshOAuth2Tokens, which is the one people leave out and then spend an afternoon debugging:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "GlueConnections",
            "Effect": "Allow",
            "Action": [
                "glue:GetConnections",
                "glue:GetConnection"
            ],
            "Resource": [
                "arn:aws:glue:*:111122223333:catalog",
                "arn:aws:glue:us-east-1:111122223333:connection/*"
            ]
        },
        {
            "Sid": "GlueActionBasedPermissions",
            "Effect": "Allow",
            "Action": [
                "glue:ListEntities",
                "glue:RefreshOAuth2Tokens"
            ],
            "Resource": ["*"]
        },
        {
            "Sid": "CloudWatchLogging",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": ["*"]
        }
    ]
}

The trust policy allows glue.amazonaws.com to assume it. Then you attach the role to the connection, which is a CLI-only step and does not appear in the console flow:

aws glue create-integration-resource-property 
  --resource-arn arn:aws:glue:us-east-1:123456789012:connection/my-zendesk-connection 
  --source-processing-properties "{"RoleArn" : "arn:aws:iam::123456789012:role/zendesk-zetl-source"}" 
  --region us-east-1

If you skip this, the connection tests fine and the integration still cannot read. That mismatch is the single most common reason a first attempt sits in NEEDS_ATTENTION with no obvious cause.

Setting up the integration

1. If you are using your own OAuth app, register it in the Zendesk admin center under the API settings and note the client ID and secret. Store the secret in AWS Secrets Manager, one secret per Glue connection.
2. In Glue Studio, create a connection under Data Connections. Choose Zendesk as the connection type, supply your instance URL and environment, and complete the OAuth redirect.
3. Create the source role with the policy above and attach it to the connection using create-integration-resource-property.
4. Prepare the target: the Glue database or S3 Table bucket, the target role, the catalog resource policy, and any Lake Formation grants your account’s permission model requires.
5. Create the integration. Select entities, set the refresh interval, and decide continuous sync versus on-demand snapshot. Both of those are locked from here.
6. Watch the first full load complete before you point anything at the tables. Full load and CDC are separate load types in the metrics, and the numbers look very different.

Once it works, move it into code. Glue zero-ETL integrations are supported by CloudFormation and the AWS CDK, which is the difference between a thing one person built in the console and a thing your team can redeploy into another account.

Monitoring: the metrics that expose drift

Zero-ETL publishes to the AWS/Glue/ZeroETL CloudWatch namespace, with dimensions for integrationArn, loadType and tableName. The metrics are InsertCount, UpdateCount, DeleteCount, IngestionSucceeded, IngestionFailed and LastSyncTimestamp.

The obvious alarm is on failure:

aws cloudwatch put-metric-alarm 
  --alarm-name zendesk-zetl-ingestion-failed 
  --namespace AWS/Glue/ZeroETL 
  --metric-name IngestionFailed 
  --dimensions Name=integrationArn,Value=<your-integration-arn> 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 1 
  --threshold 1 
  --comparison-operator GreaterThanOrEqualToThreshold 
  --treat-missing-data notBreaching

That one catches loud failures. It does not catch the quiet one, which is an integration that stops running entirely. For that, alarm on IngestionSucceeded with --treat-missing-data breaching over a window longer than your refresh interval. Absence of success is the signal, not presence of failure.

And here is the one specific to this post. Because DeleteCount carries a tableName dimension, you can see the delete gap directly in a graph. Plot DeleteCount per table. Tickets, users and organizations will show a nonzero line. Satisfaction ratings, articles, calls and legs will sit flat at zero forever. That flat line is not a broken metric, it is the behaviour, and having it on a dashboard is the cheapest way to keep the whole team aware of it.

Glue also writes a system table into the target database recording the outcome of each full load and CDC run, with per-run record, insert and delete counts. Query that when CloudWatch is not granular enough. If you already run Grafana Cloud or Datadog, pulling the AWS/Glue/ZeroETL namespace in alongside your existing infrastructure dashboards puts pipeline health next to everything else on call already watches.

Troubleshooting

Integrations move through Creating, Active, Modifying, Syncing, Needs attention, Failed and Deleting. What each one means in practice:

• Syncing means it hit a recoverable error and is re-seeding data. Not an emergency. Let it finish before you touch anything.
• Needs attention means you have to fix something, usually connection credentials, the source role, the target role, or the catalog resource policy. Once you fix it, there is no manual recovery action. Glue retries automatically on an exponential backoff schedule, so the gap between attempts grows over time. If you fixed a policy and nothing happened after two minutes, that is expected. Wait it out.
• Failed is terminal. Delete the integration and recreate it. There is no repair path.

For anything else, the CloudWatch logs emitted after each full load or CDC run carry the actual root cause. The console status is a summary, and a fairly lossy one. Read the logs first.

Common mistakes

• Building CSAT, article count or call volume metrics on raw row counts from the four entities that never delete.
• Creating the connection and assuming the integration can use it. The source role attachment is a separate CLI step.
• Choosing Redshift as the direct target during a proof of concept and locking a refresh interval you picked arbitrarily.
• Setting a 15 minute interval because it was available, then being surprised by small-file churn and CloudWatch log volume.
• Writing downstream queries that address Zendesk custom fields as nested columns. They arrive as strings.
• Alarming only on IngestionFailed, so a stalled integration produces silence and silence looks like health.
• Reaching for zero-ETL when the requirement is ticket comments or ticket metrics, which are not in the entity set at all.

Best practices

• Split delete-tracking and append-only entities into separate integrations so you can rebuild one without disturbing the other.
• Land in the SageMaker lakehouse rather than straight into Redshift unless you are certain about freshness. It keeps the refresh interval editable.
• Put DeleteCount per table on a dashboard. It makes an invisible behaviour visible to everyone, not just whoever read the docs.
• Define the integration in CloudFormation or CDK once it works, so the next environment is a deploy rather than a repeat of the console clicking.
• Keep a silver layer between the replicated tables and your BI tool. That is where you parse the stringified structs and apply the reconciliation logic, and it means schema surprises break one view instead of every dashboard.
• Document which entities do not track deletes somewhere your analysts will actually read, next to the tables, not in a wiki page nobody opens.

When to use something else

Zero-ETL is a good fit when your target is already AWS, your entities are in the supported set, and you want AWS to own the pipeline. It is the wrong fit if you need entities outside that list, need transformation at ingest, or need the same Zendesk data landing somewhere that is not AWS. Managed connector platforms like Fivetran or Airbyte cover a wider slice of the Zendesk API and will replicate to non-AWS destinations, at the cost of another vendor relationship, another bill, and data leaving your account boundary on the way through. If the seven entities cover your reporting, zero-ETL is cheaper and simpler. If they do not, forcing it is worse than paying for a tool that fits.

Frequently asked questions

Which Zendesk entities does AWS Glue zero-ETL support?

Tickets, users, organizations, satisfaction ratings, articles, calls and call legs. Tickets, users and organizations replicate creates, updates and deletes. The other four replicate creates and updates only.

Can I get ticket comments or ticket metrics through zero-ETL?

No. They are not in the entity set. Use a Glue ETL job against the same Zendesk connection, or a connector platform with broader API coverage, and join the result to your replicated tables downstream.

How fresh can the data be?

The refresh interval goes down to 15 minutes and up to six days. This is near real time replication, not streaming. If you need sub-minute latency on Zendesk events, zero-ETL is the wrong mechanism and you want webhooks into an event pipeline instead.

Can I change the refresh interval later?

Only if the target is not Redshift. With Redshift as the target the interval is fixed at creation. With the SageMaker lakehouse and other targets you can modify it afterwards.

Why is my integration stuck in NEEDS_ATTENTION after I fixed the permissions?

Because recovery is automatic but not immediate. Glue retries on exponential backoff, so if the integration has been unhealthy for a while the next retry may be some way off. There is no manual recovery command. Confirm the fix is correct, then wait.

Does zero-ETL cost extra on top of Glue?

AWS does not bill a separate charge for the integration itself. You pay for the surrounding services: target storage, Redshift or Athena query compute, S3 requests, Data Catalog usage and CloudWatch logs. Refresh interval is the main lever, since it drives how many small commits and log entries you generate.

Why do my Zendesk custom fields look like text?

The connector converts struct and list types to strings. Custom fields come through as serialized values rather than nested columns, so parse them in a downstream layer instead of querying them directly.

Conclusion

Zendesk AWS Glue zero-ETL removes a genuinely tedious pipeline from your plate. The setup is fiddly in the IAM layer and then it mostly runs itself, which is exactly the point of a managed integration.

The one thing worth carrying out of this post: four of the seven supported entities never replicate a delete. Satisfaction ratings, articles, calls and call legs only grow. Nothing warns you, every metric stays green, and your numbers drift a little further from Zendesk with every cycle. Design for that on day one, put DeleteCount per table on a dashboard, and you will not be the person explaining the discrepancy in a meeting six months from now.

Need help with your Zendesk to AWS data pipeline?

I work with teams building and fixing replication pipelines on AWS. On this kind of setup, that usually means:

• Auditing an existing Zendesk zero-ETL integration and quantifying how far the target has drifted from the source
• Designing the split between continuously synced and append-only entities, including the rebuild cadence and reconciliation queries
• Untangling the source role, target role and catalog policy layers when an integration sits in NEEDS_ATTENTION with no clear cause
• Building the silver layer that parses stringified Zendesk custom fields into something your BI tool can actually use
• Setting up CloudWatch alarms and dashboards that catch a stalled integration, not just a failed one
• Moving a console-built integration into CloudFormation or CDK so it can be deployed across accounts

Send me the integration status, a CloudWatch log excerpt, or the query that is giving you the wrong number, and I will tell you what I think is going on.

The post Zendesk Data Integration with AWS Glue Zero-ETL: The Delete Gap That Skews Your Numbers appeared first on John Nessime.