<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Zoho | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/zoho/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/zoho/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Tue, 06 Oct 2026 06:08:51 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Zoho | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/zoho/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Monitoring Zoho Integrations: Catching the Failures That Never Throw an Error</title>
		<link>https://john-nessime.com/blog/devops/monitoring-zoho-integrations/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 06 Oct 2026 06:08:51 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Workflow Automation]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Cardinality]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Deadman Alerting]]></category>
		<category><![CDATA[Embedded Metric Format]]></category>
		<category><![CDATA[Grafana]]></category>
		<category><![CDATA[Grafana Alloy]]></category>
		<category><![CDATA[LogQL]]></category>
		<category><![CDATA[Loki]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[Webhooks]]></category>
		<category><![CDATA[Zoho]]></category>
		<category><![CDATA[Zoho CRM]]></category>
		<category><![CDATA[Zoho Flow]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=395</guid>

					<description><![CDATA[<p>Zoho integrations rarely fail loudly. Channels expire after an hour, credits drain against a rolling window, and bulk writes return HTTP 200 while rejecting records. Here is how to instrument for absence using CloudWatch Embedded Metric Format, Loki's absent_over_time, and Grafana across both.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/monitoring-zoho-integrations/">Monitoring Zoho Integrations: Catching the Failures That Never Throw an Error</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message came in on a Tuesday: &#8220;Have we stopped getting leads from the website form?&#8221; Not an alert. Not a page. A salesperson noticing the pipeline looked thin.</p>



<p class="wp-block-paragraph">The sync worker was healthy. Zero errors in the last twenty-four hours. CPU flat, memory flat, no restarts, no failed jobs. Every dashboard was green because every dashboard was measuring the wrong thing. The integration had stopped receiving anything from Zoho four days earlier, and an integration that receives nothing produces no errors at all.</p>



<p class="wp-block-paragraph">That is the whole problem with monitoring Zoho integrations. The failures that hurt do not throw exceptions. They show up as silence, as slowly degrading throughput, or as HTTP 200 responses that quietly did not do what you asked. This post covers how to instrument for those three failure families using CloudWatch, Grafana and Loki, and why the obvious metrics miss all three.</p>



<h2 class="wp-block-heading">The three ways Zoho integrations actually break</h2>



<p class="wp-block-paragraph">Before touching a dashboard, it helps to be honest about what you are defending against. In my experience these fall into three groups, and they need genuinely different instrumentation:</p>



<ol class="wp-block-list">
<li><strong>Silent stop.</strong> Zoho stops sending you events. Nothing fails, because nothing happens.</li>

<li><strong>Silent throttle.</strong> You run out of API credits or hit a concurrency ceiling. Throughput degrades before anything hard-fails.</li>

<li><strong>Silent drift.</strong> Calls succeed, records do not land, and the two systems diverge without a single error log.</li>
</ol>



<p class="wp-block-paragraph">An error-rate panel catches none of these. That is not a criticism of error-rate panels, it is just what they are for.</p>



<h2 class="wp-block-heading">Failure family one: the integration that just stops</h2>



<p class="wp-block-paragraph">If you push data out of Zoho CRM using the Notifications API, you subscribe a <code>notify_url</code> to module events and Zoho posts to it when records change. It works well. It also has a detail that catches people out repeatedly.</p>



<p class="wp-block-paragraph">Notification channels expire. Zoho&#8217;s documentation is explicit: the expiry can be set to a maximum of one week from the time the channel is enabled, and if you do not specify one, or you ask for longer than a week, the default expiry is <strong>one hour</strong>. So the failure mode is not &#8220;someone forgot to renew after a year.&#8221; It is &#8220;someone deployed a subscription script without an expiry field and the integration worked beautifully for sixty minutes.&#8221;</p>



<p class="wp-block-paragraph">What makes this genuinely hard rather than just annoying is that there is no endpoint to list your currently active channels. You cannot ask Zoho what it thinks it is subscribed to. You can only ask whether events are still arriving. That single fact should shape your entire monitoring design: you are not monitoring for errors, you are monitoring for <em>absence</em>.</p>



<h3 class="wp-block-heading">Emit a heartbeat, then alarm on its absence</h3>



<p class="wp-block-paragraph">The receiving end of the webhook should emit a metric on every delivery, including deliveries you decide to discard. Embedded Metric Format is the cheapest way to do this from Lambda or a container, because you write structured JSON to stdout and CloudWatch extracts the metrics asynchronously. No <code>PutMetricData</code> call, no synchronous latency in your request path.</p>



<pre class="wp-block-code"><code>{
  "_aws": {
    "Timestamp": 1710000000000,
    "CloudWatchMetrics": [
      {
        "Namespace": "ZohoIntegration",
        "Dimensions": [["Module", "Channel"]],
        "Metrics": [
          { "Name": "NotificationsReceived", "Unit": "Count" },
          { "Name": "ChannelSecondsRemaining", "Unit": "Seconds" }
        ]
      }
    ]
  },
  "Module": "Leads",
  "Channel": "leads-prod",
  "NotificationsReceived": 1,
  "ChannelSecondsRemaining": 486000,
  "record_id": "5725767000002166520",
  "operation": "insert"
}</code></pre>



<p class="wp-block-paragraph">Note what is a dimension and what is not. <code>Module</code> and <code>Channel</code> are bounded sets, so they are safe dimensions. <code>record_id</code> sits in the log body only. EMF creates a separate custom metric for every unique dimension combination, so promoting a record ID to a dimension is how you turn a monitoring project into a billing incident. AWS says this plainly in the EMF documentation and it is worth reading twice before you ship.</p>



<p class="wp-block-paragraph">Now the alarm. A normal CloudWatch alarm goes to <code>INSUFFICIENT_DATA</code> when the metric stops reporting, and <code>INSUFFICIENT_DATA</code> does not notify. That is precisely backwards for this failure. You want missing data treated as a breach:</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-alarm 
  --alarm-name zoho-leads-notifications-silent 
  --namespace ZohoIntegration 
  --metric-name NotificationsReceived 
  --dimensions Name=Module,Value=Leads Name=Channel,Value=leads-prod 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 2 
  --threshold 1 
  --comparison-operator LessThanThreshold 
  --treat-missing-data breaching 
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:ops-alerts</code></pre>



<p class="wp-block-paragraph">The <code>--treat-missing-data breaching</code> flag is the entire point of that command. Without it, an integration that goes completely dead produces an alarm that sits quietly in an unknown state forever.</p>



<p class="wp-block-paragraph">On the Loki side the equivalent is <code>absent_over_time</code>, which returns 1 when a stream has produced no entries in the window. Run it from the Loki ruler so it fires even when nobody has a browser open:</p>



<pre class="wp-block-code"><code>groups:
  - name: zoho-integration
    interval: 5m
    rules:
      - alert: ZohoNotificationsSilent
        expr: absent_over_time({job="zoho-webhook", module="leads"}[2h])
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: "No Zoho notifications for Leads in 2h"
          description: "Channel may have expired. Re-subscribe and check channel_expiry."</code></pre>



<p class="wp-block-paragraph">One caveat worth knowing: <code>absent_over_time</code> only works for label combinations you name explicitly. It cannot tell you that a stream you never thought about has gone quiet. That is a real limitation, and it means your channel inventory has to live in your own configuration, not be discovered from Loki.</p>



<p class="wp-block-paragraph">Because you cannot list channels through the API, treat renewal as a scheduled job with its own deadman alert, and emit <code>ChannelSecondsRemaining</code> at each renewal. Alarm when it drops below roughly a day. That gives you warning before the silence starts rather than after somebody notices.</p>



<h2 class="wp-block-heading">Failure family two: throttling that is not rate limiting</h2>



<p class="wp-block-paragraph">Most engineers instrument Zoho as if it were a requests-per-minute API. It is not, and that mismatch produces dashboards that look fine right up until calls start failing.</p>



<p class="wp-block-paragraph">Zoho CRM meters API usage in <strong>credits</strong>, not calls. Different operations deduct different amounts, and bulk operations are metered per batch of records rather than per request. The budget resets on a rolling twenty-four hour window measured from each call, not at midnight. If your dashboard has a daily reset at 00:00, it is drawing a line that does not exist.</p>



<h3 class="wp-block-heading">The gauge that appears halfway through the tank</h3>



<p class="wp-block-paragraph">Here is the detail that makes credit monitoring awkward. Zoho returns remaining credits in an <code>X-API-CREDITS-REMAINING</code> response header, but only once usage has reached fifty percent or more of the daily allowance. Below that threshold the header simply is not there.</p>



<p class="wp-block-paragraph">Read that again in monitoring terms. An absent header means either &#8220;you are comfortably under half your budget&#8221; or &#8220;your instrumentation is broken and you have no idea.&#8221; Those two states look identical, and one of them is fine while the other is a blind spot. So record both facts separately:</p>



<ul class="wp-block-list">
<li>A counter for responses where the header was <em>present</em>, which effectively tells you when you crossed the halfway mark.</li>

<li>A gauge of the value itself, which only has meaning while the header exists.</li>

<li>Your own client-side count of calls made, which works regardless and is the only thing you control.</li>
</ul>



<p class="wp-block-paragraph">The first crossing of that threshold is a useful early warning in its own right. Alert on the transition, not just on a low remaining value.</p>



<h3 class="wp-block-heading">Concurrency is a separate ceiling</h3>



<p class="wp-block-paragraph">Alongside credits, Zoho applies a concurrency limit: the number of calls active simultaneously per org. There is also a sub-concurrency limit covering the expensive operations such as COQL queries, composite requests, lead conversion, and inserts or updates above a small record count. Exceeding either returns <code>TOO_MANY_REQUESTS</code>.</p>



<p class="wp-block-paragraph">Two consequences matter for monitoring. First, since the limit is on simultaneous calls rather than calls per unit time, a requests-per-minute graph will never predict a concurrency failure. You need a gauge of in-flight requests, incremented before the call and decremented in a <code>finally</code> block. Second, the sub-concurrency ceiling is the same across every CRM edition, so upgrading the plan raises the general limit but does not buy headroom on the expensive calls. Batch those and queue them rather than fanning out.</p>



<p class="wp-block-paragraph">One more source of consumption that surprises people: integration tasks inside Deluge functions consume credits from the same org budget as your external API calls. If your credit graph climbs while your service is idle, someone&#8217;s workflow rule is spending your budget. Zoho&#8217;s API Dashboard breaks consumption down by application and function, which is where to look before blaming your own code.</p>



<h2 class="wp-block-heading">Failure family three: successful calls that did nothing</h2>



<p class="wp-block-paragraph">Bulk and multi-record operations return a per-record status inside the response body. The HTTP status can be perfectly healthy while individual records were rejected for validation rules, mandatory fields, duplicate checks or picklist values that do not exist in the target layout.</p>



<p class="wp-block-paragraph">If your client only checks the HTTP status, you will log a success, increment a success counter, and move on. Weeks later somebody asks why the CRM is missing a slice of records that your logs insist were written.</p>



<p class="wp-block-paragraph">The fix is to stop counting requests and start counting records, on both sides:</p>



<ul class="wp-block-list">
<li><strong>Records submitted</strong> versus <strong>records confirmed</strong> in the per-record response status.</li>

<li>A breakdown of rejection reasons as a low-cardinality label, so a spike in one specific validation error is visible.</li>

<li>A periodic reconciliation job comparing source counts against Zoho counts for a bounded window, which is the only thing that catches drift you failed to anticipate.</li>
</ul>



<p class="wp-block-paragraph">In LogQL, the divergence between the two counters is the panel that actually earns its place on the wall:</p>



<pre class="wp-block-code"><code>sum(count_over_time({job="zoho-sync"} | json | status = "submitted" [1h]))
  -
sum(count_over_time({job="zoho-sync"} | json | status = "confirmed" [1h]))</code></pre>



<p class="wp-block-paragraph">A healthy integration holds that near zero. A number that grows steadily is drift, and it is visible weeks before anyone in sales notices.</p>



<h2 class="wp-block-heading">Deciding which signal goes where</h2>



<p class="wp-block-paragraph">Running CloudWatch, Grafana and Loki together only helps if each one has a job. The split I reach for first:</p>



<ul class="wp-block-list">
<li><strong>CloudWatch</strong> holds the numbers you alarm on, especially where the workload already runs in AWS and EMF gives you metrics for free out of log output.</li>

<li><strong>Loki</strong> holds the full request and response context: which record, which module, which validation error, which channel. This is what you actually read at three in the morning.</li>

<li><strong>Grafana</strong> is the single pane over both, with CloudWatch and Loki added as separate data sources so one dashboard can show the credit gauge next to the rejection reasons.</li>
</ul>



<p class="wp-block-paragraph">The trap that spans both backends is cardinality, and it bites in the same way for different reasons. In CloudWatch, each unique dimension combination becomes a billable custom metric. In Loki, each unique label combination becomes a separate stream, and Loki was designed for a small number of long-lived streams. Record IDs, email addresses and timestamps belong in neither.</p>



<p class="wp-block-paragraph">Loki&#8217;s answer is structured metadata: high-cardinality fields attached to log lines without creating streams or entering the index. It needs enabling per tenant:</p>



<pre class="wp-block-code"><code>limits_config:
  allow_structured_metadata: true
  volume_enabled: true</code></pre>



<p class="wp-block-paragraph">Put <code>module</code>, <code>environment</code> and <code>job</code> in labels. Put the Zoho record ID and the correlation ID in structured metadata. Query with a label selector first and filter on the rest.</p>



<p class="wp-block-paragraph">On collection, one scheduling note that does age: Promtail reached end of life on 2 March 2026, with its collection capabilities folded into Grafana Alloy. If you are still shipping through Promtail, that migration is now maintenance debt rather than an optional upgrade, and Grafana published a conversion tool for existing configurations. New pipelines should start on Alloy.</p>



<p class="wp-block-paragraph">Where you run all this is a real decision. Self-hosting Loki on a VPS from a provider like Contabo or InterServer is cheap and entirely workable for a single integration&#8217;s log volume, and it keeps the data under your control. Grafana Cloud or Amazon Managed Grafana removes the operational surface if you would rather not babysit an ingester. Both are defensible. The honest trade-off is that self-hosted Loki is easy to stand up and less easy to keep healthy once retention and compaction start to matter.</p>



<h2 class="wp-block-heading">Troubleshooting the monitoring itself</h2>



<ul class="wp-block-list">
<li><strong>EMF metrics never appear.</strong> The JSON must be a single line and structurally valid. A pretty-printed or truncated log line is ingested as an ordinary log event with no error, so check the raw event in the log stream rather than the metrics console.</li>

<li><strong>The alarm never fires despite obvious silence.</strong> Look at the alarm state. Sitting in <code>INSUFFICIENT_DATA</code> means the missing-data treatment was left at its default.</li>

<li><strong><code>absent_over_time</code> returns nothing.</strong> That is the success case. It emits a value only when the stream is empty, which makes it feel broken while everything is working.</li>

<li><strong>Loki queries slow to a crawl.</strong> Check stream count before blaming resources. A label added &#8220;temporarily&#8221; during debugging is the usual cause.</li>

<li><strong>Webhook deliveries stop after a deployment.</strong> Check whether the subscription script reran and created a channel without an expiry. An hour later it will look identical to a network problem.</li>

<li><strong>Credit consumption does not match your call count.</strong> Expected. Operations deduct different credit amounts, and Deluge integration tasks draw from the same pool.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Alerting on error rate alone, which is blind to every failure described above.</li>

<li>Creating notification channels without an explicit <code>channel_expiry</code> and inheriting the one-hour default.</li>

<li>Assuming an absent <code>X-API-CREDITS-REMAINING</code> header means healthy usage rather than sub-threshold usage.</li>

<li>Building a daily credit dashboard when the limit is a rolling window.</li>

<li>Putting record IDs into EMF dimensions or Loki labels.</li>

<li>Treating an HTTP 200 on a bulk write as confirmation that every record landed.</li>

<li>Running alert rules only in the Grafana UI, so nothing evaluates if that instance is down.</li>
</ul>



<h2 class="wp-block-heading">Best practices for monitoring Zoho integrations</h2>



<ol class="wp-block-list">
<li>Start with a deadman alert per channel. If you build one thing, build this.</li>

<li>Set <code>channel_expiry</code> explicitly, renew on a schedule, and monitor the renewal job as carefully as the integration.</li>

<li>Count records, not requests, on both sides of every write.</li>

<li>Keep a client-side call counter so you are never dependent on a header that may not be sent.</li>

<li>Gauge in-flight requests to see concurrency pressure before it becomes <code>TOO_MANY_REQUESTS</code>.</li>

<li>Attach a correlation ID to every record as structured metadata so one identifier joins logs and metrics.</li>

<li>Evaluate critical rules server-side, in the Loki ruler or as CloudWatch alarms.</li>

<li>Test by disabling a channel in a sandbox org and confirming the alert fires. An untested deadman alert is decoration.</li>
</ol>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Why did my Zoho webhook stop working on its own?</h3>



<p class="wp-block-paragraph">Almost always channel expiry. Zoho CRM notification channels last a maximum of one week, and if the expiry is omitted or set beyond that maximum, the channel defaults to one hour. Nothing is logged on your side because the deliveries simply stop arriving.</p>



<h3 class="wp-block-heading">Can I list my active Zoho notification channels?</h3>



<p class="wp-block-paragraph">There is no endpoint for enumerating active channels, which is a long-standing complaint in Zoho&#8217;s developer community. Keep the inventory in your own configuration and treat the receipt of events as the only reliable liveness signal.</p>



<h3 class="wp-block-heading">Do I need both CloudWatch and Loki?</h3>



<p class="wp-block-paragraph">No. Either can carry the whole job. The pairing makes sense when your integration already runs on AWS, because EMF gives you alarmable metrics from log output at no extra instrumentation cost, while Loki holds richer request context more cheaply. If you are starting fresh outside AWS, one backend is simpler and simpler is worth a lot.</p>



<h3 class="wp-block-heading">How do I monitor Zoho API credits properly?</h3>



<p class="wp-block-paragraph">Record the <code>X-API-CREDITS-REMAINING</code> header when it appears, alert on the moment it first appears in a window, and maintain your own call counter as the fallback. Model the budget as a rolling twenty-four hour window rather than a daily reset.</p>



<h3 class="wp-block-heading">What causes TOO_MANY_REQUESTS if I am under my credit limit?</h3>



<p class="wp-block-paragraph">Concurrency, not consumption. Credits and concurrency are independent ceilings. Expensive operations such as COQL queries and composite requests share a tighter sub-concurrency pool, so a burst of parallel queries can trip it while your credit balance is untouched.</p>



<h3 class="wp-block-heading">Should I still use Promtail for shipping integration logs?</h3>



<p class="wp-block-paragraph">No. Promtail reached end of life on 2 March 2026 and its functionality moved into Grafana Alloy. Existing deployments will keep running, but they are unsupported, and Grafana provides a tool to convert Promtail configuration to Alloy.</p>



<h3 class="wp-block-heading">How long should the deadman alert window be?</h3>



<p class="wp-block-paragraph">Base it on the quietest legitimate period for that module. A leads webhook that goes quiet overnight and at weekends needs a longer window than an order sync, or you will train the team to ignore it. Measure the real gap distribution for a fortnight before picking a number.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Monitoring Zoho integrations is mostly the discipline of detecting absence. Expired channels, exhausted credits and silently rejected records all present as nothing happening, and nothing happening is invisible to the instrumentation most teams reach for first.</p>



<p class="wp-block-paragraph">So invert the default. Alarm when the count is too low rather than when the errors are too high. Set <code>--treat-missing-data breaching</code>. Use <code>absent_over_time</code>. Count records on both sides and watch the gap. If you build only one thing from this post, build the deadman alert on notification delivery, because that is the failure that will otherwise be reported to you by a salesperson wondering where the leads went.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help instrumenting a Zoho integration?</h2>



<p class="wp-block-paragraph">Most of the work here is judgement about what to measure rather than volume of code. If you are staring at a sync that keeps going quiet, this is the sort of thing I take on:</p>



<ul class="wp-block-list">
<li>Building deadman alerting for Zoho notification channels, including renewal jobs that monitor themselves.</li>

<li>Instrumenting credit and concurrency headroom so throttling is visible before it becomes <code>TOO_MANY_REQUESTS</code>.</li>

<li>Designing EMF dimensions and Loki label schemas that will not turn into a cardinality bill.</li>

<li>Adding reconciliation jobs that compare record counts across systems and surface drift early.</li>

<li>Standing up Loki and Grafana Alloy, or migrating an existing Promtail pipeline that is now unsupported.</li>

<li>Reviewing an existing Grafana dashboard and cutting the panels that look reassuring but detect nothing.</li>
</ul>



<p class="wp-block-paragraph">Send me a sample webhook payload, a redacted response header set, or a screenshot of the dashboard you currently trust, and I will tell you what it would miss.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/monitoring-zoho-integrations/">Monitoring Zoho Integrations: Catching the Failures That Never Throw an Error</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Zoho Mail Not Receiving Email? A DNS Troubleshooting Guide</title>
		<link>https://john-nessime.com/blog/technical-guides/zoho-mail-not-receiving-email/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 31 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Networking]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[Cloudflare]]></category>
		<category><![CDATA[dig]]></category>
		<category><![CDATA[DKIM]]></category>
		<category><![CDATA[DMARC]]></category>
		<category><![CDATA[DNS]]></category>
		<category><![CDATA[DNS Cache]]></category>
		<category><![CDATA[DNS Records]]></category>
		<category><![CDATA[DNS Troubleshooting]]></category>
		<category><![CDATA[DNSSEC]]></category>
		<category><![CDATA[Domain Configuration]]></category>
		<category><![CDATA[Email Authentication]]></category>
		<category><![CDATA[Email Deliverability]]></category>
		<category><![CDATA[MX Records]]></category>
		<category><![CDATA[Nameservers]]></category>
		<category><![CDATA[SERVFAIL]]></category>
		<category><![CDATA[SMTP]]></category>
		<category><![CDATA[SPF]]></category>
		<category><![CDATA[TTL]]></category>
		<category><![CDATA[Zoho]]></category>
		<category><![CDATA[Zoho Mail]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=344</guid>

					<description><![CDATA[<p>Sending works, nothing comes in, and no bounce shows up anywhere. That asymmetry tells you almost everything: inbound mail is decided by MX records, not by SPF. This guide walks the inbound path in the order a sending server actually walks it, groups the real causes into five DNS failure families, and gives you the dig commands to prove which one you are looking at.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/zoho-mail-not-receiving-email/">Zoho Mail Not Receiving Email? A DNS Troubleshooting Guide</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket almost always reads the same way. &#8220;We can send email fine, but nothing is coming in.&#8221; Someone has been waiting two days on a signed contract that never arrived. No bounce landed in anyone&#8217;s inbox. No error appeared anywhere in the Zoho interface. The mail just went somewhere else.</p>



<p class="wp-block-paragraph">That asymmetry, sending works and receiving doesn&#8217;t, is the single most useful clue in the whole problem. It narrows the search enormously. Yet most guides on Zoho Mail not receiving email open by telling you to check your SPF record, which has nothing whatsoever to do with inbound delivery. People lose a full day tuning an SPF string while their MX records quietly point at a data center that has never heard of their domain.</p>



<p class="wp-block-paragraph">This guide walks the inbound path in the order a sending mail server actually walks it, groups the real causes by failure family, and gives you the exact commands to prove which one you&#8217;re looking at. Everything here applies whether your DNS lives at Cloudflare, Namecheap, GoDaddy, Route 53, or a cPanel box at InterServer or Contabo.</p>



<h2 class="wp-block-heading">Sending and receiving are two separate systems</h2>



<p class="wp-block-paragraph">This is worth internalizing before you touch a single DNS record, because it eliminates most of the internet&#8217;s advice on this topic in one move.</p>



<p class="wp-block-paragraph">When you send from Zoho, your client authenticates against Zoho&#8217;s outbound SMTP server with a username and password. Your domain&#8217;s DNS plays no part in that handshake. SPF, DKIM and DMARC records exist so that the <em>recipient&#8217;s</em> server can decide whether to trust mail claiming to be from you. They are judgements other people make about your outbound mail.</p>



<p class="wp-block-paragraph">When someone sends <em>to</em> you, none of that matters. Their server looks up your MX records, connects to whatever hostname it finds, and hands the message over. Your SPF record is never consulted. Your DKIM key is never consulted.</p>



<p class="wp-block-paragraph">So if outbound works and inbound doesn&#8217;t, the fault is in the MX lookup, the connection that follows it, or Zoho&#8217;s decision about what to do with the message once it arrives. Not SPF. There is one narrow exception, covered later, where Zoho&#8217;s own inbound spam checks reject mail because the <em>sender&#8217;s</em> SPF fails. That is a different record on a different domain.</p>



<h2 class="wp-block-heading">The inbound path, in the order it actually happens</h2>



<p class="wp-block-paragraph">Six things happen between someone hitting send and a message appearing in your Zoho inbox:</p>



<ol class="wp-block-list">
<li>The sending server splits the recipient address and takes the domain part.</li>



<li>It asks DNS for the MX records of that domain, following delegation from the root down to whichever nameservers the registry says are authoritative for you.</li>



<li>It sorts the results by preference number and takes the lowest one first.</li>



<li>It resolves that hostname to an address record.</li>



<li>It opens an SMTP connection and offers the message with MAIL FROM and RCPT TO.</li>



<li>The receiving server decides whether that recipient exists in its organization and where to file the message.</li>
</ol>



<p class="wp-block-paragraph">Steps two through four are pure DNS. Step five is network. Step six is Zoho configuration. Diagnosing this well is mostly a matter of finding out which step you&#8217;re stuck on instead of guessing.</p>



<h2 class="wp-block-heading">Five DNS failure families behind Zoho Mail not receiving email</h2>



<p class="wp-block-paragraph">Nearly every inbound failure I&#8217;ve seen falls into one of these. They present almost identically from the user&#8217;s chair, which is why guessing goes badly.</p>



<h3 class="wp-block-heading">1. You&#8217;re editing records at a provider that isn&#8217;t authoritative</h3>



<p class="wp-block-paragraph">This is the most common one and the most frustrating, because the records look perfect in the control panel you&#8217;re staring at. They&#8217;re just not the records the internet reads.</p>



<p class="wp-block-paragraph">It happens whenever the domain is registered in one place and the nameservers point somewhere else. You buy at Namecheap, delegate to Cloudflare, then add MX records back in the Namecheap DNS panel out of habit. Or a previous developer moved DNS to a hosting account you no longer log into. The registrar panel accepts your edits and stores them faithfully. Nobody queries them.</p>



<p class="wp-block-paragraph">Find out where authority actually sits:</p>



<pre class="wp-block-code"><code># Which nameservers does the parent zone delegate to?
dig +short NS example.com

# What do those nameservers say about your MX, directly?
dig MX example.com @ns1.yourprovider.com +norecurse

# Watch the full delegation chain from the root
dig +trace MX example.com</code></pre>



<p class="wp-block-paragraph">The <code>+trace</code> output is the one to read carefully. It shows each referral step, so you can see exactly which nameserver set the world is being pointed at. If those names don&#8217;t match the provider where you&#8217;ve been editing, you&#8217;ve found your problem and the fix takes thirty seconds.</p>



<h3 class="wp-block-heading">2. Correct-looking MX records for the wrong Zoho data center</h3>



<p class="wp-block-paragraph">This is the failure mode that stays invisible longest, and it&#8217;s the reason copying MX values off a random blog post is dangerous.</p>



<p class="wp-block-paragraph">Zoho runs regional data centers, and your account lives in exactly one of them. The mail hostnames differ by region: the international service uses <code>.com</code> hostnames, and the European and Indian deployments use their own regional equivalents. Your domain&#8217;s mailboxes exist only in the data center where your organization was created.</p>



<p class="wp-block-paragraph">Point your MX at the wrong region and everything looks healthy. The records resolve. The hostnames are genuine Zoho servers. A lookup tool will show green. But mail arrives at infrastructure that has no record of your domain, and senders get a rejection saying the address doesn&#8217;t exist, or nothing at all.</p>



<p class="wp-block-paragraph">For the international service, Zoho&#8217;s documentation specifies three records, in this order of preference:</p>



<pre class="wp-block-code"><code>Priority 10   mx.zoho.com
Priority 20   mx2.zoho.com
Priority 50   mx3.zoho.com</code></pre>



<p class="wp-block-paragraph">Do not take that as gospel for your account. The authoritative source is the DNS Mapping section of your own Zoho Mail Admin Console, which shows the exact hostnames for the data center your organization sits in. Read them there, copy them from there. If you&#8217;re unsure which region you&#8217;re on, the domain in the URL when you log into the admin console tells you.</p>



<p class="wp-block-paragraph">Verify what the world sees, not what your panel shows:</p>



<pre class="wp-block-code"><code>dig +short MX example.com

# Confirm the target hostname itself resolves
dig +short mx.zoho.com</code></pre>



<h3 class="wp-block-heading">3. Leftover MX records from whatever came before</h3>



<p class="wp-block-paragraph">MX preference is not a fallback list in the sense most people assume. A sending server tries the lowest number first, and if that server accepts the message, the transaction is over. It never touches the others.</p>



<p class="wp-block-paragraph">So a stale record from your old host sitting at priority 0 or 5 doesn&#8217;t compete with Zoho. It wins outright, every time. The mail is delivered successfully, to a mailbox nobody checks, on a server that may still be quietly accepting for your domain. The sender sees no error. You see nothing at all.</p>



<p class="wp-block-paragraph">This is also why partial migrations bite. Two providers both accepting mail for one domain means your messages split between them based on preference and retry timing, which looks like random message loss and is nearly impossible to reason about from the inside.</p>



<p class="wp-block-paragraph">The rule is simple: after cutover, the only MX records in the zone should be Zoho&#8217;s. Delete everything else, including records for subdomains you forgot about. Check the full answer rather than a truncated view:</p>



<pre class="wp-block-code"><code>dig MX example.com +noall +answer</code></pre>



<h3 class="wp-block-heading">4. A CNAME sitting where the MX records live</h3>



<p class="wp-block-paragraph">Two related problems here, and both are structural rather than typos.</p>



<p class="wp-block-paragraph">The first: an MX record must point to a hostname with an address record. Pointing an MX at a name that is itself a CNAME is forbidden by the DNS specifications, and Zoho documents this explicitly. Some resolvers cope, others don&#8217;t, and the ones that don&#8217;t will fail intermittently in ways that look like a network problem.</p>



<p class="wp-block-paragraph">The second: a CNAME at the zone apex. Classic DNS forbids a CNAME coexisting with any other record at the same name, and MX records at the apex are exactly that. Providers like Cloudflare work around this with CNAME flattening, which synthesizes an address record and does keep your MX records functional. Providers with a naive implementation may shadow or drop them. If someone added an apex CNAME to point the bare domain at a hosting platform or a site builder, that&#8217;s worth checking early.</p>



<pre class="wp-block-code"><code>dig CNAME example.com +short
dig A example.com +short</code></pre>



<p class="wp-block-paragraph">One thing that is <em>not</em> a problem, despite frequent claims otherwise: Cloudflare&#8217;s orange-cloud proxy setting. Proxying applies to HTTP traffic on address records. It has no effect on MX records, and toggling it grey will not fix inbound mail.</p>



<h3 class="wp-block-heading">5. DNSSEC left broken after a nameserver change</h3>



<p class="wp-block-paragraph">Less common, but spectacular when it happens, because it takes the entire domain down rather than just mail.</p>



<p class="wp-block-paragraph">If DNSSEC was enabled and the DS record at the registry no longer matches the signing keys at your current DNS provider, validating resolvers refuse to return any answer at all. They return SERVFAIL. Non-validating resolvers still work, which is why the site loads fine for you and mail from major providers, who mostly validate, vanishes.</p>



<p class="wp-block-paragraph">The test is a one-liner. <code>+cd</code> disables validation checking:</p>



<pre class="wp-block-code"><code># Fails with SERVFAIL if validation is broken
dig MX example.com

# Same query, validation bypassed
dig +cd MX example.com</code></pre>



<p class="wp-block-paragraph">If the second command returns your records and the first doesn&#8217;t, you have a DNSSEC mismatch. The fix is at your registrar: either update the DS record to match your current provider&#8217;s keys, or remove DNSSEC entirely, confirm mail flows, and re-enable it properly afterwards.</p>



<h2 class="wp-block-heading">When DNS is clean and mail still doesn&#8217;t arrive</h2>



<p class="wp-block-paragraph">If your MX records resolve correctly to the right data center and nothing else is in the zone, DNS has done its job. The remaining causes sit on either side of it.</p>



<h3 class="wp-block-heading">Your own server is intercepting the mail</h3>



<p class="wp-block-paragraph">This catches people running a site on cPanel or DirectAdmin, and it&#8217;s genuinely counterintuitive. Control panels have an email routing setting per domain, typically offering local, remote, or automatic delivery. If it&#8217;s set to local mail exchanger, the mail server on that box delivers messages for your domain into its own accounts and never performs an MX lookup.</p>



<p class="wp-block-paragraph">The effect is narrow but confusing: external senders reach Zoho normally, while anything sent from a script or a form <em>on that server</em> goes to a local mailbox instead. Contact form submissions disappear while ordinary email works. Set routing to remote mail exchanger on any host where the site lives but the mailboxes don&#8217;t.</p>



<h3 class="wp-block-heading">The address doesn&#8217;t exist in your Zoho organization</h3>



<p class="wp-block-paragraph">Zoho will only accept mail for addresses it knows about. A user account, an alias on a user, or a group. If someone hands out an address that was never created, Zoho rejects it and the sender gets an address-not-found bounce that they may not think to forward to you.</p>



<p class="wp-block-paragraph">Zoho supports a catch-all address for exactly this, and it&#8217;s worth setting one during a migration so misaddressed mail lands somewhere recoverable rather than bouncing into the void.</p>



<h3 class="wp-block-heading">Zoho accepted it and quarantined it</h3>



<p class="wp-block-paragraph">Under Security and Compliance in the Admin Console, Zoho lets you decide what happens to mail that fails SPF, DKIM, DMARC or DNSBL checks. The available actions include delivering it, quarantining it, or rejecting it outright, with or without a bounce.</p>



<p class="wp-block-paragraph">Set to reject-without-bounce, a sender whose own SPF is slightly wrong gets silently dropped, and neither party learns anything. This is the narrow SPF connection I mentioned earlier: it&#8217;s the sender&#8217;s SPF being judged, not yours. Check the incoming quarantine before concluding mail never arrived, and check the organization blocked list while you&#8217;re there, since a blocked domain or TLD produces exactly the same silence.</p>



<h3 class="wp-block-heading">The domain lapsed or lost verification</h3>



<p class="wp-block-paragraph">If a domain expires, its DNS stops resolving and every associated record becomes unreachable, which also invalidates the domain verification Zoho performed. Zoho&#8217;s documentation notes that after renewal you have to reconfigure MX records as part of reverification. Renewing the domain alone doesn&#8217;t restore mail.</p>



<h2 class="wp-block-heading">A troubleshooting sequence that ends in an answer</h2>



<p class="wp-block-paragraph">Work these in order. Each step either clears a layer or hands you the cause, so you never end up changing three things at once and losing track of which one worked.</p>



<ol class="wp-block-list">
<li><strong>Confirm delegation.</strong> Run <code>dig +short NS example.com</code> and check it matches the provider you&#8217;re editing. If not, stop here and fix that first.</li>



<li><strong>Read the MX records from an authoritative nameserver.</strong> Use <code>dig MX example.com @ns1.yourprovider.com</code> so you&#8217;re seeing the source of truth rather than a cached copy.</li>



<li><strong>Compare against the Admin Console.</strong> Open the DNS Mapping page in Zoho Mail and match hostnames character for character, including the regional suffix.</li>



<li><strong>Confirm nothing else is in the answer.</strong> Any non-Zoho MX record is a suspect, especially one with a lower preference number.</li>



<li><strong>Test resolution from a public resolver.</strong> Try <code>dig MX example.com @1.1.1.1</code> and <code>@8.8.8.8</code>. Differences between them mean you&#8217;re mid-propagation and waiting is the fix.</li>



<li><strong>Rule out DNSSEC.</strong> Compare <code>dig MX example.com</code> against <code>dig +cd MX example.com</code>.</li>



<li><strong>Prove the SMTP conversation.</strong> Send a real test and watch what the server says.</li>



<li><strong>Check quarantine and blocked lists</strong> in the Zoho Admin Console before concluding anything was lost.</li>
</ol>



<p class="wp-block-paragraph">For step seven, <code>swaks</code> is the cleanest tool. It speaks SMTP for you and prints the full exchange, so you see the actual response code rather than guessing from a bounce message:</p>



<pre class="wp-block-code"><code># Stop after RCPT TO so no message is actually delivered
swaks --to you@example.com --server mx.zoho.com --quit-after RCPT</code></pre>



<p class="wp-block-paragraph">A 250 after RCPT TO means Zoho recognises the address and would accept the mail. A 550 means it doesn&#8217;t know that recipient, which points straight at a missing mailbox or the wrong data center. A connection timeout usually isn&#8217;t Zoho at all: many residential and cloud providers block outbound port 25 by default, so run this from a machine you know has egress on 25. If you only have a laptop on a consumer connection, testing through a VPN endpoint that permits SMTP, or from a small VPS, avoids chasing a phantom.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li><strong>Fixing SPF to solve an inbound problem.</strong> Your SPF record has no role in receiving mail. If you can send but not receive, it is not SPF.</li>



<li><strong>Copying MX values from a tutorial.</strong> Regional deployments have different hostnames. Use your own Admin Console every time.</li>



<li><strong>Adding Zoho&#8217;s records without removing the old ones.</strong> A lower preference number beats Zoho outright and nothing warns you.</li>



<li><strong>Trusting the DNS panel over a query.</strong> A saved record and a published record are different things. Query it.</li>



<li><strong>Cutting MX over before creating the mailboxes.</strong> Every address that doesn&#8217;t exist yet bounces, and those messages are gone.</li>



<li><strong>Blaming propagation on day three.</strong> Propagation is bounded by the TTL that was in effect before the change. If it&#8217;s been longer than that, something is genuinely wrong.</li>



<li><strong>Trailing dot confusion.</strong> Some panels want <code>mx.zoho.com.</code> with the dot, others append the zone themselves and turn <code>mx.zoho.com</code> into an unresolvable name. Always confirm with a query afterwards.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li><strong>Drop the TTL before you migrate.</strong> Lower the MX TTL to a few minutes a day ahead of cutover, do the change, then raise it back. This turns a rollback from an overnight wait into a coffee break.</li>



<li><strong>Create every mailbox, alias and group before touching MX.</strong> Zoho&#8217;s own guidance is to configure users first and change MX last.</li>



<li><strong>Set a catch-all during the transition.</strong> It converts &#8220;we lost an email&#8221; into &#8220;it&#8217;s in the catch-all mailbox&#8221;.</li>



<li><strong>Keep DNS in one place.</strong> One authoritative provider, one panel, no ambiguity about where records live.</li>



<li><strong>Monitor the MX record itself.</strong> An uptime or synthetic monitoring service that checks DNS answers, or a scheduled <code>dig</code> comparison in your own pipeline, catches an accidental zone edit before a client does.</li>



<li><strong>Enable DKIM and set DMARC after inbound is confirmed working.</strong> They matter for your outbound reputation. Doing them first just adds variables while you&#8217;re debugging.</li>



<li><strong>Keep a copy of the zone.</strong> Export it before major changes. Reconstructing a zone from memory during an outage is miserable.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Why can I send from Zoho Mail but not receive?</h3>



<p class="wp-block-paragraph">Because sending authenticates directly against Zoho&#8217;s outbound servers and doesn&#8217;t consult your domain&#8217;s DNS, while receiving depends entirely on your MX records. Working outbound with broken inbound is a strong signal that the problem is an MX record, a nameserver delegation issue, or a leftover record from a previous provider.</p>



<h3 class="wp-block-heading">How long should MX changes take to work?</h3>



<p class="wp-block-paragraph">Propagation is governed by the TTL that was set on the old record, not by any fixed schedule. If the previous TTL was one hour, resolvers holding a cached copy will refresh within an hour. If it was set to a day, expect a day. Nothing you do afterwards speeds up a cache that has already been populated.</p>



<h3 class="wp-block-heading">Do I need all three Zoho MX records?</h3>



<p class="wp-block-paragraph">Mail will flow with only the primary, but the backups exist so that senders have somewhere to deliver if the primary is unreachable. Without them, sending servers queue and retry, which delays delivery and eventually bounces. Add the full set your Admin Console lists.</p>



<h3 class="wp-block-heading">Does SPF affect whether I receive email?</h3>



<p class="wp-block-paragraph">Your own SPF record does not. It&#8217;s evaluated by servers receiving mail from you. The one indirect connection is that Zoho evaluates the <em>sender&#8217;s</em> SPF on incoming mail, and if your Admin Console is set to reject or quarantine on SPF failure, legitimate mail from badly configured senders can disappear.</p>



<h3 class="wp-block-heading">Why do lookup tools say my MX records are fine when mail still fails?</h3>



<p class="wp-block-paragraph">Lookup tools verify that records exist and resolve. They can&#8217;t know which Zoho data center your organization lives in, whether the mailbox has been created, or whether a message is sitting in quarantine. A green result rules out one layer, not all of them.</p>



<h3 class="wp-block-heading">Can I run Zoho Mail alongside another mail provider on one domain?</h3>



<p class="wp-block-paragraph">Not by simply listing both sets of MX records. That splits mail unpredictably. Zoho does offer routing arrangements designed for gradual migration, configured on the Zoho side rather than by stacking MX records in DNS. If you need a genuine split, use the supported mechanism.</p>



<h3 class="wp-block-heading">My contact form emails vanish but normal email works. Why?</h3>



<p class="wp-block-paragraph">Almost always the control panel email routing setting on the server hosting the site. If it&#8217;s on local mail exchanger, mail generated on that machine is delivered into a local account rather than sent out to Zoho. Switch it to remote.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">If you take one thing from this: when Zoho Mail is not receiving email but sending works, stop looking at authentication records. Inbound mail is decided by your MX records, and those records have to be published by the nameservers that are actually authoritative for your domain, pointing at the data center your Zoho organization lives in, with nothing else in the answer competing with them.</p>



<p class="wp-block-paragraph">Query, don&#8217;t assume. Read the answer from the authoritative nameserver rather than the panel, compare it against your own Admin Console instead of a tutorial, then work down through SMTP and Zoho&#8217;s own quarantine. Almost every case resolves in that order, and the ones that don&#8217;t are usually a mailbox that was never created in the first place.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need someone to untangle your mail DNS?</h2>



<p class="wp-block-paragraph">I work on DNS and mail delivery problems like this regularly, usually for people who have already tried three fixes and made things harder to reason about. Things I can help with:</p>



<ul class="wp-block-list">
<li>Diagnosing inbound mail failures end to end, from delegation through SMTP response codes to Zoho&#8217;s quarantine</li>



<li>Planning and executing a mail migration to Zoho with a TTL schedule and a rollback path that actually works</li>



<li>Auditing and cleaning up a zone that several people have edited over the years</li>



<li>Setting up SPF, DKIM and DMARC properly once inbound is stable, including staged DMARC enforcement</li>



<li>Fixing DNSSEC mismatches after a nameserver move without taking the domain down again</li>



<li>Adding DNS and mail-flow monitoring so the next zone edit doesn&#8217;t go unnoticed for a week</li>
</ul>



<p class="wp-block-paragraph">Send me the output of <code>dig +trace MX yourdomain.com</code> and a rejection message from a sender, and I can usually tell you where it&#8217;s breaking before we&#8217;ve agreed on anything.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/zoho-mail-not-receiving-email/">Zoho Mail Not Receiving Email? A DNS Troubleshooting Guide</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Zoho Analytics Import vs Live Connect: The Trade-Offs Nobody Mentions</title>
		<link>https://john-nessime.com/blog/technical-guides/zoho-analytics-import-vs-live-connect/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 30 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon RDS]]></category>
		<category><![CDATA[Business Intelligence]]></category>
		<category><![CDATA[Caching]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Freshness]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Warehouse]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Live Connect]]></category>
		<category><![CDATA[MySQL]]></category>
		<category><![CDATA[Query Timeout]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[Zoho]]></category>
		<category><![CDATA[Zoho Analytics]]></category>
		<category><![CDATA[Zoho Databridge]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=346</guid>

					<description><![CDATA[<p>Import mode fails quietly and Live Connect fails loudly. A practical breakdown of what each Zoho Analytics connection mode actually does, where each one wins, the four levers that should drive the choice, and a decision procedure you can run before you build the workspace.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/zoho-analytics-import-vs-live-connect/">Zoho Analytics Import vs Live Connect: The Trade-Offs Nobody Mentions</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually arrives on a Monday. Someone in finance opens the revenue dashboard, compares it against the operational system, and asks why one says 41 and the other says 47. Nothing is broken. No sync failed. No alert fired. The dashboard is just showing you a snapshot from six hours ago, and absolutely nothing on the screen tells you that.</p>



<p class="wp-block-paragraph">That is the real cost of picking the wrong data connection mode, and it is why the choice between Zoho Analytics import vs Live Connect deserves more than five minutes during setup. Both modes work. Both are documented. But they fail in completely different ways, and one of those failure modes is silent.</p>



<p class="wp-block-paragraph">This post covers what each mode actually does under the hood, where each one genuinely wins, the four levers that should drive the decision, a short procedure for running that decision yourself, and the arguments people repeat that don&#8217;t hold up once you have a workspace in production.</p>



<h2 class="wp-block-heading">What the two modes actually do</h2>



<p class="wp-block-paragraph">Strip away the marketing language and the difference is where the data lives when someone opens a report.</p>



<p class="wp-block-paragraph"><strong>Data Import</strong> copies rows out of your source and stores them inside Zoho Analytics. Reports query that stored copy. You schedule refreshes to pull new data on an interval, and between refreshes your dashboard is a photograph, not a window.</p>



<p class="wp-block-paragraph"><strong>Live Connect</strong> does not copy anything. Zoho pulls metadata about your tables and columns, then generates SQL against your source database every single time a report or dashboard is opened. Your database does the work. Zoho does the rendering.</p>



<p class="wp-block-paragraph">That one architectural difference cascades into everything else: what you get billed for, which features you can use, who owns schema changes, and what happens when a query gets slow.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Import mode: where it wins and where it hurts</h2>



<h3 class="wp-block-heading">Where import mode wins</h3>



<p class="wp-block-paragraph">Import mode is the default for a reason, and the reasons are good ones.</p>



<ul class="wp-block-list">
<li><strong>Report speed is yours to control.</strong> Because the data sits in Zoho&#8217;s storage, load time doesn&#8217;t depend on how busy your production database is at 9am on a Monday.</li>



<li><strong>You get the full feature surface.</strong> Query Tables, custom import queries, data type changes on imported columns, and blending several unrelated sources into one workspace all work in import mode.</li>



<li><strong>Your production database is insulated.</strong> An analyst who builds an accidental many-to-many join hits a copy, not the system taking customer orders.</li>



<li><strong>Incremental fetch keeps sync cheap.</strong> Rather than re-pulling a whole table on every schedule, you nominate a column that identifies new rows and pull only what changed.</li>



<li><strong>It works on every plan.</strong> There is no tier gate on importing data, which matters if you are prototyping before anyone has approved a budget.</li>
</ul>



<h3 class="wp-block-heading">Where import mode hurts</h3>



<p class="wp-block-paragraph">The first problem is the one from the opening: <strong>staleness is invisible</strong>. A Zoho Analytics dashboard built on imported data looks exactly the same whether it refreshed two minutes ago or two days ago. There is no built-in &#8220;as of&#8221; banner on the chart. If your refresh schedule is daily and someone is making a call at 4pm, they are working from this morning&#8217;s world.</p>



<p class="wp-block-paragraph">The second is <strong>billing shape</strong>. Zoho Analytics prices on stored rows, counted as the sum of all rows across every table in every workspace in the account. Import mode is therefore the mode that grows your bill. Every table you sync &#8220;just in case&#8221; is a line item. Rates change, so check the current pricing page rather than trusting any figure you read in a blog post, but understand the mechanism: rows stored equals money.</p>



<p class="wp-block-paragraph">Third, <strong>the copy drifts and nobody owns it</strong>. Six months in, a table has a scheduled import that has been silently failing, or someone changed a transformation in the source and the imported copy still reflects the old logic. Import mode creates a second version of the truth, and second versions need maintenance.</p>



<p class="wp-block-paragraph">Fourth, <strong>refresh frequency is plan-gated</strong>. How often you can schedule a sync depends on your subscription tier. If your reporting requirement is &#8220;within the hour&#8221; and your plan permits fewer refreshes per day than that implies, import mode cannot meet the requirement no matter how you configure it.</p>



<h2 class="wp-block-heading">Live Connect: where it wins and where it hurts</h2>



<h3 class="wp-block-heading">Where Live Connect wins</h3>



<ul class="wp-block-list">
<li><strong>There is exactly one copy of the data.</strong> No reconciliation conversation, because there is nothing to reconcile. What the report shows is what the database holds at that moment.</li>



<li><strong>Sensitive data stays where it already is.</strong> For teams with a data residency position, or a security team that objects to bulk-copying customer records into a BI vendor&#8217;s storage, this is often the deciding argument on its own.</li>



<li><strong>Foreign keys carry over.</strong> Tables linked by foreign keys in the source get lookup relationships created automatically in Zoho, which is a genuine time saver compared to wiring lookup columns by hand.</li>



<li><strong>Row storage isn&#8217;t the cost driver.</strong> You&#8217;re not paying to warehouse a duplicate of a table you already pay to host on Amazon RDS, Snowflake, or a Postgres instance on your own VPS.</li>
</ul>



<h3 class="wp-block-heading">Where Live Connect hurts</h3>



<p class="wp-block-paragraph">Here is the one that catches people, and it is worth reading twice: <strong>Live Connect does not automatically track schema changes</strong>. Add a column, rename a column, drop a table, change a data type in the source, and Zoho will not follow along. Reports built on the affected objects land in a Mismatch state and you have to reconcile them manually through the Sync Design option in the data source settings.</p>



<p class="wp-block-paragraph">This is the inverse of what most people assume. &#8220;Live&#8221; sounds like &#8220;always current in every respect.&#8221; It means live <em>data</em>, not live <em>structure</em>. A routine migration that renames <code>customer_id</code> to <code>account_id</code> will break dashboards that nobody looked at that week, and you&#8217;ll find out from a user rather than from a monitor.</p>



<p class="wp-block-paragraph">The rest of the trade-offs:</p>



<ul class="wp-block-list">
<li><strong>Query timeout.</strong> Zoho documents a default query timeout of 2.5 minutes on Live Connect. Anything slower fails. On a source that hasn&#8217;t been tuned for analytical queries, that ceiling arrives faster than you expect.</li>



<li><strong>No Custom Query at import time.</strong> You can&#8217;t hand Zoho a filtered SELECT to narrow what it sees. The documented workaround is to create a view in the source database and connect to that instead.</li>



<li><strong>No Query Tables.</strong> The SQL-over-your-workspace feature depends on data being stored locally. Zoho notes Query Table support for selected databases is enabled on request, so it is worth asking, but do not plan around it.</li>



<li><strong>No data type changes in Zoho.</strong> If a column is text in the source and should be a date, you fix it in the source or in a view. There is no in-Zoho override.</li>



<li><strong>Lookups only within the connection.</strong> You can define lookup rules between tables in the same live connection, not across to other data sources. Reports from different sources can still be combined on a dashboard, but that is a presentation-layer join, not a data-layer one.</li>



<li><strong>It is gated to higher plans.</strong> Zoho&#8217;s documentation places Live Connect on the Premium and Enterprise tiers. If you&#8217;re on a lower plan, the decision has already been made for you.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The four levers behind Zoho Analytics import vs Live Connect</h2>



<p class="wp-block-paragraph">Feature checklists make this look like a close call. It usually isn&#8217;t, once you work out which of these four you actually care about.</p>



<p class="wp-block-paragraph"><strong>1. Your freshness contract.</strong> Not &#8220;how fresh would be nice.&#8221; What decision gets made from this dashboard, and how wrong can the data be before that decision goes bad? A weekly pipeline review tolerates a nightly sync comfortably. A dispatcher deciding which order to fulfil next does not. Write the tolerance down as a number of minutes before you look at any feature table.</p>



<p class="wp-block-paragraph"><strong>2. Where the cost lands.</strong> Import mode moves cost onto your Zoho row count. Live Connect moves it onto your database, in query load and in whatever you pay for the compute that serves it. Neither is free. A warehouse that bills per query scanned is a different economic proposition under Live Connect than a fixed-price MySQL instance on Contabo or InterServer, where extra query load costs nothing until you outgrow the box.</p>



<p class="wp-block-paragraph"><strong>3. Where transformation happens.</strong> If your model needs joins, unions, derived columns, and cleanup, import mode gives you Query Tables to do that inside Zoho. Live Connect pushes all of it down into the source, which means you need database write access and someone comfortable maintaining views. That&#8217;s a team-capability question as much as a technical one.</p>



<p class="wp-block-paragraph"><strong>4. Who owns schema change.</strong> Under Live Connect, every source migration is potentially a reporting incident. That is fine if the same team owns both, and painful if the application team ships migrations without telling anyone. Import mode is more forgiving here, since Zoho can automatically pick up column additions and deletions on synced tables.</p>



<h2 class="wp-block-heading">A decision procedure you can run in ten minutes</h2>



<ol class="wp-block-list">
<li><strong>Check the plan gate first.</strong> If you&#8217;re not on a tier that includes Live Connect, stop. Import mode, tuned well, is the answer.</li>



<li><strong>Write down the freshness tolerance in minutes.</strong> If the number is larger than your plan&#8217;s shortest available sync interval, import mode satisfies it. If it&#8217;s smaller, Live Connect is the only mode that can meet it.</li>



<li><strong>Time the heaviest query.</strong> Take the ugliest aggregation your dashboard will need and run it against the source with realistic filters. If it takes anywhere near 2.5 minutes, Live Connect will fail under real usage. Tune the source or choose import.</li>



<li><strong>List the transformations you need.</strong> If any of them require a Query Table, joining across unrelated sources, or changing a column&#8217;s data type in Zoho, import mode is doing that work.</li>



<li><strong>Ask who ships schema migrations.</strong> If it isn&#8217;t the people who own the dashboards, budget for Sync Design work every release, or use import mode.</li>



<li><strong>Check the security position last, because it can override everything.</strong> If the answer to &#8220;may customer records be stored in a third-party BI platform&#8221; is no, Live Connect is the only option and you build around its constraints.</li>
</ol>



<p class="wp-block-paragraph">In practice, mixed estates are common and reasonable. Live Connect for the operational workspace where freshness is the point, import mode for the historical and blended reporting where transformation and speed matter more.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<p class="wp-block-paragraph"><em>&#8220;Live Connect is real-time, so it&#8217;s more accurate.&#8221;</em> It&#8217;s more current. Accuracy is a modelling problem, not a transport problem. A live connection to a table with duplicate lookup keys produces wrong numbers instantly instead of wrong numbers on a schedule.</p>



<p class="wp-block-paragraph"><em>&#8220;Live Connect is cheaper because you don&#8217;t pay for rows.&#8221;</em> Sometimes. You&#8217;ve moved the cost, not removed it. Every dashboard open is a set of queries against your source. On per-query or per-compute-second billing, an enthusiastic team with auto-refreshing dashboards can generate a bill that makes row storage look like a bargain.</p>



<p class="wp-block-paragraph"><em>&#8220;Just enable the cache and Live Connect gets fast.&#8221;</em> Zoho does offer report caching on Live Connect workspaces, with a refresh interval you set in minutes. It genuinely helps. But note what you&#8217;ve just built: a copy of the data with a staleness window, which is the thing you chose Live Connect to avoid. Caching is a legitimate tuning tool, not a free upgrade.</p>



<p class="wp-block-paragraph"><em>&#8220;We&#8217;ll switch modes later if it doesn&#8217;t work out.&#8221;</em> Switching is not a toggle. Zoho does not permit importing data into an existing Live Connect workspace, and the two modes support different feature sets, so a migration means rebuilding the workspace and its reports. Choose deliberately the first time.</p>



<h2 class="wp-block-heading">If you go Live Connect, tune the source first</h2>



<p class="wp-block-paragraph">Live Connect performance is your database&#8217;s performance. Zoho&#8217;s own guidance is blunt about this, and it matches what you&#8217;d do for any reporting workload.</p>



<p class="wp-block-paragraph">Index the columns that lookups and joins run through. Without them, every dashboard open is a table scan:</p>



<pre class="wp-block-code"><code>-- Index the join key that Zoho will generate queries against
CREATE INDEX idx_orders_customer_id ON orders (customer_id);

-- Confirm the planner actually uses it (PostgreSQL)
EXPLAIN ANALYZE
SELECT customer_id, SUM(total)
FROM orders
WHERE order_date &gt;= CURRENT_DATE - INTERVAL '90 days'
GROUP BY customer_id;</code></pre>



<p class="wp-block-paragraph">Because Custom Query isn&#8217;t available in Live Connect, a view is how you narrow and shape what Zoho sees. Connect to the view rather than the raw table:</p>



<pre class="wp-block-code"><code>CREATE VIEW reporting_orders AS
SELECT
    o.id,
    o.customer_id,
    o.order_date,
    o.total,
    CASE WHEN o.status = 'complete' THEN 1 ELSE 0 END AS is_complete
FROM orders o
WHERE o.order_date &gt;= CURRENT_DATE - INTERVAL '2 years';</code></pre>



<p class="wp-block-paragraph">For calculations that get repeated across many reports, a materialized view precomputes the work so the 2.5 minute ceiling stops being a threat. You trade a refresh schedule for query speed, which is a much smaller staleness window than a full import.</p>



<p class="wp-block-paragraph">The rest of the list is short and worth following:</p>



<ul class="wp-block-list">
<li>Keep formula columns simple, and prefer CASE WHEN over deeply nested IF statements.</li>



<li>Make sure lookup columns hold unique values. Duplicate keys degrade query performance badly and quietly inflate aggregates.</li>



<li>Avoid many-to-many joins and circular join paths in your model.</li>



<li>Give Zoho a database user with SELECT plus metadata read on exactly the tables it needs, and nothing more. Missing metadata permission is a common cause of a table that connects but never loads.</li>



<li>Decide your connectivity path deliberately: allowlisting Zoho&#8217;s published IP ranges for a cloud database, or Zoho Databridge as an agent when the database sits in a private network. SSL and SSH tunneling are both supported on the connection.</li>



<li>Put a change-notification step in your migration checklist. If a release renames or drops a column, someone should run Sync Design before users find the Mismatch.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I switch a workspace from import mode to Live Connect?</h3>



<p class="wp-block-paragraph">Not as an in-place setting change. Zoho does not permit importing data into an existing Live Connect workspace, and the feature sets differ enough that reports built one way won&#8217;t simply carry over. Treat a mode change as a rebuild and plan the reporting downtime accordingly.</p>



<h3 class="wp-block-heading">Does Live Connect count against my row limit?</h3>



<p class="wp-block-paragraph">Zoho&#8217;s row-based pricing counts rows stored across your workspace tables, and Live Connect doesn&#8217;t store your source data in Zoho. The practical implication is that Live Connect shifts your constraint from storage to source query capacity. Confirm the exact treatment for your account with Zoho before you build a cost model on it, since add-ons and plan definitions change.</p>



<h3 class="wp-block-heading">Why do my Live Connect reports suddenly show a view-cannot-be-accessed error?</h3>



<p class="wp-block-paragraph">Almost always because a table or column the report depends on was renamed or dropped in the source. Live Connect doesn&#8217;t track structural changes automatically. Open the data source settings, check the Mismatch tab, and use Sync Design to remap. Insufficient SELECT or metadata permissions on the source produce a similar symptom.</p>



<h3 class="wp-block-heading">What is incremental fetch, and does it apply to both modes?</h3>



<p class="wp-block-paragraph">Incremental fetch is an import-mode feature. Instead of re-pulling every row on each schedule, you nominate a column that identifies new records, such as an auto-incrementing ID or a timestamp, and choose whether new rows are appended or matched and updated. It cuts sync time dramatically on large transactional tables. It has no meaning under Live Connect, where nothing is fetched on a schedule in the first place.</p>



<h3 class="wp-block-heading">Can I combine Live Connect tables with data from other sources?</h3>



<p class="wp-block-paragraph">Not at the data layer. Lookup rules only work between tables within the same live connection. You can place reports from a Live Connect workspace and reports from other sources side by side on a dashboard, but you cannot join them into a single dataset the way you can with imported tables in one workspace.</p>



<h3 class="wp-block-heading">Is Live Connect available for on-premise and private databases?</h3>



<p class="wp-block-paragraph">Yes, through Zoho Databridge, the agent that bridges a database in a private network to Zoho&#8217;s servers. You choose the connection type during setup, so the same agent supports either mode. If your database is publicly reachable instead, you allowlist Zoho&#8217;s published IP ranges and connect directly.</p>



<h3 class="wp-block-heading">Which mode is better for a Zoho CRM or Zoho Desk workspace?</h3>



<p class="wp-block-paragraph">Neither, in the sense that the question doesn&#8217;t apply. Zoho&#8217;s own application connectors sync data into Analytics on a schedule, which behaves like import mode. Live Connect is for external databases and data warehouses. If you need faster refreshes on Zoho app data, the lever is the connector&#8217;s sync schedule and your plan tier, not the connection mode.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The Zoho Analytics import vs Live Connect decision is not a performance question. It&#8217;s a question about which failure you&#8217;d rather have.</p>



<p class="wp-block-paragraph">Import mode fails quietly. Numbers stay confidently wrong until someone reconciles them by hand, and the bill grows with every table you sync out of habit. Live Connect fails loudly. Someone renames a column and a dashboard breaks in front of a user, or a heavy query hits the timeout during the Monday review.</p>



<p class="wp-block-paragraph">Loud failures are usually easier to live with, because you find out about them. But they only work if you have the source database, the access to tune it, and a change process that includes reporting. If you don&#8217;t have those three, import mode with an honest refresh schedule and a documented staleness window will serve you far better than a live connection you can&#8217;t maintain.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second opinion on your Zoho Analytics setup?</h2>



<p class="wp-block-paragraph">Most of the Zoho Analytics work I get called into is somebody discovering the mode choice was made in five minutes eighteen months ago. Things I can help with:</p>



<ul class="wp-block-list">
<li>Auditing an existing workspace and telling you honestly whether the connection mode still fits what you&#8217;re using it for</li>



<li>Profiling and tuning the source database behind a Live Connect workspace so dashboards stop hitting the query timeout</li>



<li>Designing reporting views and materialized views that push transformation into the database where Live Connect can use it</li>



<li>Setting up Zoho Databridge for databases in private networks, including SSL and SSH tunneling and the firewall rules around it</li>



<li>Configuring incremental fetch on import-mode tables to cut sync times and stop row counts drifting upward</li>



<li>Building a schema-change process so source migrations don&#8217;t turn into reporting incidents</li>
</ul>



<p class="wp-block-paragraph">If you want a concrete starting point, send me the slow query, the Mismatch screenshot, or the EXPLAIN output from the aggregation that&#8217;s timing out. That&#8217;s usually enough to tell where the actual problem is.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/zoho-analytics-import-vs-live-connect/">Zoho Analytics Import vs Live Connect: The Trade-Offs Nobody Mentions</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Run It Twice, Get Two Answers: Building an ETL Pipeline From Zoho CRM to Amazon S3</title>
		<link>https://john-nessime.com/blog/devops/zoho-crm-to-amazon-s3/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 02 Aug 2026 18:16:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[Zoho]]></category>
		<category><![CDATA[Zoho CRM]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=86</guid>

					<description><![CDATA[<p>You re-run the same extract for the same window and get a different set of rows. Nothing errored. You were paginating a result set that kept changing while you read it. Here's how to build a Zoho CRM to S3 pipeline whose runs are repeatable, from closed read windows to Bulk Read and deletions.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/zoho-crm-to-amazon-s3/">Run It Twice, Get Two Answers: Building an ETL Pipeline From Zoho CRM to Amazon S3</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The monthly deal count in your warehouse is short by about thirty rows. You re-run the extract for exactly the same window, expecting to confirm the bug, and this time you get a different thirty. Run it a third time and the number moves again.</p>



<p class="wp-block-paragraph">Nothing is broken in the way you are looking for. The records exist in Zoho. Your job did not error. What happened is that you asked for every deal modified since your last watermark, sorted, and then walked through it page by page while sales reps kept working. Records got modified during the read, changed their position in the sort order, and shifted from page 4 to page 2 after you had already read page 2. So you never saw them.</p>



<p class="wp-block-paragraph">That is the failure that defines building an <strong>ETL pipeline from Zoho CRM to Amazon S3</strong>, and it does not look like a bug. It looks like flaky data. It only shows up on busy modules, it never produces an error, and re-running the job produces a different wrong answer, which is the single most confusing symptom in data engineering.</p>



<p class="wp-block-paragraph">The fix is one rule, and everything else in this post follows from it. This covers that rule, which extraction API to use and when, authentication and the data centre trap that wastes an afternoon, handling deletions (Zoho is genuinely good here), shaping nested CRM JSON for S3, and staying inside your API credits.</p>



<h2 class="wp-block-heading">The rule: bound both ends of the window</h2>



<p class="wp-block-paragraph">Almost every incremental extract is written like this:</p>



<pre class="wp-block-code"><code>Modified_Time &gt; {last_run}</code></pre>



<p class="wp-block-paragraph">That query has no upper bound, which means the result set keeps growing while you read it. You are paginating a moving target. Add an upper bound and put it slightly in the past:</p>



<pre class="wp-block-code"><code>Modified_Time &gt; {last_run} AND Modified_Time &lt; {now_minus_lag}</code></pre>



<p class="wp-block-paragraph">Now the set is frozen. Records modified during your run land after the upper bound and get picked up next time. The extract becomes repeatable: run it twice, get the same rows twice. That property is worth more than any amount of retry logic, because it means a failed run costs you nothing and a suspicious number can be checked by re-running.</p>



<p class="wp-block-paragraph">Two details that matter. The lag needs to comfortably exceed how long your extract takes plus any clock skew between you and Zoho; ten minutes is a sensible starting point and costs you ten minutes of freshness. And <strong>only advance the watermark after the entire window has landed in S3</strong>, never after the API call succeeds. Those are different moments, and the gap between them is where data goes missing.</p>



<pre class="wp-block-code"><code>#!/usr/bin/env bash
set -euo pipefail

LAG_MINUTES=10
T1=$(cat state/deals.watermark)
T2=$(date -u -d "-${LAG_MINUTES} minutes" +%Y-%m-%dT%H:%M:%S+00:00)

extract_window "$T1" "$T2"

# Watermark advances only once the data is durably in S3.
echo "$T2" &gt; state/deals.watermark</code></pre>



<p class="wp-block-paragraph">Keep that state somewhere durable and versioned, not on the box running the job. DynamoDB, Parameter Store, or a small object in S3 all work. A watermark file on an ephemeral runner is a watermark you will lose.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Authentication, and the trap that costs an afternoon</h2>



<p class="wp-block-paragraph">Zoho uses OAuth 2.0. You register a client in the Zoho API Console, get a client ID and secret, exchange a grant token for a refresh token once, and then trade the refresh token for short-lived access tokens from then on. Access tokens last an hour, so cache them rather than requesting one per call.</p>



<p class="wp-block-paragraph">Here is the part that catches everyone: <strong>Zoho runs multiple data centres and they are separate worlds.</strong> An account in the EU data centre authenticates against a different accounts domain and calls a different API domain than a US account. A token issued in one region is meaningless in another, and the error you get back does not say &#8220;wrong data centre&#8221;, it says the token is invalid.</p>



<p class="wp-block-paragraph">So make the domain a configuration value from the first line of code, never a hardcoded string. The examples below use the US domain. If your org lives elsewhere, both the accounts domain and the API domain change together.</p>



<p class="wp-block-paragraph">Two more things worth getting right at the start. Request the narrowest OAuth scopes that work, read-only on the modules you actually extract, because scopes are easy to widen later and awkward to explain in an audit. And store the refresh token in Secrets Manager rather than an environment variable, since it does not expire and is effectively a permanent key to your CRM.</p>



<h2 class="wp-block-heading">Which API to extract with</h2>



<p class="wp-block-paragraph">Three options, and the choice is mostly about volume.</p>



<h3 class="wp-block-heading">Bulk Read, for anything large</h3>



<p class="wp-block-paragraph">This is the right default for a lake. You POST a job description, Zoho runs the export server-side, and you poll for status or supply a callback URL. When it finishes you get a download URL for a ZIP of CSV. A single job handles up to 200,000 records, with paging beyond that, and it does not consume your standard API limits the way record-by-record calls do.</p>



<p class="wp-block-paragraph">The important property for our purposes: the pagination problem disappears. Zoho assembles the export on its side, so there is no page-by-page read for records to slip between.</p>



<pre class="wp-block-code"><code>curl "https://www.zohoapis.com/crm/bulk/v8/read" 
  -X POST 
  -H "Authorization: Zoho-oauthtoken $ACCESS_TOKEN" 
  -H "Content-Type: application/json" 
  -d @job.json</code></pre>



<pre class="wp-block-code"><code>{
  "query": {
    "module": { "api_name": "Deals" },
    "fields": ["id", "Deal_Name", "Amount", "Stage", "Modified_Time"],
    "criteria": {
      "group_operator": "and",
      "group": [
        {
          "field": { "api_name": "Modified_Time" },
          "comparator": "greater_than",
          "value": "{T1}"
        },
        {
          "field": { "api_name": "Modified_Time" },
          "comparator": "less_than",
          "value": "{T2}"
        }
      ]
    },
    "page": 1
  }
}</code></pre>



<p class="wp-block-paragraph">Two limits to design around. The download URL is only valid for about a day, so fetch and land the file promptly rather than queueing it for a later step. And downloads are rate limited per minute, so if you fan out across many modules at once you will start collecting 429s.</p>



<h3 class="wp-block-heading">COQL, for moderate volumes and real filtering</h3>



<p class="wp-block-paragraph">Zoho&#8217;s SQL-like query API. Up to 2,000 records per call and up to 100,000 records total per unique criteria through pagination. Past that, Zoho&#8217;s own documentation tells you to use Bulk Read, which is good advice to take rather than work around.</p>



<p class="wp-block-paragraph">If you do paginate COQL, paginate by key rather than by offset. Sort by <code>id</code> and carry the last ID you saw into the next call, so shifting records cannot cause a skip:</p>



<pre class="wp-block-code"><code>{
  "select_query": "select id, Deal_Name, Amount, Stage, Modified_Time from Deals where (Modified_Time &gt; '{T1}' and Modified_Time &lt; '{T2}') and id &gt; {last_id} order by id asc limit 2000"
}</code></pre>



<p class="wp-block-paragraph">Note the shape: a closed time window that freezes the set, plus a keyset cursor that walks it deterministically. Offset pagination gives you neither. Zoho&#8217;s own documentation uses this pattern in its examples, which is a fair hint about what they expect.</p>



<h3 class="wp-block-heading">Get Records, for small modules and metadata</h3>



<p class="wp-block-paragraph">The plain module endpoint returns up to 200 records a page and supports an <code>If-Modified-Since</code> header for incremental reads. Fine for lookup tables and small custom modules. Not what you want pointed at a Deals module with six figures of rows.</p>



<p class="wp-block-paragraph">Worth saying plainly: if all of this sounds like a fortnight of work you would rather not own, a managed connector from Fivetran or Airbyte solves the extraction half and you spend your time on the modelling instead. Price it against your own hours honestly, because the build-it-yourself option is habitually costed at zero.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Deletions, which Zoho actually handles well</h2>



<p class="wp-block-paragraph">An incremental extract can never see a deletion, because a record that no longer exists cannot appear in a query for modified records. Most CRMs make you infer this. Zoho gives you a direct endpoint, and it is one of the nicer parts of the API.</p>



<pre class="wp-block-code"><code># type: all | recycle | permanent
curl "https://www.zohoapis.com/crm/v8/Deals/deleted?type=all&amp;per_page=200&amp;page=1" 
  -X GET 
  -H "Authorization: Zoho-oauthtoken $ACCESS_TOKEN" 
  -H "If-Modified-Since: {T1}"</code></pre>



<p class="wp-block-paragraph">The retention windows are generous: records in the recycle bin are retrievable for up to 60 days from deletion, and permanently deleted records for up to 120 days. Compared with warehouses where the detection window is a couple of weeks and not guaranteed, that is a lot of slack. It also means there is no excuse for missing a deletion, since any sane sync interval sits comfortably inside it.</p>



<p class="wp-block-paragraph">Run the deleted-records call as part of every extract cycle, using the same closed window, and mark the matching rows as deleted in your curated layer rather than removing them from raw. Soft-delete preserves the audit trail and lets you answer &#8220;when did this disappear&#8221; later.</p>



<p class="wp-block-paragraph">Even with a good endpoint, run a periodic full ID reconciliation as a backstop, monthly is plenty. Pull just the ID column for the whole module, diff it against the lake, and flag anything you still believe in that Zoho does not. It catches the cases the deletion endpoint does not describe cleanly, such as records merged or moved between modules.</p>



<h2 class="wp-block-heading">Shaping CRM data for S3</h2>



<p class="wp-block-paragraph">Land raw first, exactly as returned, in its own prefix. Then convert to Parquet in a second step. Skipping the raw layer feels efficient right up to the first time you need to reprocess a month with corrected logic.</p>



<pre class="wp-block-code"><code># Partition by the window's upper bound, not by a business date.
# One bad run is then one partition to replace.
s3://acme-lake/raw/zoho_crm/deals/window_end=YYYY-MM-DDTHH/
s3://acme-lake/curated/zoho_crm/deals/window_end=YYYY-MM-DDTHH/</code></pre>



<p class="wp-block-paragraph">Zoho returns nested structures that do not map onto a flat table. Lookup fields come back as objects with an ID and a display name. Multi-select fields and tags come back as arrays. Subforms come back as arrays of objects. You have three choices per field and should make each one deliberately:</p>



<ul class="wp-block-list">
<li><strong>Flatten into columns.</strong> A lookup becomes <code>account_id</code> and <code>account_name</code>. Correct for anything you filter or join on.</li>
<li><strong>Split into a child table.</strong> Subforms and line items become their own Parquet dataset keyed by parent ID. Correct when the nested rows are things people count.</li>
<li><strong>Keep as a JSON string column.</strong> Fine for rarely-queried arrays, and Athena can parse it on demand. Not fine for anything in a regular report.</li>
</ul>



<p class="wp-block-paragraph">Pin your types explicitly during conversion rather than letting the writer infer them from a batch. Inference is the reason a column is a string in January&#8217;s files and a double in February&#8217;s, and Athena will happily read both and quietly fail to reconcile them.</p>



<p class="wp-block-paragraph">Register the result in the Glue Data Catalog and query with Athena. If you expect to apply updates and deletes in place rather than rebuilding partitions, Apache Iceberg earns its extra setup: row-level operations and schema evolution are exactly what a CRM feed generates.</p>



<p class="wp-block-paragraph">On the AWS side, the orchestration is unremarkable and should stay that way: EventBridge on a schedule, Lambda or a small container for the extract, Step Functions if you need to poll a Bulk Read job and branch on the result. Resist Glue jobs for the extraction itself unless you are already deep in Glue; a Python container you can run locally is easier to debug at seven in the morning.</p>



<h2 class="wp-block-heading">API credits</h2>



<p class="wp-block-paragraph">Zoho meters API usage as credits, allocated by edition and user count, over a rolling 24-hour window, and shared with every other integration on the org. There are also concurrency limits, so hammering the API in parallel fails differently from exceeding your daily allowance.</p>



<ul class="wp-block-list">
<li><strong>Bulk Read for volume.</strong> It sidesteps the standard limits and is the whole reason the API exists.</li>
<li><strong>Ask for fewer fields.</strong> Selecting only the columns you actually land reduces payload and processing on both sides.</li>
<li><strong>Mind the COQL limit value.</strong> Credits scale with the page size you request, so grabbing 2,000 rows costs more than grabbing 200. Larger pages are still usually the better trade, but know you are making it.</li>
<li><strong>Give the pipeline its own connected app and user.</strong> Then consumption is attributable when someone asks who exhausted the credits.</li>
<li><strong>Back off properly on 429.</strong> Exponential backoff with jitter, not a fixed sleep, and treat rate limiting as an expected condition rather than an error.</li>
</ul>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Two runs of the same window return different rows</h3>



<p class="wp-block-paragraph">Open-ended window, offset pagination, or both. Add the upper bound, switch to keyset pagination or Bulk Read, and the symptom disappears. This is the one to check before anything else, because it makes every other investigation unreliable.</p>



<h3 class="wp-block-heading">Invalid token, but the credentials are definitely right</h3>



<p class="wp-block-paragraph">Data centre mismatch. Confirm which region the org lives in and that your accounts domain and API domain both match it. This is far more common than an actually bad token.</p>



<h3 class="wp-block-heading">Row counts drift upward over time</h3>



<p class="wp-block-paragraph">Deletions are not being applied. Run the deleted-records endpoint for the last 60 days and see how much comes back, then run a full ID reconciliation to catch the rest.</p>



<h3 class="wp-block-heading">Bulk Read job finished but the file is gone</h3>



<p class="wp-block-paragraph">The download URL expires after roughly a day. Land the file as soon as the job reports complete, rather than deferring it to a downstream step that might not run until tomorrow.</p>



<h3 class="wp-block-heading">Athena fails with a schema mismatch across partitions</h3>



<p class="wp-block-paragraph">Type inference changed between runs, usually because a nullable field was all-null in one batch and populated in the next. Define the schema explicitly at write time and reprocess the affected partitions from raw.</p>



<h3 class="wp-block-heading">A new custom field never appears</h3>



<p class="wp-block-paragraph">You are selecting fields explicitly, which is correct, and nobody told you a field was added. Poll the module&#8217;s field metadata on a schedule and alert on changes, so schema drift is a notification rather than a discovery six weeks later.</p>



<h3 class="wp-block-heading">Frequent 429s</h3>



<p class="wp-block-paragraph">Either concurrency or the download rate limit, depending on which call is failing. Serialise the module extracts rather than fanning them all out at once, and add jittered backoff.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>An open-ended <code>Modified_Time</code> filter with no upper bound.</li>
<li>Offset pagination over a dataset that is being modified while you read it.</li>
<li>Advancing the watermark when the API call succeeds rather than when the data lands.</li>
<li>Keeping watermark state on an ephemeral runner.</li>
<li>Hardcoding the Zoho API domain and discovering data centres the hard way.</li>
<li>Never calling the deleted-records endpoint, so counts only ever grow.</li>
<li>Deferring the Bulk Read download until after the URL has expired.</li>
<li>Landing only Parquet with no raw layer, so reprocessing means re-extracting.</li>
<li>Letting the Parquet writer infer types per batch.</li>
<li>Flattening subforms into a wide table and double-counting parent rows.</li>
<li>Fanning out every module in parallel and collecting rate limits.</li>
<li>Sharing the pipeline&#8217;s connected app with other integrations, so nobody can attribute credit usage.</li>
<li>Storing the refresh token in an environment variable and forgetting it never expires.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Closed read windows with a deliberate lag, always.</li>
<li>Keyset pagination when you paginate at all, Bulk Read when you can avoid it.</li>
<li>Watermarks in durable storage, advanced only on confirmed landing.</li>
<li>Idempotent runs, so re-running a window is always safe.</li>
<li>Deleted-records endpoint every cycle, full ID reconciliation monthly.</li>
<li>Raw layer untouched, curated layer derived, partitions keyed by window end.</li>
<li>Explicit schemas on write, explicit field lists on read.</li>
<li>Its own connected app, narrow scopes, refresh token in Secrets Manager.</li>
<li>Jittered exponential backoff and serialised module extracts.</li>
<li>Alerting on schema drift and on row-count delta against Zoho.</li>
<li>Encryption and a retention policy on CRM data in S3 from day one, since it is personal data.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">What is the best way to export Zoho CRM data to S3?</h3>



<p class="wp-block-paragraph">Bulk Read for anything of size, on a schedule, with closed time windows. COQL for moderate volumes where you want real filtering. The plain records endpoint only for small modules. Land raw output first, convert to Parquet second.</p>



<h3 class="wp-block-heading">Why does my extract return different results each run?</h3>



<p class="wp-block-paragraph">Your query has no upper time bound, so the result set changes while you paginate through it. Bound both ends of the window and put the upper bound a few minutes in the past. That single change makes runs repeatable.</p>



<h3 class="wp-block-heading">How do I capture deleted records?</h3>



<p class="wp-block-paragraph">Call the module&#8217;s deleted-records endpoint each cycle. Recycle-bin deletions stay retrievable for 60 days and permanent deletions for 120, so a daily or hourly sync has plenty of margin. Add a monthly full ID reconciliation as a backstop.</p>



<h3 class="wp-block-heading">Will this exhaust my Zoho API credits?</h3>



<p class="wp-block-paragraph">Not if you use Bulk Read for the heavy lifting, since it does not draw on standard API limits the way per-record calls do. The credit pool is shared across the org, so give the pipeline its own connected app and monitor consumption rather than finding out when someone else&#8217;s integration breaks.</p>



<h3 class="wp-block-heading">How do I handle subforms and multi-select fields?</h3>



<p class="wp-block-paragraph">Decide per field. Flatten lookups into ID and name columns, split subforms and line items into their own child datasets keyed by parent ID, and keep genuinely peripheral arrays as JSON strings. Flattening a subform into the parent row is how you end up double-counting deals.</p>



<h3 class="wp-block-heading">Build it or buy a connector?</h3>



<p class="wp-block-paragraph">Buy it if Zoho CRM is one source among several and you have no unusual requirements; managed connectors handle pagination, deletes and schema drift, which is most of the work described here. Build it when you need control over the shape of the output, want to avoid per-row pricing, or already run the surrounding infrastructure.</p>



<h3 class="wp-block-heading">How fresh can the data realistically be?</h3>



<p class="wp-block-paragraph">Hourly is comfortable and suits nearly all reporting. Every fifteen minutes is achievable with COQL on modest modules. Anything closer to real time means webhooks or notification subscriptions rather than polling, which is a considerably larger commitment for a benefit most dashboards do not use.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">An extract that returns a different answer each time it runs is not a data quality problem, it is a read consistency problem, and no amount of downstream validation will fix it. Bound both ends of the window, put the upper bound in the past, paginate by key rather than offset, and advance the watermark only when the data is safely in S3.</p>



<p class="wp-block-paragraph">Get that right and the rest of the pipeline becomes ordinary engineering: land it, convert it, catalogue it, query it. Get it wrong and you will spend months chasing numbers that move every time you look at them.</p>



<h2 class="wp-block-heading">Want this built or reviewed?</h2>



<p class="wp-block-paragraph">Most CRM pipelines I get handed work fine on a quiet module and lose rows on the busy one, which is the hardest version to notice. Work I take on:</p>



<ul class="wp-block-list">
<li>Building a Zoho CRM to S3 pipeline end to end: OAuth, Bulk Read extraction, Parquet conversion, Glue catalog, Athena query layer.</li>
<li>Auditing an existing pipeline for read consistency and telling you whether it is silently dropping records.</li>
<li>Reworking incremental logic into closed windows, keyset pagination and durable watermarks.</li>
<li>Deletion handling and reconciliation jobs so row counts stay equal to the CRM.</li>
<li>Modelling nested CRM data into a schema that does not double-count.</li>
<li>Orchestration and alerting on AWS: EventBridge, Lambda or containers, Step Functions, and monitoring that catches drift early.</li>
</ul>



<p class="wp-block-paragraph">Send me your extract query and how you paginate, and I will tell you whether it can lose rows.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/zoho-crm-to-amazon-s3/">Run It Twice, Get Two Answers: Building an ETL Pipeline From Zoho CRM to Amazon S3</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
