<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Zoho CRM | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/zoho-crm/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/zoho-crm/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Tue, 06 Oct 2026 06:08:51 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Zoho CRM | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/zoho-crm/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Monitoring Zoho Integrations: Catching the Failures That Never Throw an Error</title>
		<link>https://john-nessime.com/blog/devops/monitoring-zoho-integrations/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 06 Oct 2026 06:08:51 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Workflow Automation]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Cardinality]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Deadman Alerting]]></category>
		<category><![CDATA[Embedded Metric Format]]></category>
		<category><![CDATA[Grafana]]></category>
		<category><![CDATA[Grafana Alloy]]></category>
		<category><![CDATA[LogQL]]></category>
		<category><![CDATA[Loki]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[Webhooks]]></category>
		<category><![CDATA[Zoho]]></category>
		<category><![CDATA[Zoho CRM]]></category>
		<category><![CDATA[Zoho Flow]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=395</guid>

					<description><![CDATA[<p>Zoho integrations rarely fail loudly. Channels expire after an hour, credits drain against a rolling window, and bulk writes return HTTP 200 while rejecting records. Here is how to instrument for absence using CloudWatch Embedded Metric Format, Loki's absent_over_time, and Grafana across both.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/monitoring-zoho-integrations/">Monitoring Zoho Integrations: Catching the Failures That Never Throw an Error</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message came in on a Tuesday: &#8220;Have we stopped getting leads from the website form?&#8221; Not an alert. Not a page. A salesperson noticing the pipeline looked thin.</p>



<p class="wp-block-paragraph">The sync worker was healthy. Zero errors in the last twenty-four hours. CPU flat, memory flat, no restarts, no failed jobs. Every dashboard was green because every dashboard was measuring the wrong thing. The integration had stopped receiving anything from Zoho four days earlier, and an integration that receives nothing produces no errors at all.</p>



<p class="wp-block-paragraph">That is the whole problem with monitoring Zoho integrations. The failures that hurt do not throw exceptions. They show up as silence, as slowly degrading throughput, or as HTTP 200 responses that quietly did not do what you asked. This post covers how to instrument for those three failure families using CloudWatch, Grafana and Loki, and why the obvious metrics miss all three.</p>



<h2 class="wp-block-heading">The three ways Zoho integrations actually break</h2>



<p class="wp-block-paragraph">Before touching a dashboard, it helps to be honest about what you are defending against. In my experience these fall into three groups, and they need genuinely different instrumentation:</p>



<ol class="wp-block-list">
<li><strong>Silent stop.</strong> Zoho stops sending you events. Nothing fails, because nothing happens.</li>

<li><strong>Silent throttle.</strong> You run out of API credits or hit a concurrency ceiling. Throughput degrades before anything hard-fails.</li>

<li><strong>Silent drift.</strong> Calls succeed, records do not land, and the two systems diverge without a single error log.</li>
</ol>



<p class="wp-block-paragraph">An error-rate panel catches none of these. That is not a criticism of error-rate panels, it is just what they are for.</p>



<h2 class="wp-block-heading">Failure family one: the integration that just stops</h2>



<p class="wp-block-paragraph">If you push data out of Zoho CRM using the Notifications API, you subscribe a <code>notify_url</code> to module events and Zoho posts to it when records change. It works well. It also has a detail that catches people out repeatedly.</p>



<p class="wp-block-paragraph">Notification channels expire. Zoho&#8217;s documentation is explicit: the expiry can be set to a maximum of one week from the time the channel is enabled, and if you do not specify one, or you ask for longer than a week, the default expiry is <strong>one hour</strong>. So the failure mode is not &#8220;someone forgot to renew after a year.&#8221; It is &#8220;someone deployed a subscription script without an expiry field and the integration worked beautifully for sixty minutes.&#8221;</p>



<p class="wp-block-paragraph">What makes this genuinely hard rather than just annoying is that there is no endpoint to list your currently active channels. You cannot ask Zoho what it thinks it is subscribed to. You can only ask whether events are still arriving. That single fact should shape your entire monitoring design: you are not monitoring for errors, you are monitoring for <em>absence</em>.</p>



<h3 class="wp-block-heading">Emit a heartbeat, then alarm on its absence</h3>



<p class="wp-block-paragraph">The receiving end of the webhook should emit a metric on every delivery, including deliveries you decide to discard. Embedded Metric Format is the cheapest way to do this from Lambda or a container, because you write structured JSON to stdout and CloudWatch extracts the metrics asynchronously. No <code>PutMetricData</code> call, no synchronous latency in your request path.</p>



<pre class="wp-block-code"><code>{
  "_aws": {
    "Timestamp": 1710000000000,
    "CloudWatchMetrics": [
      {
        "Namespace": "ZohoIntegration",
        "Dimensions": [["Module", "Channel"]],
        "Metrics": [
          { "Name": "NotificationsReceived", "Unit": "Count" },
          { "Name": "ChannelSecondsRemaining", "Unit": "Seconds" }
        ]
      }
    ]
  },
  "Module": "Leads",
  "Channel": "leads-prod",
  "NotificationsReceived": 1,
  "ChannelSecondsRemaining": 486000,
  "record_id": "5725767000002166520",
  "operation": "insert"
}</code></pre>



<p class="wp-block-paragraph">Note what is a dimension and what is not. <code>Module</code> and <code>Channel</code> are bounded sets, so they are safe dimensions. <code>record_id</code> sits in the log body only. EMF creates a separate custom metric for every unique dimension combination, so promoting a record ID to a dimension is how you turn a monitoring project into a billing incident. AWS says this plainly in the EMF documentation and it is worth reading twice before you ship.</p>



<p class="wp-block-paragraph">Now the alarm. A normal CloudWatch alarm goes to <code>INSUFFICIENT_DATA</code> when the metric stops reporting, and <code>INSUFFICIENT_DATA</code> does not notify. That is precisely backwards for this failure. You want missing data treated as a breach:</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-alarm 
  --alarm-name zoho-leads-notifications-silent 
  --namespace ZohoIntegration 
  --metric-name NotificationsReceived 
  --dimensions Name=Module,Value=Leads Name=Channel,Value=leads-prod 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 2 
  --threshold 1 
  --comparison-operator LessThanThreshold 
  --treat-missing-data breaching 
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:ops-alerts</code></pre>



<p class="wp-block-paragraph">The <code>--treat-missing-data breaching</code> flag is the entire point of that command. Without it, an integration that goes completely dead produces an alarm that sits quietly in an unknown state forever.</p>



<p class="wp-block-paragraph">On the Loki side the equivalent is <code>absent_over_time</code>, which returns 1 when a stream has produced no entries in the window. Run it from the Loki ruler so it fires even when nobody has a browser open:</p>



<pre class="wp-block-code"><code>groups:
  - name: zoho-integration
    interval: 5m
    rules:
      - alert: ZohoNotificationsSilent
        expr: absent_over_time({job="zoho-webhook", module="leads"}[2h])
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: "No Zoho notifications for Leads in 2h"
          description: "Channel may have expired. Re-subscribe and check channel_expiry."</code></pre>



<p class="wp-block-paragraph">One caveat worth knowing: <code>absent_over_time</code> only works for label combinations you name explicitly. It cannot tell you that a stream you never thought about has gone quiet. That is a real limitation, and it means your channel inventory has to live in your own configuration, not be discovered from Loki.</p>



<p class="wp-block-paragraph">Because you cannot list channels through the API, treat renewal as a scheduled job with its own deadman alert, and emit <code>ChannelSecondsRemaining</code> at each renewal. Alarm when it drops below roughly a day. That gives you warning before the silence starts rather than after somebody notices.</p>



<h2 class="wp-block-heading">Failure family two: throttling that is not rate limiting</h2>



<p class="wp-block-paragraph">Most engineers instrument Zoho as if it were a requests-per-minute API. It is not, and that mismatch produces dashboards that look fine right up until calls start failing.</p>



<p class="wp-block-paragraph">Zoho CRM meters API usage in <strong>credits</strong>, not calls. Different operations deduct different amounts, and bulk operations are metered per batch of records rather than per request. The budget resets on a rolling twenty-four hour window measured from each call, not at midnight. If your dashboard has a daily reset at 00:00, it is drawing a line that does not exist.</p>



<h3 class="wp-block-heading">The gauge that appears halfway through the tank</h3>



<p class="wp-block-paragraph">Here is the detail that makes credit monitoring awkward. Zoho returns remaining credits in an <code>X-API-CREDITS-REMAINING</code> response header, but only once usage has reached fifty percent or more of the daily allowance. Below that threshold the header simply is not there.</p>



<p class="wp-block-paragraph">Read that again in monitoring terms. An absent header means either &#8220;you are comfortably under half your budget&#8221; or &#8220;your instrumentation is broken and you have no idea.&#8221; Those two states look identical, and one of them is fine while the other is a blind spot. So record both facts separately:</p>



<ul class="wp-block-list">
<li>A counter for responses where the header was <em>present</em>, which effectively tells you when you crossed the halfway mark.</li>

<li>A gauge of the value itself, which only has meaning while the header exists.</li>

<li>Your own client-side count of calls made, which works regardless and is the only thing you control.</li>
</ul>



<p class="wp-block-paragraph">The first crossing of that threshold is a useful early warning in its own right. Alert on the transition, not just on a low remaining value.</p>



<h3 class="wp-block-heading">Concurrency is a separate ceiling</h3>



<p class="wp-block-paragraph">Alongside credits, Zoho applies a concurrency limit: the number of calls active simultaneously per org. There is also a sub-concurrency limit covering the expensive operations such as COQL queries, composite requests, lead conversion, and inserts or updates above a small record count. Exceeding either returns <code>TOO_MANY_REQUESTS</code>.</p>



<p class="wp-block-paragraph">Two consequences matter for monitoring. First, since the limit is on simultaneous calls rather than calls per unit time, a requests-per-minute graph will never predict a concurrency failure. You need a gauge of in-flight requests, incremented before the call and decremented in a <code>finally</code> block. Second, the sub-concurrency ceiling is the same across every CRM edition, so upgrading the plan raises the general limit but does not buy headroom on the expensive calls. Batch those and queue them rather than fanning out.</p>



<p class="wp-block-paragraph">One more source of consumption that surprises people: integration tasks inside Deluge functions consume credits from the same org budget as your external API calls. If your credit graph climbs while your service is idle, someone&#8217;s workflow rule is spending your budget. Zoho&#8217;s API Dashboard breaks consumption down by application and function, which is where to look before blaming your own code.</p>



<h2 class="wp-block-heading">Failure family three: successful calls that did nothing</h2>



<p class="wp-block-paragraph">Bulk and multi-record operations return a per-record status inside the response body. The HTTP status can be perfectly healthy while individual records were rejected for validation rules, mandatory fields, duplicate checks or picklist values that do not exist in the target layout.</p>



<p class="wp-block-paragraph">If your client only checks the HTTP status, you will log a success, increment a success counter, and move on. Weeks later somebody asks why the CRM is missing a slice of records that your logs insist were written.</p>



<p class="wp-block-paragraph">The fix is to stop counting requests and start counting records, on both sides:</p>



<ul class="wp-block-list">
<li><strong>Records submitted</strong> versus <strong>records confirmed</strong> in the per-record response status.</li>

<li>A breakdown of rejection reasons as a low-cardinality label, so a spike in one specific validation error is visible.</li>

<li>A periodic reconciliation job comparing source counts against Zoho counts for a bounded window, which is the only thing that catches drift you failed to anticipate.</li>
</ul>



<p class="wp-block-paragraph">In LogQL, the divergence between the two counters is the panel that actually earns its place on the wall:</p>



<pre class="wp-block-code"><code>sum(count_over_time({job="zoho-sync"} | json | status = "submitted" [1h]))
  -
sum(count_over_time({job="zoho-sync"} | json | status = "confirmed" [1h]))</code></pre>



<p class="wp-block-paragraph">A healthy integration holds that near zero. A number that grows steadily is drift, and it is visible weeks before anyone in sales notices.</p>



<h2 class="wp-block-heading">Deciding which signal goes where</h2>



<p class="wp-block-paragraph">Running CloudWatch, Grafana and Loki together only helps if each one has a job. The split I reach for first:</p>



<ul class="wp-block-list">
<li><strong>CloudWatch</strong> holds the numbers you alarm on, especially where the workload already runs in AWS and EMF gives you metrics for free out of log output.</li>

<li><strong>Loki</strong> holds the full request and response context: which record, which module, which validation error, which channel. This is what you actually read at three in the morning.</li>

<li><strong>Grafana</strong> is the single pane over both, with CloudWatch and Loki added as separate data sources so one dashboard can show the credit gauge next to the rejection reasons.</li>
</ul>



<p class="wp-block-paragraph">The trap that spans both backends is cardinality, and it bites in the same way for different reasons. In CloudWatch, each unique dimension combination becomes a billable custom metric. In Loki, each unique label combination becomes a separate stream, and Loki was designed for a small number of long-lived streams. Record IDs, email addresses and timestamps belong in neither.</p>



<p class="wp-block-paragraph">Loki&#8217;s answer is structured metadata: high-cardinality fields attached to log lines without creating streams or entering the index. It needs enabling per tenant:</p>



<pre class="wp-block-code"><code>limits_config:
  allow_structured_metadata: true
  volume_enabled: true</code></pre>



<p class="wp-block-paragraph">Put <code>module</code>, <code>environment</code> and <code>job</code> in labels. Put the Zoho record ID and the correlation ID in structured metadata. Query with a label selector first and filter on the rest.</p>



<p class="wp-block-paragraph">On collection, one scheduling note that does age: Promtail reached end of life on 2 March 2026, with its collection capabilities folded into Grafana Alloy. If you are still shipping through Promtail, that migration is now maintenance debt rather than an optional upgrade, and Grafana published a conversion tool for existing configurations. New pipelines should start on Alloy.</p>



<p class="wp-block-paragraph">Where you run all this is a real decision. Self-hosting Loki on a VPS from a provider like Contabo or InterServer is cheap and entirely workable for a single integration&#8217;s log volume, and it keeps the data under your control. Grafana Cloud or Amazon Managed Grafana removes the operational surface if you would rather not babysit an ingester. Both are defensible. The honest trade-off is that self-hosted Loki is easy to stand up and less easy to keep healthy once retention and compaction start to matter.</p>



<h2 class="wp-block-heading">Troubleshooting the monitoring itself</h2>



<ul class="wp-block-list">
<li><strong>EMF metrics never appear.</strong> The JSON must be a single line and structurally valid. A pretty-printed or truncated log line is ingested as an ordinary log event with no error, so check the raw event in the log stream rather than the metrics console.</li>

<li><strong>The alarm never fires despite obvious silence.</strong> Look at the alarm state. Sitting in <code>INSUFFICIENT_DATA</code> means the missing-data treatment was left at its default.</li>

<li><strong><code>absent_over_time</code> returns nothing.</strong> That is the success case. It emits a value only when the stream is empty, which makes it feel broken while everything is working.</li>

<li><strong>Loki queries slow to a crawl.</strong> Check stream count before blaming resources. A label added &#8220;temporarily&#8221; during debugging is the usual cause.</li>

<li><strong>Webhook deliveries stop after a deployment.</strong> Check whether the subscription script reran and created a channel without an expiry. An hour later it will look identical to a network problem.</li>

<li><strong>Credit consumption does not match your call count.</strong> Expected. Operations deduct different credit amounts, and Deluge integration tasks draw from the same pool.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Alerting on error rate alone, which is blind to every failure described above.</li>

<li>Creating notification channels without an explicit <code>channel_expiry</code> and inheriting the one-hour default.</li>

<li>Assuming an absent <code>X-API-CREDITS-REMAINING</code> header means healthy usage rather than sub-threshold usage.</li>

<li>Building a daily credit dashboard when the limit is a rolling window.</li>

<li>Putting record IDs into EMF dimensions or Loki labels.</li>

<li>Treating an HTTP 200 on a bulk write as confirmation that every record landed.</li>

<li>Running alert rules only in the Grafana UI, so nothing evaluates if that instance is down.</li>
</ul>



<h2 class="wp-block-heading">Best practices for monitoring Zoho integrations</h2>



<ol class="wp-block-list">
<li>Start with a deadman alert per channel. If you build one thing, build this.</li>

<li>Set <code>channel_expiry</code> explicitly, renew on a schedule, and monitor the renewal job as carefully as the integration.</li>

<li>Count records, not requests, on both sides of every write.</li>

<li>Keep a client-side call counter so you are never dependent on a header that may not be sent.</li>

<li>Gauge in-flight requests to see concurrency pressure before it becomes <code>TOO_MANY_REQUESTS</code>.</li>

<li>Attach a correlation ID to every record as structured metadata so one identifier joins logs and metrics.</li>

<li>Evaluate critical rules server-side, in the Loki ruler or as CloudWatch alarms.</li>

<li>Test by disabling a channel in a sandbox org and confirming the alert fires. An untested deadman alert is decoration.</li>
</ol>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Why did my Zoho webhook stop working on its own?</h3>



<p class="wp-block-paragraph">Almost always channel expiry. Zoho CRM notification channels last a maximum of one week, and if the expiry is omitted or set beyond that maximum, the channel defaults to one hour. Nothing is logged on your side because the deliveries simply stop arriving.</p>



<h3 class="wp-block-heading">Can I list my active Zoho notification channels?</h3>



<p class="wp-block-paragraph">There is no endpoint for enumerating active channels, which is a long-standing complaint in Zoho&#8217;s developer community. Keep the inventory in your own configuration and treat the receipt of events as the only reliable liveness signal.</p>



<h3 class="wp-block-heading">Do I need both CloudWatch and Loki?</h3>



<p class="wp-block-paragraph">No. Either can carry the whole job. The pairing makes sense when your integration already runs on AWS, because EMF gives you alarmable metrics from log output at no extra instrumentation cost, while Loki holds richer request context more cheaply. If you are starting fresh outside AWS, one backend is simpler and simpler is worth a lot.</p>



<h3 class="wp-block-heading">How do I monitor Zoho API credits properly?</h3>



<p class="wp-block-paragraph">Record the <code>X-API-CREDITS-REMAINING</code> header when it appears, alert on the moment it first appears in a window, and maintain your own call counter as the fallback. Model the budget as a rolling twenty-four hour window rather than a daily reset.</p>



<h3 class="wp-block-heading">What causes TOO_MANY_REQUESTS if I am under my credit limit?</h3>



<p class="wp-block-paragraph">Concurrency, not consumption. Credits and concurrency are independent ceilings. Expensive operations such as COQL queries and composite requests share a tighter sub-concurrency pool, so a burst of parallel queries can trip it while your credit balance is untouched.</p>



<h3 class="wp-block-heading">Should I still use Promtail for shipping integration logs?</h3>



<p class="wp-block-paragraph">No. Promtail reached end of life on 2 March 2026 and its functionality moved into Grafana Alloy. Existing deployments will keep running, but they are unsupported, and Grafana provides a tool to convert Promtail configuration to Alloy.</p>



<h3 class="wp-block-heading">How long should the deadman alert window be?</h3>



<p class="wp-block-paragraph">Base it on the quietest legitimate period for that module. A leads webhook that goes quiet overnight and at weekends needs a longer window than an order sync, or you will train the team to ignore it. Measure the real gap distribution for a fortnight before picking a number.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Monitoring Zoho integrations is mostly the discipline of detecting absence. Expired channels, exhausted credits and silently rejected records all present as nothing happening, and nothing happening is invisible to the instrumentation most teams reach for first.</p>



<p class="wp-block-paragraph">So invert the default. Alarm when the count is too low rather than when the errors are too high. Set <code>--treat-missing-data breaching</code>. Use <code>absent_over_time</code>. Count records on both sides and watch the gap. If you build only one thing from this post, build the deadman alert on notification delivery, because that is the failure that will otherwise be reported to you by a salesperson wondering where the leads went.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help instrumenting a Zoho integration?</h2>



<p class="wp-block-paragraph">Most of the work here is judgement about what to measure rather than volume of code. If you are staring at a sync that keeps going quiet, this is the sort of thing I take on:</p>



<ul class="wp-block-list">
<li>Building deadman alerting for Zoho notification channels, including renewal jobs that monitor themselves.</li>

<li>Instrumenting credit and concurrency headroom so throttling is visible before it becomes <code>TOO_MANY_REQUESTS</code>.</li>

<li>Designing EMF dimensions and Loki label schemas that will not turn into a cardinality bill.</li>

<li>Adding reconciliation jobs that compare record counts across systems and surface drift early.</li>

<li>Standing up Loki and Grafana Alloy, or migrating an existing Promtail pipeline that is now unsupported.</li>

<li>Reviewing an existing Grafana dashboard and cutting the panels that look reassuring but detect nothing.</li>
</ul>



<p class="wp-block-paragraph">Send me a sample webhook payload, a redacted response header set, or a screenshot of the dashboard you currently trust, and I will tell you what it would miss.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/monitoring-zoho-integrations/">Monitoring Zoho Integrations: Catching the Failures That Never Throw an Error</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Run It Twice, Get Two Answers: Building an ETL Pipeline From Zoho CRM to Amazon S3</title>
		<link>https://john-nessime.com/blog/devops/zoho-crm-to-amazon-s3/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 02 Aug 2026 18:16:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[Zoho]]></category>
		<category><![CDATA[Zoho CRM]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=86</guid>

					<description><![CDATA[<p>You re-run the same extract for the same window and get a different set of rows. Nothing errored. You were paginating a result set that kept changing while you read it. Here's how to build a Zoho CRM to S3 pipeline whose runs are repeatable, from closed read windows to Bulk Read and deletions.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/zoho-crm-to-amazon-s3/">Run It Twice, Get Two Answers: Building an ETL Pipeline From Zoho CRM to Amazon S3</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The monthly deal count in your warehouse is short by about thirty rows. You re-run the extract for exactly the same window, expecting to confirm the bug, and this time you get a different thirty. Run it a third time and the number moves again.</p>



<p class="wp-block-paragraph">Nothing is broken in the way you are looking for. The records exist in Zoho. Your job did not error. What happened is that you asked for every deal modified since your last watermark, sorted, and then walked through it page by page while sales reps kept working. Records got modified during the read, changed their position in the sort order, and shifted from page 4 to page 2 after you had already read page 2. So you never saw them.</p>



<p class="wp-block-paragraph">That is the failure that defines building an <strong>ETL pipeline from Zoho CRM to Amazon S3</strong>, and it does not look like a bug. It looks like flaky data. It only shows up on busy modules, it never produces an error, and re-running the job produces a different wrong answer, which is the single most confusing symptom in data engineering.</p>



<p class="wp-block-paragraph">The fix is one rule, and everything else in this post follows from it. This covers that rule, which extraction API to use and when, authentication and the data centre trap that wastes an afternoon, handling deletions (Zoho is genuinely good here), shaping nested CRM JSON for S3, and staying inside your API credits.</p>



<h2 class="wp-block-heading">The rule: bound both ends of the window</h2>



<p class="wp-block-paragraph">Almost every incremental extract is written like this:</p>



<pre class="wp-block-code"><code>Modified_Time &gt; {last_run}</code></pre>



<p class="wp-block-paragraph">That query has no upper bound, which means the result set keeps growing while you read it. You are paginating a moving target. Add an upper bound and put it slightly in the past:</p>



<pre class="wp-block-code"><code>Modified_Time &gt; {last_run} AND Modified_Time &lt; {now_minus_lag}</code></pre>



<p class="wp-block-paragraph">Now the set is frozen. Records modified during your run land after the upper bound and get picked up next time. The extract becomes repeatable: run it twice, get the same rows twice. That property is worth more than any amount of retry logic, because it means a failed run costs you nothing and a suspicious number can be checked by re-running.</p>



<p class="wp-block-paragraph">Two details that matter. The lag needs to comfortably exceed how long your extract takes plus any clock skew between you and Zoho; ten minutes is a sensible starting point and costs you ten minutes of freshness. And <strong>only advance the watermark after the entire window has landed in S3</strong>, never after the API call succeeds. Those are different moments, and the gap between them is where data goes missing.</p>



<pre class="wp-block-code"><code>#!/usr/bin/env bash
set -euo pipefail

LAG_MINUTES=10
T1=$(cat state/deals.watermark)
T2=$(date -u -d "-${LAG_MINUTES} minutes" +%Y-%m-%dT%H:%M:%S+00:00)

extract_window "$T1" "$T2"

# Watermark advances only once the data is durably in S3.
echo "$T2" &gt; state/deals.watermark</code></pre>



<p class="wp-block-paragraph">Keep that state somewhere durable and versioned, not on the box running the job. DynamoDB, Parameter Store, or a small object in S3 all work. A watermark file on an ephemeral runner is a watermark you will lose.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Authentication, and the trap that costs an afternoon</h2>



<p class="wp-block-paragraph">Zoho uses OAuth 2.0. You register a client in the Zoho API Console, get a client ID and secret, exchange a grant token for a refresh token once, and then trade the refresh token for short-lived access tokens from then on. Access tokens last an hour, so cache them rather than requesting one per call.</p>



<p class="wp-block-paragraph">Here is the part that catches everyone: <strong>Zoho runs multiple data centres and they are separate worlds.</strong> An account in the EU data centre authenticates against a different accounts domain and calls a different API domain than a US account. A token issued in one region is meaningless in another, and the error you get back does not say &#8220;wrong data centre&#8221;, it says the token is invalid.</p>



<p class="wp-block-paragraph">So make the domain a configuration value from the first line of code, never a hardcoded string. The examples below use the US domain. If your org lives elsewhere, both the accounts domain and the API domain change together.</p>



<p class="wp-block-paragraph">Two more things worth getting right at the start. Request the narrowest OAuth scopes that work, read-only on the modules you actually extract, because scopes are easy to widen later and awkward to explain in an audit. And store the refresh token in Secrets Manager rather than an environment variable, since it does not expire and is effectively a permanent key to your CRM.</p>



<h2 class="wp-block-heading">Which API to extract with</h2>



<p class="wp-block-paragraph">Three options, and the choice is mostly about volume.</p>



<h3 class="wp-block-heading">Bulk Read, for anything large</h3>



<p class="wp-block-paragraph">This is the right default for a lake. You POST a job description, Zoho runs the export server-side, and you poll for status or supply a callback URL. When it finishes you get a download URL for a ZIP of CSV. A single job handles up to 200,000 records, with paging beyond that, and it does not consume your standard API limits the way record-by-record calls do.</p>



<p class="wp-block-paragraph">The important property for our purposes: the pagination problem disappears. Zoho assembles the export on its side, so there is no page-by-page read for records to slip between.</p>



<pre class="wp-block-code"><code>curl "https://www.zohoapis.com/crm/bulk/v8/read" 
  -X POST 
  -H "Authorization: Zoho-oauthtoken $ACCESS_TOKEN" 
  -H "Content-Type: application/json" 
  -d @job.json</code></pre>



<pre class="wp-block-code"><code>{
  "query": {
    "module": { "api_name": "Deals" },
    "fields": ["id", "Deal_Name", "Amount", "Stage", "Modified_Time"],
    "criteria": {
      "group_operator": "and",
      "group": [
        {
          "field": { "api_name": "Modified_Time" },
          "comparator": "greater_than",
          "value": "{T1}"
        },
        {
          "field": { "api_name": "Modified_Time" },
          "comparator": "less_than",
          "value": "{T2}"
        }
      ]
    },
    "page": 1
  }
}</code></pre>



<p class="wp-block-paragraph">Two limits to design around. The download URL is only valid for about a day, so fetch and land the file promptly rather than queueing it for a later step. And downloads are rate limited per minute, so if you fan out across many modules at once you will start collecting 429s.</p>



<h3 class="wp-block-heading">COQL, for moderate volumes and real filtering</h3>



<p class="wp-block-paragraph">Zoho&#8217;s SQL-like query API. Up to 2,000 records per call and up to 100,000 records total per unique criteria through pagination. Past that, Zoho&#8217;s own documentation tells you to use Bulk Read, which is good advice to take rather than work around.</p>



<p class="wp-block-paragraph">If you do paginate COQL, paginate by key rather than by offset. Sort by <code>id</code> and carry the last ID you saw into the next call, so shifting records cannot cause a skip:</p>



<pre class="wp-block-code"><code>{
  "select_query": "select id, Deal_Name, Amount, Stage, Modified_Time from Deals where (Modified_Time &gt; '{T1}' and Modified_Time &lt; '{T2}') and id &gt; {last_id} order by id asc limit 2000"
}</code></pre>



<p class="wp-block-paragraph">Note the shape: a closed time window that freezes the set, plus a keyset cursor that walks it deterministically. Offset pagination gives you neither. Zoho&#8217;s own documentation uses this pattern in its examples, which is a fair hint about what they expect.</p>



<h3 class="wp-block-heading">Get Records, for small modules and metadata</h3>



<p class="wp-block-paragraph">The plain module endpoint returns up to 200 records a page and supports an <code>If-Modified-Since</code> header for incremental reads. Fine for lookup tables and small custom modules. Not what you want pointed at a Deals module with six figures of rows.</p>



<p class="wp-block-paragraph">Worth saying plainly: if all of this sounds like a fortnight of work you would rather not own, a managed connector from Fivetran or Airbyte solves the extraction half and you spend your time on the modelling instead. Price it against your own hours honestly, because the build-it-yourself option is habitually costed at zero.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Deletions, which Zoho actually handles well</h2>



<p class="wp-block-paragraph">An incremental extract can never see a deletion, because a record that no longer exists cannot appear in a query for modified records. Most CRMs make you infer this. Zoho gives you a direct endpoint, and it is one of the nicer parts of the API.</p>



<pre class="wp-block-code"><code># type: all | recycle | permanent
curl "https://www.zohoapis.com/crm/v8/Deals/deleted?type=all&amp;per_page=200&amp;page=1" 
  -X GET 
  -H "Authorization: Zoho-oauthtoken $ACCESS_TOKEN" 
  -H "If-Modified-Since: {T1}"</code></pre>



<p class="wp-block-paragraph">The retention windows are generous: records in the recycle bin are retrievable for up to 60 days from deletion, and permanently deleted records for up to 120 days. Compared with warehouses where the detection window is a couple of weeks and not guaranteed, that is a lot of slack. It also means there is no excuse for missing a deletion, since any sane sync interval sits comfortably inside it.</p>



<p class="wp-block-paragraph">Run the deleted-records call as part of every extract cycle, using the same closed window, and mark the matching rows as deleted in your curated layer rather than removing them from raw. Soft-delete preserves the audit trail and lets you answer &#8220;when did this disappear&#8221; later.</p>



<p class="wp-block-paragraph">Even with a good endpoint, run a periodic full ID reconciliation as a backstop, monthly is plenty. Pull just the ID column for the whole module, diff it against the lake, and flag anything you still believe in that Zoho does not. It catches the cases the deletion endpoint does not describe cleanly, such as records merged or moved between modules.</p>



<h2 class="wp-block-heading">Shaping CRM data for S3</h2>



<p class="wp-block-paragraph">Land raw first, exactly as returned, in its own prefix. Then convert to Parquet in a second step. Skipping the raw layer feels efficient right up to the first time you need to reprocess a month with corrected logic.</p>



<pre class="wp-block-code"><code># Partition by the window's upper bound, not by a business date.
# One bad run is then one partition to replace.
s3://acme-lake/raw/zoho_crm/deals/window_end=YYYY-MM-DDTHH/
s3://acme-lake/curated/zoho_crm/deals/window_end=YYYY-MM-DDTHH/</code></pre>



<p class="wp-block-paragraph">Zoho returns nested structures that do not map onto a flat table. Lookup fields come back as objects with an ID and a display name. Multi-select fields and tags come back as arrays. Subforms come back as arrays of objects. You have three choices per field and should make each one deliberately:</p>



<ul class="wp-block-list">
<li><strong>Flatten into columns.</strong> A lookup becomes <code>account_id</code> and <code>account_name</code>. Correct for anything you filter or join on.</li>
<li><strong>Split into a child table.</strong> Subforms and line items become their own Parquet dataset keyed by parent ID. Correct when the nested rows are things people count.</li>
<li><strong>Keep as a JSON string column.</strong> Fine for rarely-queried arrays, and Athena can parse it on demand. Not fine for anything in a regular report.</li>
</ul>



<p class="wp-block-paragraph">Pin your types explicitly during conversion rather than letting the writer infer them from a batch. Inference is the reason a column is a string in January&#8217;s files and a double in February&#8217;s, and Athena will happily read both and quietly fail to reconcile them.</p>



<p class="wp-block-paragraph">Register the result in the Glue Data Catalog and query with Athena. If you expect to apply updates and deletes in place rather than rebuilding partitions, Apache Iceberg earns its extra setup: row-level operations and schema evolution are exactly what a CRM feed generates.</p>



<p class="wp-block-paragraph">On the AWS side, the orchestration is unremarkable and should stay that way: EventBridge on a schedule, Lambda or a small container for the extract, Step Functions if you need to poll a Bulk Read job and branch on the result. Resist Glue jobs for the extraction itself unless you are already deep in Glue; a Python container you can run locally is easier to debug at seven in the morning.</p>



<h2 class="wp-block-heading">API credits</h2>



<p class="wp-block-paragraph">Zoho meters API usage as credits, allocated by edition and user count, over a rolling 24-hour window, and shared with every other integration on the org. There are also concurrency limits, so hammering the API in parallel fails differently from exceeding your daily allowance.</p>



<ul class="wp-block-list">
<li><strong>Bulk Read for volume.</strong> It sidesteps the standard limits and is the whole reason the API exists.</li>
<li><strong>Ask for fewer fields.</strong> Selecting only the columns you actually land reduces payload and processing on both sides.</li>
<li><strong>Mind the COQL limit value.</strong> Credits scale with the page size you request, so grabbing 2,000 rows costs more than grabbing 200. Larger pages are still usually the better trade, but know you are making it.</li>
<li><strong>Give the pipeline its own connected app and user.</strong> Then consumption is attributable when someone asks who exhausted the credits.</li>
<li><strong>Back off properly on 429.</strong> Exponential backoff with jitter, not a fixed sleep, and treat rate limiting as an expected condition rather than an error.</li>
</ul>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Two runs of the same window return different rows</h3>



<p class="wp-block-paragraph">Open-ended window, offset pagination, or both. Add the upper bound, switch to keyset pagination or Bulk Read, and the symptom disappears. This is the one to check before anything else, because it makes every other investigation unreliable.</p>



<h3 class="wp-block-heading">Invalid token, but the credentials are definitely right</h3>



<p class="wp-block-paragraph">Data centre mismatch. Confirm which region the org lives in and that your accounts domain and API domain both match it. This is far more common than an actually bad token.</p>



<h3 class="wp-block-heading">Row counts drift upward over time</h3>



<p class="wp-block-paragraph">Deletions are not being applied. Run the deleted-records endpoint for the last 60 days and see how much comes back, then run a full ID reconciliation to catch the rest.</p>



<h3 class="wp-block-heading">Bulk Read job finished but the file is gone</h3>



<p class="wp-block-paragraph">The download URL expires after roughly a day. Land the file as soon as the job reports complete, rather than deferring it to a downstream step that might not run until tomorrow.</p>



<h3 class="wp-block-heading">Athena fails with a schema mismatch across partitions</h3>



<p class="wp-block-paragraph">Type inference changed between runs, usually because a nullable field was all-null in one batch and populated in the next. Define the schema explicitly at write time and reprocess the affected partitions from raw.</p>



<h3 class="wp-block-heading">A new custom field never appears</h3>



<p class="wp-block-paragraph">You are selecting fields explicitly, which is correct, and nobody told you a field was added. Poll the module&#8217;s field metadata on a schedule and alert on changes, so schema drift is a notification rather than a discovery six weeks later.</p>



<h3 class="wp-block-heading">Frequent 429s</h3>



<p class="wp-block-paragraph">Either concurrency or the download rate limit, depending on which call is failing. Serialise the module extracts rather than fanning them all out at once, and add jittered backoff.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>An open-ended <code>Modified_Time</code> filter with no upper bound.</li>
<li>Offset pagination over a dataset that is being modified while you read it.</li>
<li>Advancing the watermark when the API call succeeds rather than when the data lands.</li>
<li>Keeping watermark state on an ephemeral runner.</li>
<li>Hardcoding the Zoho API domain and discovering data centres the hard way.</li>
<li>Never calling the deleted-records endpoint, so counts only ever grow.</li>
<li>Deferring the Bulk Read download until after the URL has expired.</li>
<li>Landing only Parquet with no raw layer, so reprocessing means re-extracting.</li>
<li>Letting the Parquet writer infer types per batch.</li>
<li>Flattening subforms into a wide table and double-counting parent rows.</li>
<li>Fanning out every module in parallel and collecting rate limits.</li>
<li>Sharing the pipeline&#8217;s connected app with other integrations, so nobody can attribute credit usage.</li>
<li>Storing the refresh token in an environment variable and forgetting it never expires.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Closed read windows with a deliberate lag, always.</li>
<li>Keyset pagination when you paginate at all, Bulk Read when you can avoid it.</li>
<li>Watermarks in durable storage, advanced only on confirmed landing.</li>
<li>Idempotent runs, so re-running a window is always safe.</li>
<li>Deleted-records endpoint every cycle, full ID reconciliation monthly.</li>
<li>Raw layer untouched, curated layer derived, partitions keyed by window end.</li>
<li>Explicit schemas on write, explicit field lists on read.</li>
<li>Its own connected app, narrow scopes, refresh token in Secrets Manager.</li>
<li>Jittered exponential backoff and serialised module extracts.</li>
<li>Alerting on schema drift and on row-count delta against Zoho.</li>
<li>Encryption and a retention policy on CRM data in S3 from day one, since it is personal data.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">What is the best way to export Zoho CRM data to S3?</h3>



<p class="wp-block-paragraph">Bulk Read for anything of size, on a schedule, with closed time windows. COQL for moderate volumes where you want real filtering. The plain records endpoint only for small modules. Land raw output first, convert to Parquet second.</p>



<h3 class="wp-block-heading">Why does my extract return different results each run?</h3>



<p class="wp-block-paragraph">Your query has no upper time bound, so the result set changes while you paginate through it. Bound both ends of the window and put the upper bound a few minutes in the past. That single change makes runs repeatable.</p>



<h3 class="wp-block-heading">How do I capture deleted records?</h3>



<p class="wp-block-paragraph">Call the module&#8217;s deleted-records endpoint each cycle. Recycle-bin deletions stay retrievable for 60 days and permanent deletions for 120, so a daily or hourly sync has plenty of margin. Add a monthly full ID reconciliation as a backstop.</p>



<h3 class="wp-block-heading">Will this exhaust my Zoho API credits?</h3>



<p class="wp-block-paragraph">Not if you use Bulk Read for the heavy lifting, since it does not draw on standard API limits the way per-record calls do. The credit pool is shared across the org, so give the pipeline its own connected app and monitor consumption rather than finding out when someone else&#8217;s integration breaks.</p>



<h3 class="wp-block-heading">How do I handle subforms and multi-select fields?</h3>



<p class="wp-block-paragraph">Decide per field. Flatten lookups into ID and name columns, split subforms and line items into their own child datasets keyed by parent ID, and keep genuinely peripheral arrays as JSON strings. Flattening a subform into the parent row is how you end up double-counting deals.</p>



<h3 class="wp-block-heading">Build it or buy a connector?</h3>



<p class="wp-block-paragraph">Buy it if Zoho CRM is one source among several and you have no unusual requirements; managed connectors handle pagination, deletes and schema drift, which is most of the work described here. Build it when you need control over the shape of the output, want to avoid per-row pricing, or already run the surrounding infrastructure.</p>



<h3 class="wp-block-heading">How fresh can the data realistically be?</h3>



<p class="wp-block-paragraph">Hourly is comfortable and suits nearly all reporting. Every fifteen minutes is achievable with COQL on modest modules. Anything closer to real time means webhooks or notification subscriptions rather than polling, which is a considerably larger commitment for a benefit most dashboards do not use.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">An extract that returns a different answer each time it runs is not a data quality problem, it is a read consistency problem, and no amount of downstream validation will fix it. Bound both ends of the window, put the upper bound in the past, paginate by key rather than offset, and advance the watermark only when the data is safely in S3.</p>



<p class="wp-block-paragraph">Get that right and the rest of the pipeline becomes ordinary engineering: land it, convert it, catalogue it, query it. Get it wrong and you will spend months chasing numbers that move every time you look at them.</p>



<h2 class="wp-block-heading">Want this built or reviewed?</h2>



<p class="wp-block-paragraph">Most CRM pipelines I get handed work fine on a quiet module and lose rows on the busy one, which is the hardest version to notice. Work I take on:</p>



<ul class="wp-block-list">
<li>Building a Zoho CRM to S3 pipeline end to end: OAuth, Bulk Read extraction, Parquet conversion, Glue catalog, Athena query layer.</li>
<li>Auditing an existing pipeline for read consistency and telling you whether it is silently dropping records.</li>
<li>Reworking incremental logic into closed windows, keyset pagination and durable watermarks.</li>
<li>Deletion handling and reconciliation jobs so row counts stay equal to the CRM.</li>
<li>Modelling nested CRM data into a schema that does not double-count.</li>
<li>Orchestration and alerting on AWS: EventBridge, Lambda or containers, Step Functions, and monitoring that catches drift early.</li>
</ul>



<p class="wp-block-paragraph">Send me your extract query and how you paginate, and I will tell you whether it can lose rows.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/zoho-crm-to-amazon-s3/">Run It Twice, Get Two Answers: Building an ETL Pipeline From Zoho CRM to Amazon S3</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
