<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>LogQL | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/logql/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/logql/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Fri, 06 Nov 2026 11:29:15 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>LogQL | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/logql/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Monitoring Zoho Integrations: Catching the Failures That Never Throw an Error</title>
		<link>https://john-nessime.com/blog/devops/monitoring-zoho-integrations/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 06 Oct 2026 06:08:51 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Workflow Automation]]></category>
		<category><![CDATA[Alerting]]></category>
		<category><![CDATA[Cardinality]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Deadman Alerting]]></category>
		<category><![CDATA[Embedded Metric Format]]></category>
		<category><![CDATA[Grafana]]></category>
		<category><![CDATA[Grafana Alloy]]></category>
		<category><![CDATA[LogQL]]></category>
		<category><![CDATA[Loki]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[Webhooks]]></category>
		<category><![CDATA[Zoho]]></category>
		<category><![CDATA[Zoho CRM]]></category>
		<category><![CDATA[Zoho Flow]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=395</guid>

					<description><![CDATA[<p>Zoho integrations rarely fail loudly. Channels expire after an hour, credits drain against a rolling window, and bulk writes return HTTP 200 while rejecting records. Here is how to instrument for absence using CloudWatch Embedded Metric Format, Loki's absent_over_time, and Grafana across both.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/monitoring-zoho-integrations/">Monitoring Zoho Integrations: Catching the Failures That Never Throw an Error</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message came in on a Tuesday: &#8220;Have we stopped getting leads from the website form?&#8221; Not an alert. Not a page. A salesperson noticing the pipeline looked thin.</p>



<p class="wp-block-paragraph">The sync worker was healthy. Zero errors in the last twenty-four hours. CPU flat, memory flat, no restarts, no failed jobs. Every dashboard was green because every dashboard was measuring the wrong thing. The integration had stopped receiving anything from Zoho four days earlier, and an integration that receives nothing produces no errors at all.</p>



<p class="wp-block-paragraph">That is the whole problem with monitoring Zoho integrations. The failures that hurt do not throw exceptions. They show up as silence, as slowly degrading throughput, or as HTTP 200 responses that quietly did not do what you asked. This post covers how to instrument for those three failure families using CloudWatch, Grafana and Loki, and why the obvious metrics miss all three.</p>



<h2 class="wp-block-heading">The three ways Zoho integrations actually break</h2>



<p class="wp-block-paragraph">Before touching a dashboard, it helps to be honest about what you are defending against. In my experience these fall into three groups, and they need genuinely different instrumentation:</p>



<ol class="wp-block-list">
<li><strong>Silent stop.</strong> Zoho stops sending you events. Nothing fails, because nothing happens.</li>

<li><strong>Silent throttle.</strong> You run out of API credits or hit a concurrency ceiling. Throughput degrades before anything hard-fails.</li>

<li><strong>Silent drift.</strong> Calls succeed, records do not land, and the two systems diverge without a single error log.</li>
</ol>



<p class="wp-block-paragraph">An error-rate panel catches none of these. That is not a criticism of error-rate panels, it is just what they are for.</p>



<h2 class="wp-block-heading">Failure family one: the integration that just stops</h2>



<p class="wp-block-paragraph">If you push data out of Zoho CRM using the Notifications API, you subscribe a <code>notify_url</code> to module events and Zoho posts to it when records change. It works well. It also has a detail that catches people out repeatedly.</p>



<p class="wp-block-paragraph">Notification channels expire. Zoho&#8217;s documentation is explicit: the expiry can be set to a maximum of one week from the time the channel is enabled, and if you do not specify one, or you ask for longer than a week, the default expiry is <strong>one hour</strong>. So the failure mode is not &#8220;someone forgot to renew after a year.&#8221; It is &#8220;someone deployed a subscription script without an expiry field and the integration worked beautifully for sixty minutes.&#8221;</p>



<p class="wp-block-paragraph">What makes this genuinely hard rather than just annoying is that there is no endpoint to list your currently active channels. You cannot ask Zoho what it thinks it is subscribed to. You can only ask whether events are still arriving. That single fact should shape your entire monitoring design: you are not monitoring for errors, you are monitoring for <em>absence</em>.</p>



<h3 class="wp-block-heading">Emit a heartbeat, then alarm on its absence</h3>



<p class="wp-block-paragraph">The receiving end of the webhook should emit a metric on every delivery, including deliveries you decide to discard. Embedded Metric Format is the cheapest way to do this from Lambda or a container, because you write structured JSON to stdout and CloudWatch extracts the metrics asynchronously. No <code>PutMetricData</code> call, no synchronous latency in your request path.</p>



<pre class="wp-block-code"><code>{
  "_aws": {
    "Timestamp": 1710000000000,
    "CloudWatchMetrics": [
      {
        "Namespace": "ZohoIntegration",
        "Dimensions": [["Module", "Channel"]],
        "Metrics": [
          { "Name": "NotificationsReceived", "Unit": "Count" },
          { "Name": "ChannelSecondsRemaining", "Unit": "Seconds" }
        ]
      }
    ]
  },
  "Module": "Leads",
  "Channel": "leads-prod",
  "NotificationsReceived": 1,
  "ChannelSecondsRemaining": 486000,
  "record_id": "5725767000002166520",
  "operation": "insert"
}</code></pre>



<p class="wp-block-paragraph">Note what is a dimension and what is not. <code>Module</code> and <code>Channel</code> are bounded sets, so they are safe dimensions. <code>record_id</code> sits in the log body only. EMF creates a separate custom metric for every unique dimension combination, so promoting a record ID to a dimension is how you turn a monitoring project into a billing incident. AWS says this plainly in the EMF documentation and it is worth reading twice before you ship.</p>



<p class="wp-block-paragraph">Now the alarm. A normal CloudWatch alarm goes to <code>INSUFFICIENT_DATA</code> when the metric stops reporting, and <code>INSUFFICIENT_DATA</code> does not notify. That is precisely backwards for this failure. You want missing data treated as a breach:</p>



<pre class="wp-block-code"><code>aws cloudwatch put-metric-alarm 
  --alarm-name zoho-leads-notifications-silent 
  --namespace ZohoIntegration 
  --metric-name NotificationsReceived 
  --dimensions Name=Module,Value=Leads Name=Channel,Value=leads-prod 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 2 
  --threshold 1 
  --comparison-operator LessThanThreshold 
  --treat-missing-data breaching 
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:ops-alerts</code></pre>



<p class="wp-block-paragraph">The <code>--treat-missing-data breaching</code> flag is the entire point of that command. Without it, an integration that goes completely dead produces an alarm that sits quietly in an unknown state forever.</p>



<p class="wp-block-paragraph">On the Loki side the equivalent is <code>absent_over_time</code>, which returns 1 when a stream has produced no entries in the window. Run it from the Loki ruler so it fires even when nobody has a browser open:</p>



<pre class="wp-block-code"><code>groups:
  - name: zoho-integration
    interval: 5m
    rules:
      - alert: ZohoNotificationsSilent
        expr: absent_over_time({job="zoho-webhook", module="leads"}[2h])
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: "No Zoho notifications for Leads in 2h"
          description: "Channel may have expired. Re-subscribe and check channel_expiry."</code></pre>



<p class="wp-block-paragraph">One caveat worth knowing: <code>absent_over_time</code> only works for label combinations you name explicitly. It cannot tell you that a stream you never thought about has gone quiet. That is a real limitation, and it means your channel inventory has to live in your own configuration, not be discovered from Loki.</p>



<p class="wp-block-paragraph">Because you cannot list channels through the API, treat renewal as a scheduled job with its own deadman alert, and emit <code>ChannelSecondsRemaining</code> at each renewal. Alarm when it drops below roughly a day. That gives you warning before the silence starts rather than after somebody notices.</p>



<h2 class="wp-block-heading">Failure family two: throttling that is not rate limiting</h2>



<p class="wp-block-paragraph">Most engineers instrument Zoho as if it were a requests-per-minute API. It is not, and that mismatch produces dashboards that look fine right up until calls start failing.</p>



<p class="wp-block-paragraph">Zoho CRM meters API usage in <strong>credits</strong>, not calls. Different operations deduct different amounts, and bulk operations are metered per batch of records rather than per request. The budget resets on a rolling twenty-four hour window measured from each call, not at midnight. If your dashboard has a daily reset at 00:00, it is drawing a line that does not exist.</p>



<h3 class="wp-block-heading">The gauge that appears halfway through the tank</h3>



<p class="wp-block-paragraph">Here is the detail that makes credit monitoring awkward. Zoho returns remaining credits in an <code>X-API-CREDITS-REMAINING</code> response header, but only once usage has reached fifty percent or more of the daily allowance. Below that threshold the header simply is not there.</p>



<p class="wp-block-paragraph">Read that again in monitoring terms. An absent header means either &#8220;you are comfortably under half your budget&#8221; or &#8220;your instrumentation is broken and you have no idea.&#8221; Those two states look identical, and one of them is fine while the other is a blind spot. So record both facts separately:</p>



<ul class="wp-block-list">
<li>A counter for responses where the header was <em>present</em>, which effectively tells you when you crossed the halfway mark.</li>

<li>A gauge of the value itself, which only has meaning while the header exists.</li>

<li>Your own client-side count of calls made, which works regardless and is the only thing you control.</li>
</ul>



<p class="wp-block-paragraph">The first crossing of that threshold is a useful early warning in its own right. Alert on the transition, not just on a low remaining value.</p>



<h3 class="wp-block-heading">Concurrency is a separate ceiling</h3>



<p class="wp-block-paragraph">Alongside credits, Zoho applies a concurrency limit: the number of calls active simultaneously per org. There is also a sub-concurrency limit covering the expensive operations such as COQL queries, composite requests, lead conversion, and inserts or updates above a small record count. Exceeding either returns <code>TOO_MANY_REQUESTS</code>.</p>



<p class="wp-block-paragraph">Two consequences matter for monitoring. First, since the limit is on simultaneous calls rather than calls per unit time, a requests-per-minute graph will never predict a concurrency failure. You need a gauge of in-flight requests, incremented before the call and decremented in a <code>finally</code> block. Second, the sub-concurrency ceiling is the same across every CRM edition, so upgrading the plan raises the general limit but does not buy headroom on the expensive calls. Batch those and queue them rather than fanning out.</p>



<p class="wp-block-paragraph">One more source of consumption that surprises people: integration tasks inside Deluge functions consume credits from the same org budget as your external API calls. If your credit graph climbs while your service is idle, someone&#8217;s workflow rule is spending your budget. Zoho&#8217;s API Dashboard breaks consumption down by application and function, which is where to look before blaming your own code.</p>



<h2 class="wp-block-heading">Failure family three: successful calls that did nothing</h2>



<p class="wp-block-paragraph">Bulk and multi-record operations return a per-record status inside the response body. The HTTP status can be perfectly healthy while individual records were rejected for validation rules, mandatory fields, duplicate checks or picklist values that do not exist in the target layout.</p>



<p class="wp-block-paragraph">If your client only checks the HTTP status, you will log a success, increment a success counter, and move on. Weeks later somebody asks why the CRM is missing a slice of records that your logs insist were written.</p>



<p class="wp-block-paragraph">The fix is to stop counting requests and start counting records, on both sides:</p>



<ul class="wp-block-list">
<li><strong>Records submitted</strong> versus <strong>records confirmed</strong> in the per-record response status.</li>

<li>A breakdown of rejection reasons as a low-cardinality label, so a spike in one specific validation error is visible.</li>

<li>A periodic reconciliation job comparing source counts against Zoho counts for a bounded window, which is the only thing that catches drift you failed to anticipate.</li>
</ul>



<p class="wp-block-paragraph">In LogQL, the divergence between the two counters is the panel that actually earns its place on the wall:</p>



<pre class="wp-block-code"><code>sum(count_over_time({job="zoho-sync"} | json | status = "submitted" [1h]))
  -
sum(count_over_time({job="zoho-sync"} | json | status = "confirmed" [1h]))</code></pre>



<p class="wp-block-paragraph">A healthy integration holds that near zero. A number that grows steadily is drift, and it is visible weeks before anyone in sales notices.</p>



<h2 class="wp-block-heading">Deciding which signal goes where</h2>



<p class="wp-block-paragraph">Running CloudWatch, Grafana and Loki together only helps if each one has a job. The split I reach for first:</p>



<ul class="wp-block-list">
<li><strong>CloudWatch</strong> holds the numbers you alarm on, especially where the workload already runs in AWS and EMF gives you metrics for free out of log output.</li>

<li><strong>Loki</strong> holds the full request and response context: which record, which module, which validation error, which channel. This is what you actually read at three in the morning.</li>

<li><strong>Grafana</strong> is the single pane over both, with CloudWatch and Loki added as separate data sources so one dashboard can show the credit gauge next to the rejection reasons.</li>
</ul>



<p class="wp-block-paragraph">The trap that spans both backends is cardinality, and it bites in the same way for different reasons. In CloudWatch, each unique dimension combination becomes a billable custom metric. In Loki, each unique label combination becomes a separate stream, and Loki was designed for a small number of long-lived streams. Record IDs, email addresses and timestamps belong in neither.</p>



<p class="wp-block-paragraph">Loki&#8217;s answer is structured metadata: high-cardinality fields attached to log lines without creating streams or entering the index. It needs enabling per tenant:</p>



<pre class="wp-block-code"><code>limits_config:
  allow_structured_metadata: true
  volume_enabled: true</code></pre>



<p class="wp-block-paragraph">Put <code>module</code>, <code>environment</code> and <code>job</code> in labels. Put the Zoho record ID and the correlation ID in structured metadata. Query with a label selector first and filter on the rest.</p>



<p class="wp-block-paragraph">On collection, one scheduling note that does age: Promtail reached end of life on 2 March 2026, with its collection capabilities folded into Grafana Alloy. If you are still shipping through Promtail, that migration is now maintenance debt rather than an optional upgrade, and Grafana published a conversion tool for existing configurations. New pipelines should start on Alloy.</p>



<p class="wp-block-paragraph">Where you run all this is a real decision. Self-hosting Loki on a VPS from a provider like Contabo or InterServer is cheap and entirely workable for a single integration&#8217;s log volume, and it keeps the data under your control. Grafana Cloud or Amazon Managed Grafana removes the operational surface if you would rather not babysit an ingester. Both are defensible. The honest trade-off is that self-hosted Loki is easy to stand up and less easy to keep healthy once retention and compaction start to matter.</p>



<h2 class="wp-block-heading">Troubleshooting the monitoring itself</h2>



<ul class="wp-block-list">
<li><strong>EMF metrics never appear.</strong> The JSON must be a single line and structurally valid. A pretty-printed or truncated log line is ingested as an ordinary log event with no error, so check the raw event in the log stream rather than the metrics console.</li>

<li><strong>The alarm never fires despite obvious silence.</strong> Look at the alarm state. Sitting in <code>INSUFFICIENT_DATA</code> means the missing-data treatment was left at its default.</li>

<li><strong><code>absent_over_time</code> returns nothing.</strong> That is the success case. It emits a value only when the stream is empty, which makes it feel broken while everything is working.</li>

<li><strong>Loki queries slow to a crawl.</strong> Check stream count before blaming resources. A label added &#8220;temporarily&#8221; during debugging is the usual cause.</li>

<li><strong>Webhook deliveries stop after a deployment.</strong> Check whether the subscription script reran and created a channel without an expiry. An hour later it will look identical to a network problem.</li>

<li><strong>Credit consumption does not match your call count.</strong> Expected. Operations deduct different credit amounts, and Deluge integration tasks draw from the same pool.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Alerting on error rate alone, which is blind to every failure described above.</li>

<li>Creating notification channels without an explicit <code>channel_expiry</code> and inheriting the one-hour default.</li>

<li>Assuming an absent <code>X-API-CREDITS-REMAINING</code> header means healthy usage rather than sub-threshold usage.</li>

<li>Building a daily credit dashboard when the limit is a rolling window.</li>

<li>Putting record IDs into EMF dimensions or Loki labels.</li>

<li>Treating an HTTP 200 on a bulk write as confirmation that every record landed.</li>

<li>Running alert rules only in the Grafana UI, so nothing evaluates if that instance is down.</li>
</ul>



<h2 class="wp-block-heading">Best practices for monitoring Zoho integrations</h2>



<ol class="wp-block-list">
<li>Start with a deadman alert per channel. If you build one thing, build this.</li>

<li>Set <code>channel_expiry</code> explicitly, renew on a schedule, and monitor the renewal job as carefully as the integration.</li>

<li>Count records, not requests, on both sides of every write.</li>

<li>Keep a client-side call counter so you are never dependent on a header that may not be sent.</li>

<li>Gauge in-flight requests to see concurrency pressure before it becomes <code>TOO_MANY_REQUESTS</code>.</li>

<li>Attach a correlation ID to every record as structured metadata so one identifier joins logs and metrics.</li>

<li>Evaluate critical rules server-side, in the Loki ruler or as CloudWatch alarms.</li>

<li>Test by disabling a channel in a sandbox org and confirming the alert fires. An untested deadman alert is decoration.</li>
</ol>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Why did my Zoho webhook stop working on its own?</h3>



<p class="wp-block-paragraph">Almost always channel expiry. Zoho CRM notification channels last a maximum of one week, and if the expiry is omitted or set beyond that maximum, the channel defaults to one hour. Nothing is logged on your side because the deliveries simply stop arriving.</p>



<h3 class="wp-block-heading">Can I list my active Zoho notification channels?</h3>



<p class="wp-block-paragraph">There is no endpoint for enumerating active channels, which is a long-standing complaint in Zoho&#8217;s developer community. Keep the inventory in your own configuration and treat the receipt of events as the only reliable liveness signal.</p>



<h3 class="wp-block-heading">Do I need both CloudWatch and Loki?</h3>



<p class="wp-block-paragraph">No. Either can carry the whole job. The pairing makes sense when your integration already runs on AWS, because EMF gives you alarmable metrics from log output at no extra instrumentation cost, while Loki holds richer request context more cheaply. If you are starting fresh outside AWS, one backend is simpler and simpler is worth a lot.</p>



<h3 class="wp-block-heading">How do I monitor Zoho API credits properly?</h3>



<p class="wp-block-paragraph">Record the <code>X-API-CREDITS-REMAINING</code> header when it appears, alert on the moment it first appears in a window, and maintain your own call counter as the fallback. Model the budget as a rolling twenty-four hour window rather than a daily reset.</p>



<h3 class="wp-block-heading">What causes TOO_MANY_REQUESTS if I am under my credit limit?</h3>



<p class="wp-block-paragraph">Concurrency, not consumption. Credits and concurrency are independent ceilings. Expensive operations such as COQL queries and composite requests share a tighter sub-concurrency pool, so a burst of parallel queries can trip it while your credit balance is untouched.</p>



<h3 class="wp-block-heading">Should I still use Promtail for shipping integration logs?</h3>



<p class="wp-block-paragraph">No. Promtail reached end of life on 2 March 2026 and its functionality moved into Grafana Alloy. Existing deployments will keep running, but they are unsupported, and Grafana provides a tool to convert Promtail configuration to Alloy.</p>



<h3 class="wp-block-heading">How long should the deadman alert window be?</h3>



<p class="wp-block-paragraph">Base it on the quietest legitimate period for that module. A leads webhook that goes quiet overnight and at weekends needs a longer window than an order sync, or you will train the team to ignore it. Measure the real gap distribution for a fortnight before picking a number.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Monitoring Zoho integrations is mostly the discipline of detecting absence. Expired channels, exhausted credits and silently rejected records all present as nothing happening, and nothing happening is invisible to the instrumentation most teams reach for first.</p>



<p class="wp-block-paragraph">So invert the default. Alarm when the count is too low rather than when the errors are too high. Set <code>--treat-missing-data breaching</code>. Use <code>absent_over_time</code>. Count records on both sides and watch the gap. If you build only one thing from this post, build the deadman alert on notification delivery, because that is the failure that will otherwise be reported to you by a salesperson wondering where the leads went.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help instrumenting a Zoho integration?</h2>



<p class="wp-block-paragraph">Most of the work here is judgement about what to measure rather than volume of code. If you are staring at a sync that keeps going quiet, this is the sort of thing I take on:</p>



<ul class="wp-block-list">
<li>Building deadman alerting for Zoho notification channels, including renewal jobs that monitor themselves.</li>

<li>Instrumenting credit and concurrency headroom so throttling is visible before it becomes <code>TOO_MANY_REQUESTS</code>.</li>

<li>Designing EMF dimensions and Loki label schemas that will not turn into a cardinality bill.</li>

<li>Adding reconciliation jobs that compare record counts across systems and surface drift early.</li>

<li>Standing up Loki and Grafana Alloy, or migrating an existing Promtail pipeline that is now unsupported.</li>

<li>Reviewing an existing Grafana dashboard and cutting the panels that look reassuring but detect nothing.</li>
</ul>



<p class="wp-block-paragraph">Send me a sample webhook payload, a redacted response header set, or a screenshot of the dashboard you currently trust, and I will tell you what it would miss.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/monitoring-zoho-integrations/">Monitoring Zoho Integrations: Catching the Failures That Never Throw an Error</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>One Label Too Many: Centralized Logging With Loki Without Wrecking It</title>
		<link>https://john-nessime.com/blog/technical-guides/centralized-logging-loki/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 01 Aug 2026 09:24:17 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Docker]]></category>
		<category><![CDATA[Grafana]]></category>
		<category><![CDATA[Grafana Alloy]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Linux]]></category>
		<category><![CDATA[Logging]]></category>
		<category><![CDATA[LogQL]]></category>
		<category><![CDATA[Logs]]></category>
		<category><![CDATA[Loki]]></category>
		<category><![CDATA[Monitoring]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[Prometheus]]></category>
		<category><![CDATA[Self Hosting]]></category>
		<category><![CDATA[SRE]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=68</guid>

					<description><![CDATA[<p>Loki was fast until someone added one more label. Now queries time out during incidents and the index no longer fits in memory. Here's the model you have to internalise, what replaced Promtail, and how to keep labels, retention and storage costs under control.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/centralized-logging-loki/">One Label Too Many: Centralized Logging With Loki Without Wrecking It</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The alert fires at eleven at night. You open Grafana, pick the right time range, type a query, and watch the spinner. Forty seconds later it times out. You narrow the range to five minutes and it comes back, eventually. The one tool you need during an incident is the one that has become unusable.</p>



<p class="wp-block-paragraph">Nothing about the Loki config changed. What changed is that three weeks ago somebody added <code>request_id</code> as a label because it seemed helpful, and Loki has been quietly building millions of separate streams ever since. The index that used to fit comfortably in memory does not. Every query now has to open thousands of tiny chunks instead of a handful of large ones.</p>



<p class="wp-block-paragraph">This is the failure mode that defines <strong>centralized logging with Loki</strong>, and it is almost entirely self-inflicted. Loki is cheap and fast when you use it the way it was designed and miserable when you bring habits from Elasticsearch. Nothing warns you at the moment you make the mistake. The bill comes weeks later, during an incident.</p>



<p class="wp-block-paragraph">This covers the model you have to internalise, how to choose labels, what replaced Promtail (it is gone, and every tutorial older than this year is wrong about it), the LogQL patterns worth knowing, and how to keep retention and storage costs under control.</p>



<h2 class="wp-block-heading">Loki is not Elasticsearch, and that is the entire design</h2>



<p class="wp-block-paragraph">Elasticsearch builds a full inverted index over every token in every log line. Powerful, and the index often ends up as large as the logs themselves. That is where the cost comes from.</p>



<p class="wp-block-paragraph">Loki indexes only labels. The log content is compressed into chunks and pushed to object storage, untouched and unindexed. A query does two things: it uses the index to select which streams to read, then it brute-force scans the contents of those chunks in parallel.</p>



<p class="wp-block-paragraph">The consequence is a rule worth writing on a wall: <strong>labels select, filters search.</strong> Labels exist to narrow down which chunks get opened. Everything else is a filter applied at query time. Scanning is genuinely fast because it is embarrassingly parallel, so filtering on content is not the expensive operation people assume.</p>



<p class="wp-block-paragraph">What is expensive is streams. A stream is one unique combination of label values, and each gets its own chunks. Do the arithmetic before you add a label:</p>



<ul class="wp-block-list">
<li>8 jobs, 3 environments, 12 hosts gives you 288 streams. Comfortable.</li>
<li>Add <code>request_id</code>, and you multiply that by however many requests you serve. The index explodes, chunks get flushed while nearly empty, and object storage fills with millions of tiny files that queries must open individually.</li>
</ul>



<p class="wp-block-paragraph">Ingesters also hold every active stream in memory. Cardinality does not just make queries slow; it is how Loki gets OOM-killed.</p>



<h2 class="wp-block-heading">Choosing labels</h2>



<p class="wp-block-paragraph">A label earns its place if it satisfies all three: the set of values is bounded and small, you know the values in advance, and you would actually use it to narrow a search.</p>



<ul class="wp-block-list">
<li><strong>Good:</strong> <code>job</code>, <code>service</code>, <code>env</code>, <code>host</code>, <code>namespace</code>, <code>level</code>.</li>
<li><strong>Bad:</strong> <code>request_id</code>, <code>trace_id</code>, <code>user_id</code>, <code>session_id</code>, full URL paths, IP addresses, timestamps, durations, anything derived from the log content itself.</li>
</ul>



<p class="wp-block-paragraph">Two subtler traps. <code>instance</code> including an ephemeral port is unbounded. And in container environments, pod names churn constantly, so labelling by pod name gives you a new stream every deployment forever.</p>



<p class="wp-block-paragraph">Aim to keep the total under a few thousand active streams. Check rather than guess, using <code>logcli</code>, which reports cardinality per label:</p>



<pre class="wp-block-code"><code># Which of your labels is quietly out of control?
logcli series '{}' --analyze-labels</code></pre>



<p class="wp-block-paragraph">Run that before you have a problem. It is the single most useful diagnostic here and almost nobody knows about it.</p>



<h3 class="wp-block-heading">The escape hatch: structured metadata</h3>



<p class="wp-block-paragraph">&#8220;Never label a trace ID&#8221; is correct but unsatisfying, because sometimes you genuinely need to find every line for one trace. Loki 3 answers this with structured metadata: key-value pairs attached to a log line, stored alongside it, <em>not</em> in the index and not creating streams.</p>



<pre class="wp-block-code"><code>limits_config:
  # Off by default on older configs. Turn it on before your
  # agent starts sending structured metadata, or the writes
  # get rejected and you spend an hour puzzling over it.
  allow_structured_metadata: true</code></pre>



<p class="wp-block-paragraph">Query it with the same label-filter syntax, and it costs you nothing in cardinality:</p>



<pre class="wp-block-code"><code>{job="api", env="production"} | trace_id="9f2c1b4e"</code></pre>



<p class="wp-block-paragraph">So the decision becomes three-way rather than two-way: bounded and used for selection becomes a label, high-cardinality but frequently searched becomes structured metadata, and everything else stays in the line and gets parsed at query time.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Promtail is gone. Use Alloy.</h2>



<p class="wp-block-paragraph">This one has a date and it matters, because it invalidates most of the Loki material on the internet.</p>



<p class="wp-block-paragraph">Promtail, the agent every Loki tutorial tells you to install, was deprecated in February 2025 and reached end of life on 2 March 2026. No security patches, no bug fixes, no updates. Grafana Alloy, their distribution of the OpenTelemetry Collector, is the replacement, and it handles logs, metrics, traces and profiles in one binary instead of running a separate agent per signal.</p>



<p class="wp-block-paragraph">If you already run Promtail, there is a conversion tool rather than a rewrite:</p>



<pre class="wp-block-code"><code># Converts an existing Promtail config to Alloy syntax.
# Read the output before deploying it; the converter flags
# anything it could not translate cleanly.
alloy convert --source-format=promtail --output=config.alloy promtail.yaml</code></pre>



<p class="wp-block-paragraph">Alloy configuration is a component graph rather than a YAML pipeline. Components reference each other&#8217;s outputs, which reads oddly at first and then makes more sense than nested YAML stages:</p>



<pre class="wp-block-code"><code>// Find the files, and attach the labels that will select them later.
// Note what is NOT here: nothing derived from log content.
local.file_match "system" {
  path_targets = [{
    __path__ = "/var/log/syslog",
    job      = "syslog",
    host     = constants.hostname,
    env      = "production",
  }]
}

loki.source.file "system" {
  targets    = local.file_match.system.targets
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}</code></pre>



<p class="wp-block-paragraph">Between source and write you can insert a <code>loki.process</code> component to parse lines, drop noise, and promote fields into structured metadata. Do the dropping here rather than at the Loki end: logs you never ship cost nothing to store, index or query, and debug-level chatter from a busy service is usually the single largest slice of ingest.</p>



<p class="wp-block-paragraph">For Docker hosts, Alloy has components that discover containers and read their logs directly, which is tidier than tailing the JSON files under <code>/var/lib/docker</code> and surviving a logging-driver change.</p>



<h2 class="wp-block-heading">Deployment shape and storage</h2>



<p class="wp-block-paragraph">Loki ships as one binary that runs in different modes depending on the <code>-target</code> flag.</p>



<ul class="wp-block-list">
<li><strong>Monolithic.</strong> Everything in one process. Right for a single VPS, a homelab, or anything up to a modest daily volume. Do not feel bad about this; a lot of people run microservices mode for a workload one process would handle.</li>
<li><strong>Simple scalable.</strong> Splits into read, write and backend targets behind a load balancer. The sensible next step, and where most self-hosted setups should stop.</li>
<li><strong>Microservices.</strong> Every component separately. Genuinely necessary at large scale and a substantial operational burden below it.</li>
</ul>



<p class="wp-block-paragraph">For storage, use TSDB with schema v13. Older guides show <code>boltdb-shipper</code>; that is the previous generation. Chunks belong in object storage in anything you care about, because filesystem storage means one disk, one node, and no redundancy.</p>



<pre class="wp-block-code"><code>schema_config:
  configs:
    # Schema changes are additive. Add a new entry with a future
    # 'from' date rather than editing the existing one, or Loki
    # cannot read the data it already wrote.
    - from: 2024-01-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h</code></pre>



<p class="wp-block-paragraph">That &#8220;additive&#8221; note is worth taking seriously. Editing an existing schema entry in place is one of the few ways to make historical logs unreadable.</p>



<p class="wp-block-paragraph">Any S3-compatible store works, which is where the cost advantage lives. Chunks are compressed and write-once, so cheaper tiers suit them well: S3 itself, or MinIO if you want it self-hosted, or one of the cheaper compatible providers like Backblaze B2 or Cloudflare R2. Watch egress pricing rather than storage pricing, since queries read chunks back.</p>



<h2 class="wp-block-heading">LogQL worth actually knowing</h2>



<p class="wp-block-paragraph">Every query starts with a stream selector in braces. That part is mandatory and it is what decides how much data gets read.</p>



<pre class="wp-block-code"><code># Order matters for performance:
#   1. select streams (index lookup)
#   2. filter lines (cheap string match, runs on every line)
#   3. parse (expensive, so run it on the smallest set possible)
{job="api", env="production"} |= "timeout" | json | status="500"</code></pre>



<p class="wp-block-paragraph">People routinely write <code>| json</code> before the line filter and then wonder why the query is slow. Parsing every line in the range to discard most of them is exactly the wrong order.</p>



<p class="wp-block-paragraph">Loki also turns logs into metrics, which is where it earns its place next to Prometheus:</p>



<pre class="wp-block-code"><code># Error rate per service, graphable and alertable.
sum by (job) (rate({env="production"} |= "level=error" [5m]))

# Which streams are eating your ingest budget?
topk(10, sum by (job, host) (count_over_time({env="production"}[1h])))</code></pre>



<p class="wp-block-paragraph">That second one is worth saving as a dashboard panel. Ingest problems announce themselves there long before they show up as a slow query.</p>



<p class="wp-block-paragraph">Loki&#8217;s ruler evaluates the same expressions as alerting rules and sends them to Alertmanager, so &#8220;alert when this service logs more than N errors in five minutes&#8221; needs no extra component if you already run Prometheus.</p>



<h2 class="wp-block-heading">Retention, and why it silently does nothing</h2>



<p class="wp-block-paragraph">Setting <code>retention_period</code> is not enough. Retention is enforced by the compactor, and the compactor does not do it unless you tell it to. Plenty of people set a retention period, watch their bucket keep growing, and never connect the two.</p>



<pre class="wp-block-code"><code>compactor:
  working_directory: /loki/compactor
  # Without this, the compactor compacts the index and
  # deletes nothing. This is the line people miss.
  retention_enabled: true
  delete_request_store: s3

limits_config:
  retention_period: 720h        # 30 days, global default</code></pre>



<p class="wp-block-paragraph">Per-stream retention lets you keep what matters and discard what does not: audit logs for a year, application debug logs for three days. That difference tends to matter more to the bill than any storage tier choice.</p>



<p class="wp-block-paragraph">Deletion is asynchronous. Chunks disappear from object storage some time after the index entries go, so do not panic if the bucket does not shrink immediately.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Queries are slow or time out</h3>



<p class="wp-block-paragraph">Cardinality first. Run <code>logcli series '{}' --analyze-labels</code> and look for a label with a suspiciously large value count. Then check query shape: a selector that matches everything, a parser before a line filter, or a time range far wider than needed. Adding query frontend caching helps repeated queries but will not rescue a bad label set.</p>



<h3 class="wp-block-heading">Stream limit exceeded</h3>



<p class="wp-block-paragraph">Loki is telling you the truth. Raising <code>max_global_streams_per_user</code> buys time and makes the underlying problem worse. Find the label that is generating streams and move it to structured metadata or into the line.</p>



<h3 class="wp-block-heading">Rate limit or ingestion errors</h3>



<p class="wp-block-paragraph">Two different limits produce similar-looking rejections: a global ingestion rate, and a per-stream rate. Per-stream limits usually mean one very chatty stream rather than too much traffic overall, and the fix is to split it by a label or drop the noise at the agent.</p>



<h3 class="wp-block-heading">Logs are not appearing at all</h3>



<p class="wp-block-paragraph">Work forward from the agent. Check the agent&#8217;s own logs first, then whether it can reach the push endpoint, then whether the file is actually readable by the agent&#8217;s user. Container log permissions and SELinux catch people constantly. In Grafana, remember the label browser only shows labels seen recently, so an empty dropdown often means nothing has arrived, not that the query is wrong.</p>



<h3 class="wp-block-heading">Logs stop after an agent restart</h3>



<p class="wp-block-paragraph">The position file that tracks how far it read is not persisting. In a container, that means it is not on a mounted volume, so every restart re-reads or skips depending on configuration.</p>



<h3 class="wp-block-heading">Storage keeps growing despite retention</h3>



<p class="wp-block-paragraph">Confirm <code>retention_enabled: true</code> is actually set, that exactly one compactor is running, and that it has write access to the object store. A compactor that cannot delete usually says so in its logs and nowhere else.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating labels like Elasticsearch fields and labelling everything.</li>
<li>Promoting a parsed value into a label without checking how many distinct values it has.</li>
<li>Following a tutorial that installs Promtail, which is now end of life.</li>
<li>Writing <code>| json</code> before the line filter.</li>
<li>Setting <code>retention_period</code> without <code>retention_enabled: true</code>.</li>
<li>Running more than one compactor.</li>
<li>Editing an existing <code>schema_config</code> entry instead of adding a new one.</li>
<li>Filesystem storage on a single node for anything you would miss.</li>
<li>Shipping every debug line and then paying to store, index and scan it.</li>
<li>Raising stream limits instead of fixing cardinality.</li>
<li>Running microservices mode for a workload the single binary handles comfortably.</li>
<li>No dashboard showing ingest volume per stream, so growth is invisible until queries break.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Decide labels deliberately: bounded, known in advance, used for selection.</li>
<li>Put high-cardinality but searchable values in structured metadata, not labels.</li>
<li>Run <code>--analyze-labels</code> periodically, not just when something breaks.</li>
<li>Drop noise at the agent, before it costs you anything downstream.</li>
<li>Emit structured logs from your applications so parsers are cheap and reliable.</li>
<li>Start monolithic and move to simple scalable only when you have a reason.</li>
<li>TSDB, schema v13, chunks in object storage.</li>
<li>Turn retention on properly and use per-stream retention to keep costs sane.</li>
<li>Order queries: select, filter, parse.</li>
<li>Alert on log-derived metrics through the ruler rather than eyeballing dashboards.</li>
<li>Graph ingest volume per job so cardinality growth shows up as a trend.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Is Loki a drop-in replacement for the ELK stack?</h3>



<p class="wp-block-paragraph">No, and expecting it to be is where most disappointment comes from. Loki is dramatically cheaper to run and much better suited to &#8220;show me the logs for this service around this time&#8221;. Elasticsearch is better at ad-hoc full-text search across everything with no idea where to start. Pick based on which of those you do more often.</p>



<h3 class="wp-block-heading">Why is my Loki so slow?</h3>



<p class="wp-block-paragraph">Almost always label cardinality. Run <code>logcli series '{}' --analyze-labels</code> and look for the outlier. The second most common cause is a query that parses before it filters. Hardware is a distant third.</p>



<h3 class="wp-block-heading">Should I still use Promtail?</h3>



<p class="wp-block-paragraph">No. It reached end of life in March 2026 and receives no security patches. Use Grafana Alloy, and run <code>alloy convert</code> against your existing config rather than rewriting it from scratch.</p>



<h3 class="wp-block-heading">How many labels is too many?</h3>



<p class="wp-block-paragraph">The count matters less than the product of their distinct values. Five labels with a handful of values each is fine. Three labels where one has fifty thousand values is not. Multiply before you add.</p>



<h3 class="wp-block-heading">Can I run Loki on a single VPS?</h3>



<p class="wp-block-paragraph">Yes, and for a handful of services it is a perfectly good answer. Monolithic mode alongside Grafana and Prometheus fits on a modest box from any provider. Point chunk storage at an object store rather than local disk so a dead server does not take your history with it.</p>



<h3 class="wp-block-heading">How do I search for a specific request across services?</h3>



<p class="wp-block-paragraph">Attach the trace or request ID as structured metadata at the agent, then filter on it. You get the high-cardinality lookup you wanted without creating a stream per request.</p>



<h3 class="wp-block-heading">Self-host or use a managed service?</h3>



<p class="wp-block-paragraph">Self-hosting a monolithic Loki is genuinely low effort. Self-hosting a highly available multi-tenant Loki is a real ongoing job, and at that point Grafana Cloud or a hosted alternative deserves an honest cost comparison including your time. The break-even is further out than vendors suggest and closer than enthusiasts admit.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">Labels select, filters search. Every serious problem with centralized logging with Loki traces back to someone treating a label as a search field. Loki will accept it without complaint, and the cost arrives weeks later as a query that will not finish.</p>



<p class="wp-block-paragraph">So before adding any label, multiply out how many streams it creates. If the answer is more than a few thousand, it belongs in structured metadata or in the log line. And check your cardinality on a schedule, because this degrades gradually and you will not notice the day it crosses the line.</p>



<h2 class="wp-block-heading">Need a logging setup that stays fast?</h2>



<p class="wp-block-paragraph">Most Loki installations I get asked to look at were fine on day one and got slower every week. Work I take on:</p>



<ul class="wp-block-list">
<li>Diagnosing a slow Loki: cardinality analysis, query shape review, and a concrete list of labels to move.</li>
<li>Migrating Promtail configurations to Grafana Alloy, including Docker and Kubernetes log collection.</li>
<li>Building a centralized logging stack from scratch: Loki, Alloy, Grafana, object storage, retention and dashboards.</li>
<li>Restructuring labels and introducing structured metadata without losing queryability.</li>
<li>Retention and storage cost work: per-stream policies, agent-side dropping, object storage tiering.</li>
<li>Log-based alerting through the ruler, wired into an existing Prometheus and Alertmanager setup.</li>
</ul>



<p class="wp-block-paragraph">Send me the output of <code>logcli series '{}' --analyze-labels</code> and a query that feels slow, and I will tell you where the problem is.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/centralized-logging-loki/">One Label Too Many: Centralized Logging With Loki Without Wrecking It</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
