The sync worker was healthy. Zero errors in the last twenty-four hours. CPU flat, memory flat, no restarts, no failed jobs. Every dashboard was green because every dashboard was measuring the wrong thing. The integration had stopped receiving anything from Zoho four days earlier, and an integration that receives nothing produces no errors at all.

That is the whole problem with monitoring Zoho integrations. The failures that hurt do not throw exceptions. They show up as silence, as slowly degrading throughput, or as HTTP 200 responses that quietly did not do what you asked. This post covers how to instrument for those three failure families using CloudWatch, Grafana and Loki, and why the obvious metrics miss all three.

The three ways Zoho integrations actually break

Before touching a dashboard, it helps to be honest about what you are defending against. In my experience these fall into three groups, and they need genuinely different instrumentation:

1. Silent stop. Zoho stops sending you events. Nothing fails, because nothing happens.
2. Silent throttle. You run out of API credits or hit a concurrency ceiling. Throughput degrades before anything hard-fails.
3. Silent drift. Calls succeed, records do not land, and the two systems diverge without a single error log.

An error-rate panel catches none of these. That is not a criticism of error-rate panels, it is just what they are for.

Failure family one: the integration that just stops

If you push data out of Zoho CRM using the Notifications API, you subscribe a notify_url to module events and Zoho posts to it when records change. It works well. It also has a detail that catches people out repeatedly.

Notification channels expire. Zoho’s documentation is explicit: the expiry can be set to a maximum of one week from the time the channel is enabled, and if you do not specify one, or you ask for longer than a week, the default expiry is one hour. So the failure mode is not “someone forgot to renew after a year.” It is “someone deployed a subscription script without an expiry field and the integration worked beautifully for sixty minutes.”

What makes this genuinely hard rather than just annoying is that there is no endpoint to list your currently active channels. You cannot ask Zoho what it thinks it is subscribed to. You can only ask whether events are still arriving. That single fact should shape your entire monitoring design: you are not monitoring for errors, you are monitoring for absence.

Emit a heartbeat, then alarm on its absence

The receiving end of the webhook should emit a metric on every delivery, including deliveries you decide to discard. Embedded Metric Format is the cheapest way to do this from Lambda or a container, because you write structured JSON to stdout and CloudWatch extracts the metrics asynchronously. No PutMetricData call, no synchronous latency in your request path.

{
  "_aws": {
    "Timestamp": 1710000000000,
    "CloudWatchMetrics": [
      {
        "Namespace": "ZohoIntegration",
        "Dimensions": [["Module", "Channel"]],
        "Metrics": [
          { "Name": "NotificationsReceived", "Unit": "Count" },
          { "Name": "ChannelSecondsRemaining", "Unit": "Seconds" }
        ]
      }
    ]
  },
  "Module": "Leads",
  "Channel": "leads-prod",
  "NotificationsReceived": 1,
  "ChannelSecondsRemaining": 486000,
  "record_id": "5725767000002166520",
  "operation": "insert"
}

Note what is a dimension and what is not. Module and Channel are bounded sets, so they are safe dimensions. record_id sits in the log body only. EMF creates a separate custom metric for every unique dimension combination, so promoting a record ID to a dimension is how you turn a monitoring project into a billing incident. AWS says this plainly in the EMF documentation and it is worth reading twice before you ship.

Now the alarm. A normal CloudWatch alarm goes to INSUFFICIENT_DATA when the metric stops reporting, and INSUFFICIENT_DATA does not notify. That is precisely backwards for this failure. You want missing data treated as a breach:

aws cloudwatch put-metric-alarm 
  --alarm-name zoho-leads-notifications-silent 
  --namespace ZohoIntegration 
  --metric-name NotificationsReceived 
  --dimensions Name=Module,Value=Leads Name=Channel,Value=leads-prod 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 2 
  --threshold 1 
  --comparison-operator LessThanThreshold 
  --treat-missing-data breaching 
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:ops-alerts

The --treat-missing-data breaching flag is the entire point of that command. Without it, an integration that goes completely dead produces an alarm that sits quietly in an unknown state forever.

On the Loki side the equivalent is absent_over_time, which returns 1 when a stream has produced no entries in the window. Run it from the Loki ruler so it fires even when nobody has a browser open:

groups:
  - name: zoho-integration
    interval: 5m
    rules:
      - alert: ZohoNotificationsSilent
        expr: absent_over_time({job="zoho-webhook", module="leads"}[2h])
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: "No Zoho notifications for Leads in 2h"
          description: "Channel may have expired. Re-subscribe and check channel_expiry."

One caveat worth knowing: absent_over_time only works for label combinations you name explicitly. It cannot tell you that a stream you never thought about has gone quiet. That is a real limitation, and it means your channel inventory has to live in your own configuration, not be discovered from Loki.

Because you cannot list channels through the API, treat renewal as a scheduled job with its own deadman alert, and emit ChannelSecondsRemaining at each renewal. Alarm when it drops below roughly a day. That gives you warning before the silence starts rather than after somebody notices.

Failure family two: throttling that is not rate limiting

Most engineers instrument Zoho as if it were a requests-per-minute API. It is not, and that mismatch produces dashboards that look fine right up until calls start failing.

Zoho CRM meters API usage in credits, not calls. Different operations deduct different amounts, and bulk operations are metered per batch of records rather than per request. The budget resets on a rolling twenty-four hour window measured from each call, not at midnight. If your dashboard has a daily reset at 00:00, it is drawing a line that does not exist.

The gauge that appears halfway through the tank

Here is the detail that makes credit monitoring awkward. Zoho returns remaining credits in an X-API-CREDITS-REMAINING response header, but only once usage has reached fifty percent or more of the daily allowance. Below that threshold the header simply is not there.

Read that again in monitoring terms. An absent header means either “you are comfortably under half your budget” or “your instrumentation is broken and you have no idea.” Those two states look identical, and one of them is fine while the other is a blind spot. So record both facts separately:

• A counter for responses where the header was present, which effectively tells you when you crossed the halfway mark.
• A gauge of the value itself, which only has meaning while the header exists.
• Your own client-side count of calls made, which works regardless and is the only thing you control.

The first crossing of that threshold is a useful early warning in its own right. Alert on the transition, not just on a low remaining value.

Concurrency is a separate ceiling

Alongside credits, Zoho applies a concurrency limit: the number of calls active simultaneously per org. There is also a sub-concurrency limit covering the expensive operations such as COQL queries, composite requests, lead conversion, and inserts or updates above a small record count. Exceeding either returns TOO_MANY_REQUESTS.

Two consequences matter for monitoring. First, since the limit is on simultaneous calls rather than calls per unit time, a requests-per-minute graph will never predict a concurrency failure. You need a gauge of in-flight requests, incremented before the call and decremented in a finally block. Second, the sub-concurrency ceiling is the same across every CRM edition, so upgrading the plan raises the general limit but does not buy headroom on the expensive calls. Batch those and queue them rather than fanning out.

One more source of consumption that surprises people: integration tasks inside Deluge functions consume credits from the same org budget as your external API calls. If your credit graph climbs while your service is idle, someone’s workflow rule is spending your budget. Zoho’s API Dashboard breaks consumption down by application and function, which is where to look before blaming your own code.

Failure family three: successful calls that did nothing

Bulk and multi-record operations return a per-record status inside the response body. The HTTP status can be perfectly healthy while individual records were rejected for validation rules, mandatory fields, duplicate checks or picklist values that do not exist in the target layout.

If your client only checks the HTTP status, you will log a success, increment a success counter, and move on. Weeks later somebody asks why the CRM is missing a slice of records that your logs insist were written.

The fix is to stop counting requests and start counting records, on both sides:

• Records submitted versus records confirmed in the per-record response status.
• A breakdown of rejection reasons as a low-cardinality label, so a spike in one specific validation error is visible.
• A periodic reconciliation job comparing source counts against Zoho counts for a bounded window, which is the only thing that catches drift you failed to anticipate.

In LogQL, the divergence between the two counters is the panel that actually earns its place on the wall:

sum(count_over_time({job="zoho-sync"} | json | status = "submitted" [1h]))
  -
sum(count_over_time({job="zoho-sync"} | json | status = "confirmed" [1h]))

A healthy integration holds that near zero. A number that grows steadily is drift, and it is visible weeks before anyone in sales notices.

Deciding which signal goes where

Running CloudWatch, Grafana and Loki together only helps if each one has a job. The split I reach for first:

• CloudWatch holds the numbers you alarm on, especially where the workload already runs in AWS and EMF gives you metrics for free out of log output.
• Loki holds the full request and response context: which record, which module, which validation error, which channel. This is what you actually read at three in the morning.
• Grafana is the single pane over both, with CloudWatch and Loki added as separate data sources so one dashboard can show the credit gauge next to the rejection reasons.

The trap that spans both backends is cardinality, and it bites in the same way for different reasons. In CloudWatch, each unique dimension combination becomes a billable custom metric. In Loki, each unique label combination becomes a separate stream, and Loki was designed for a small number of long-lived streams. Record IDs, email addresses and timestamps belong in neither.

Loki’s answer is structured metadata: high-cardinality fields attached to log lines without creating streams or entering the index. It needs enabling per tenant:

limits_config:
  allow_structured_metadata: true
  volume_enabled: true

Put module, environment and job in labels. Put the Zoho record ID and the correlation ID in structured metadata. Query with a label selector first and filter on the rest.

On collection, one scheduling note that does age: Promtail reached end of life on 2 March 2026, with its collection capabilities folded into Grafana Alloy. If you are still shipping through Promtail, that migration is now maintenance debt rather than an optional upgrade, and Grafana published a conversion tool for existing configurations. New pipelines should start on Alloy.

Where you run all this is a real decision. Self-hosting Loki on a VPS from a provider like Contabo or InterServer is cheap and entirely workable for a single integration’s log volume, and it keeps the data under your control. Grafana Cloud or Amazon Managed Grafana removes the operational surface if you would rather not babysit an ingester. Both are defensible. The honest trade-off is that self-hosted Loki is easy to stand up and less easy to keep healthy once retention and compaction start to matter.

Troubleshooting the monitoring itself

• EMF metrics never appear. The JSON must be a single line and structurally valid. A pretty-printed or truncated log line is ingested as an ordinary log event with no error, so check the raw event in the log stream rather than the metrics console.
• The alarm never fires despite obvious silence. Look at the alarm state. Sitting in INSUFFICIENT_DATA means the missing-data treatment was left at its default.
• absent_over_time returns nothing. That is the success case. It emits a value only when the stream is empty, which makes it feel broken while everything is working.
• Loki queries slow to a crawl. Check stream count before blaming resources. A label added “temporarily” during debugging is the usual cause.
• Webhook deliveries stop after a deployment. Check whether the subscription script reran and created a channel without an expiry. An hour later it will look identical to a network problem.
• Credit consumption does not match your call count. Expected. Operations deduct different credit amounts, and Deluge integration tasks draw from the same pool.

Common mistakes

• Alerting on error rate alone, which is blind to every failure described above.
• Creating notification channels without an explicit channel_expiry and inheriting the one-hour default.
• Assuming an absent X-API-CREDITS-REMAINING header means healthy usage rather than sub-threshold usage.
• Building a daily credit dashboard when the limit is a rolling window.
• Putting record IDs into EMF dimensions or Loki labels.
• Treating an HTTP 200 on a bulk write as confirmation that every record landed.
• Running alert rules only in the Grafana UI, so nothing evaluates if that instance is down.

Best practices for monitoring Zoho integrations

1. Start with a deadman alert per channel. If you build one thing, build this.
2. Set channel_expiry explicitly, renew on a schedule, and monitor the renewal job as carefully as the integration.
3. Count records, not requests, on both sides of every write.
4. Keep a client-side call counter so you are never dependent on a header that may not be sent.
5. Gauge in-flight requests to see concurrency pressure before it becomes TOO_MANY_REQUESTS.
6. Attach a correlation ID to every record as structured metadata so one identifier joins logs and metrics.
7. Evaluate critical rules server-side, in the Loki ruler or as CloudWatch alarms.
8. Test by disabling a channel in a sandbox org and confirming the alert fires. An untested deadman alert is decoration.

Frequently asked questions

Why did my Zoho webhook stop working on its own?

Almost always channel expiry. Zoho CRM notification channels last a maximum of one week, and if the expiry is omitted or set beyond that maximum, the channel defaults to one hour. Nothing is logged on your side because the deliveries simply stop arriving.

Can I list my active Zoho notification channels?

There is no endpoint for enumerating active channels, which is a long-standing complaint in Zoho’s developer community. Keep the inventory in your own configuration and treat the receipt of events as the only reliable liveness signal.

Do I need both CloudWatch and Loki?

No. Either can carry the whole job. The pairing makes sense when your integration already runs on AWS, because EMF gives you alarmable metrics from log output at no extra instrumentation cost, while Loki holds richer request context more cheaply. If you are starting fresh outside AWS, one backend is simpler and simpler is worth a lot.

How do I monitor Zoho API credits properly?

Record the X-API-CREDITS-REMAINING header when it appears, alert on the moment it first appears in a window, and maintain your own call counter as the fallback. Model the budget as a rolling twenty-four hour window rather than a daily reset.

What causes TOO_MANY_REQUESTS if I am under my credit limit?

Concurrency, not consumption. Credits and concurrency are independent ceilings. Expensive operations such as COQL queries and composite requests share a tighter sub-concurrency pool, so a burst of parallel queries can trip it while your credit balance is untouched.

Should I still use Promtail for shipping integration logs?

No. Promtail reached end of life on 2 March 2026 and its functionality moved into Grafana Alloy. Existing deployments will keep running, but they are unsupported, and Grafana provides a tool to convert Promtail configuration to Alloy.

How long should the deadman alert window be?

Base it on the quietest legitimate period for that module. A leads webhook that goes quiet overnight and at weekends needs a longer window than an order sync, or you will train the team to ignore it. Measure the real gap distribution for a fortnight before picking a number.

The one thing worth remembering

Monitoring Zoho integrations is mostly the discipline of detecting absence. Expired channels, exhausted credits and silently rejected records all present as nothing happening, and nothing happening is invisible to the instrumentation most teams reach for first.

So invert the default. Alarm when the count is too low rather than when the errors are too high. Set --treat-missing-data breaching. Use absent_over_time. Count records on both sides and watch the gap. If you build only one thing from this post, build the deadman alert on notification delivery, because that is the failure that will otherwise be reported to you by a salesperson wondering where the leads went.

Need help instrumenting a Zoho integration?

Most of the work here is judgement about what to measure rather than volume of code. If you are staring at a sync that keeps going quiet, this is the sort of thing I take on:

• Building deadman alerting for Zoho notification channels, including renewal jobs that monitor themselves.
• Instrumenting credit and concurrency headroom so throttling is visible before it becomes TOO_MANY_REQUESTS.
• Designing EMF dimensions and Loki label schemas that will not turn into a cardinality bill.
• Adding reconciliation jobs that compare record counts across systems and surface drift early.
• Standing up Loki and Grafana Alloy, or migrating an existing Promtail pipeline that is now unsupported.
• Reviewing an existing Grafana dashboard and cutting the panels that look reassuring but detect nothing.

Send me a sample webhook payload, a redacted response header set, or a screenshot of the dashboard you currently trust, and I will tell you what it would miss.

The post Monitoring Zoho Integrations: Catching the Failures That Never Throw an Error appeared first on John Nessime.

That slope is the signature. CloudWatch rarely fails loudly, it accumulates: a log group created automatically by a service you enabled once, retention left at the default, a metric filter with a dimension nobody thought about, and a dashboard quietly calling the API every few seconds forever.

This post breaks down where CloudWatch costs actually come from, why the same log line gets charged more than once, how cardinality multiplies your bill without your traffic changing, and how to attribute spend to specific log groups before you delete anything. It is organised by cost lever rather than by feature, because that is the order you work in when the invoice lands.

The four meters behind your CloudWatch bill

CloudWatch is not one product with one price. It is a set of independent meters that happen to share a console. Working out which meter is running is most of the job, because the fix for each is completely different.

• Ingestion. Per GB of log data accepted into a log group. Usually the largest component on log-heavy accounts.
• Storage. Per GB-month for what is retained afterwards. Cheap per unit, but it compounds because it never stops unless you tell it to.
• Analysis. Logs Insights bills on data scanned by a query, not on rows returned and not per query.
• Metrics and API calls. Custom metrics bill per unique time series per month. Separately, calls like GetMetricData bill per metric requested, which is how third-party dashboards end up on an AWS invoice.

The thing worth internalising: a single log line can be billed three times. Once when it arrives, again every month it sits in storage, and again every time a query scans the block it lives in. None of that is hidden, but nothing in the console surfaces it either.

Ingestion is the meter that actually hurts

Uncompressed in, compressed out

Ingestion is metered on the uncompressed volume of events you send. Storage is billed on the compressed size CloudWatch keeps. Text logs compress well, so the stored figure can be a small fraction of what you were charged to get the data in.

That asymmetry is why people look at a log group holding a few gigabytes, decide it is harmless, and miss that it cost several times that to arrive. The storedBytes value shown on a log group is the compressed number. It tells you about your storage line, not your ingestion line.

The log groups you never created

Most runaway ingestion comes from things switched on with a checkbox. VPC Flow Logs pointed at CloudWatch. EKS control plane logging with every type enabled. Lambda functions emitting a full structured payload per invocation. CloudTrail delivering to a log group as well as S3. RDS general query logs left on after a debugging session.

None of those feel like a decision at the time, and all of them scale with traffic rather than headcount. That is why the slope keeps rising while the team stays the same size.

The most effective lever here is not a CloudWatch setting at all. It is cutting verbosity at the source, before the bytes leave your application. Dropping debug-level logging in production, sampling health check lines, trimming repeated stack frames. Filtering after ingestion does nothing for this meter, because you already paid.

Retention: the one-line fix almost nobody applies

Log groups default to Never Expire. Not thirty days, not a year. Forever. Every log group created implicitly by another AWS service inherits that default, so an account accumulates hundreds of them without a single deliberate retention decision.

Find the offenders first. This lists every log group with no retention policy, alongside its compressed size:

aws logs describe-log-groups 
  --query 'logGroups[?!retentionInDays].[logGroupName,storedBytes]' 
  --output table

The ?!retentionInDays filter selects entries where the field is absent, which is how the API represents indefinite retention. There is no explicit “never” value to match on.

aws logs put-retention-policy 
  --log-group-name /aws/lambda/order-processor 
  --retention-in-days 30

Three things are worth knowing before running that across an account.

1. The value is an enum, not a free integer. Accepted values run 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731 and upward to ten years. Asking for 45 days is rejected.
2. Deletion is not instant. Expired events are marked for deletion and typically removed within about three days. They stop contributing to your storage charge once marked, and are excluded from the reported storedBytes, so the billing effect lands before the data physically goes.
3. Reverting means removing the policy, not setting zero. To go back to indefinite retention you call delete-retention-policy rather than passing a zero value.

Dry run before you bulk-apply. This prints what would change without touching anything:

aws logs describe-log-groups 
  --query 'logGroups[?!retentionInDays].logGroupName' 
  --output text | tr 't' 'n' | while read -r lg; do
    echo "would set 30-day retention on: $lg"
  done

Retention is where compliance and cost pull against each other, and cost should lose. If an auditor needs seven years of logs, seven years is the answer. The move is to export them to S3 and apply lifecycle rules there, rather than paying CloudWatch rates for archival data you will query twice a decade.

Cardinality: the multiplier nobody sees coming

This is the part that surprises people, because it breaks the intuition that cost tracks traffic. Cardinality can multiply your metrics bill by an order of magnitude while request volume stays flat.

CloudWatch treats every unique combination of namespace, metric name and dimension values as a separate metric. Not a variant of one metric. A separate, separately billed metric. Publish OrderLatency with a Region dimension holding three values and you have three metrics. Add a CustomerId dimension and you have three times however many customers you have.

The dangerous dimension values are the ones that look like identifiers: request IDs, trace IDs, pod names, session tokens, customer IDs, IP addresses, URL paths with IDs embedded in them. Each new value is a new billable series that persists and accrues.

Metric filters have a guard rail, and it bites

Metrics extracted from logs by a metric filter bill as custom metrics. A filter can carry up to three dimensions, and AWS documents an explicit safety mechanism: CloudWatch Logs disables a metric filter that generates a thousand distinct dimension name/value pairs within a period.

That guard rail is useful, but read what it means operationally. Hitting it does not just cost money, it silently stops the filter, so the metric you built an alarm on stops receiving data. If that alarm treats missing data as not breaching, it goes quiet rather than firing. You lose the alert and keep the bill.

Audit what exists before adding more, and count what a namespace actually produces:

aws logs describe-metric-filters 
  --log-group-name /aws/lambda/order-processor

aws cloudwatch list-metrics 
  --namespace MyApp/Orders 
  --query 'length(Metrics)'

If that count is materially larger than the number of distinct things you consciously measure, you have found your multiplier.

EMF does exactly what you tell it

Embedded Metric Format is a good design: you emit structured events, CloudWatch extracts metrics from declared fields, and you get high-cardinality context in the logs with low-cardinality metrics for alarming. That split is the point.

The failure mode is promoting a high-cardinality field into the dimension set. AWS documents this plainly: declare something like a request ID as a dimension and EMF will, by design, create a custom metric for every unique combination. Keep identifiers in the log body as properties, and keep dimensions to values you would put on a dashboard axis.

Anomaly detection alarms cost more than they look

An anomaly detection alarm does not evaluate one metric. It evaluates the source metric plus the upper and lower bounds of the predicted band, and those bounds are metrics too. That is roughly a threefold multiple against a standard threshold alarm, per alarm.

It earns its keep on genuinely seasonal signals where a static threshold produces constant noise. Switching an entire alarm estate to it “to be safe” triples a line item for very little detection benefit.

Query cost spikes exactly when you need it most

Logs Insights bills on the volume a query scans. Not rows returned. Not query count. Scanned bytes. A query returning twelve lines that scans a terabyte costs the same as one returning a million lines that scans a terabyte.

During an incident, an engineer runs a dozen exploratory queries across a wildcard log group selection and a wide time window. The pricing model charges you hardest at the moment the tool is most valuable, which is an uncomfortable incentive. Two habits reduce it without reducing your ability to debug:

• Narrow the time window first, then the filter. Scanning is bounded by the selected range across the selected groups. Starting at fifteen minutes and widening is far cheaper than starting at a week and narrowing.
• Select specific log groups, not patterns. Querying across every Lambda log group multiplies scanned bytes by the number of functions when you only care about one.

Watch for scheduled queries too. A dashboard widget or automation running an Insights query every few minutes against a large log group is a fixed monthly cost nobody remembers creating.

Finding out where your CloudWatch costs come from

Attribute before you cut. Deleting log groups because they look big is how teams lose the one log they needed three weeks later.

1. Split the bill by meter. In Cost Explorer, filter service to CloudWatch and group by usage type. The region-prefixed usage types map to the meters directly: DataProcessing-Bytes is log ingestion, TimedStorage-ByteHrs is log storage, DataScanned-Bytes is Logs Insights, and CW:Requests and CW:GMD-Metrics cover API calls.
2. Rank log groups by ingestion. The IncomingBytes metric in the AWS/Logs namespace, dimensioned by log group, is the per-group ingestion figure. Graph it with the Sum statistic. One documented gotcha: use an absolute range covering exactly thirty days, because a relative range or a different length produces a number that does not correspond to a billing month.
3. Check stored bytes separately. A log group can be small in ingestion but large in storage because it has accumulated since the account was created. That is a retention problem, not a verbosity problem, and the fix differs.
4. Count custom metrics per namespace. Compare the count against your mental model of what you measure. The gap is cardinality.
5. Look inside the noisiest group. Once you know which log group dominates, find out which streams are responsible before changing any application code.

For that last step, a Logs Insights query over a short window is enough. Keep the range tight, because this query is itself billed on what it scans:

stats count(*) as events, sum(strlen(@message)) as approx_bytes by @logStream
| sort approx_bytes desc
| limit 20

That approximates byte volume per stream, which is usually enough to identify the chatty component. It is an approximation, not a billing figure, because it counts message characters rather than the metered event size.

If you manage several accounts and want attribution done for you, cost platforms such as Vantage and CloudZero break CloudWatch spend down by usage type and resource without you building dashboards. Worth the spend if client reporting is part of the job. For one account and one engineer, Cost Explorer plus the steps above gets the same answer.

Cheaper destinations, and what they cost you

Once you know your ingestion is dominated by logs you rarely read, there are three real options. None is free of trade-offs.

• The Infrequent Access log class. Set at the log group level, it ingests at a materially lower per-GB rate in exchange for a reduced feature set. Historically that meant no metric filters, no alarming from logs and no Live Tail, though the capability list has expanded since launch. Check the current log classes documentation rather than assuming, and note that a log group you extract metrics from is not a candidate.
• Subscription filter to S3 via Amazon Data Firehose. Sends log data to object storage where lifecycle rules and Athena queries apply. You trade Insights and the console experience for much cheaper long-term retention. Good for audit and forensic logs, poor for anything you troubleshoot weekly.
• A separate log stack. Grafana Cloud, or self-hosted Loki and Prometheus on a VPS from a provider like Contabo or InterServer, moves cost from per-GB metering to a fixed monthly box you administer. The economics can be dramatically better at volume. The price is that you now own retention, backups, upgrades and the availability of your own observability during an outage, which is exactly when you least want to be debugging it.

The honest version: CloudWatch is expensive partly because it is operated for you and wired into everything. At a few dozen gigabytes a month, migrating away costs more engineering time than it saves. At terabytes, the arithmetic flips hard.

Troubleshooting a bill that will not go down

You set retention everywhere and storage barely moved. Check whether storage was ever the dominant cost. If your usage type breakdown was mostly DataProcessing-Bytes, retention was never going to fix it.

Custom metric count grows with no code changes. Something is emitting a dimension whose value set is open-ended. Look for deployment identifiers, container IDs, or a version string appended to a dimension.

An alarm went quiet and the bill jumped the same week. Suspect a disabled metric filter. Check the filter still exists and is producing data, and verify the alarm’s missing data treatment. Alarms treating missing data as not breaching fail silently by design.

API request charges you cannot place. External dashboards are the usual answer. A Grafana instance polling CloudWatch, or any tool calling GetMetricData on a short interval, bills per metric requested. Widening the refresh interval on dashboards nobody watches live is a quick win.

Costs dropped, then came back. Look at your infrastructure as code. If Terraform or CloudFormation creates log groups without a retention argument, every apply restores the default. Fixing it in the console fixes it until the next deploy.

Common mistakes

• Treating storage as the problem because it is the number the console shows. Ingestion is usually larger and invisible in the log group list.
• Adding a dimension because it would be useful to filter by, without checking how many distinct values it can take.
• Filtering logs after they reach CloudWatch. The ingestion charge already applied.
• Deleting log groups to cut cost without checking whether anything alarms on them.
• Setting retention in the console while the IaC that owns the log group has none configured.
• Enabling every EKS control plane log type when only the audit log is consumed.

Best practices for controlling CloudWatch costs

• Make retention a required argument. If log groups come from Terraform or CloudFormation, treat a missing retention setting as a policy violation and fail the plan. Checkov and similar scanners have rules for exactly this.
• Decide verbosity per environment, not per service. Debug in staging, info in production, with a documented way to raise it temporarily during an incident.
• Keep dimensions bounded, identifiers in the message body. If you cannot enumerate a dimension’s possible values on a whiteboard, it does not belong in the dimension set.
• Alarm on the bill itself. A billing alarm on estimated charges, or an anomaly monitor scoped to CloudWatch, turns a quarterly surprise into a same-week signal.
• Review the top five log groups by ingestion monthly. Five minutes with the IncomingBytes graph catches new noise before it compounds.
• Route archival logs out deliberately. Anything kept for compliance rather than operations belongs in S3 with lifecycle rules from the start.

Frequently asked questions

Why is my CloudWatch bill so high when my traffic has not changed?

Two causes account for most cases. Storage accumulating because log groups default to indefinite retention, so the bill grows every month even at constant ingestion. Or metric cardinality, where a dimension with an open-ended value set keeps creating new billable series. Neither tracks request volume, which is why the bill and the traffic graph disagree.

Does deleting a log group reduce my bill immediately?

It stops future storage charges for that data and nothing else. Ingestion already billed is spent, and deleting the group does not stop whatever was writing to it, which will simply recreate it. Turn off the source first, then delete.

What counts as one custom metric for billing?

One unique combination of namespace, metric name and dimension values. The same metric name published with three different dimension value sets is three billable metrics. This is the single most important sentence in CloudWatch pricing and the easiest one to skim past.

Should I switch log groups to the Infrequent Access class?

For logs you keep but rarely read, it is usually a straightforward saving on ingestion. For any group you extract metrics from, alarm on, or stream live, it is not a candidate. The feature gap between the classes has narrowed since launch, so check current documentation rather than ruling it out on old information.

Is Logs Insights charged per query or per gigabyte?

Per gigabyte scanned. Rows returned are irrelevant. A tightly filtered query over a wide range across many log groups is expensive; a broad query over fifteen minutes on one log group is cheap. Narrow the range before you narrow the filter.

Do VPC Flow Logs have to go to CloudWatch?

No. Flow logs can be delivered to S3 instead, avoiding CloudWatch ingestion and storage entirely, at the cost of losing Insights queries over them. If you analyse flow logs occasionally with Athena rather than continuously in the console, S3 is the better destination.

The one thing to take away

CloudWatch costs are not one number with one cause. They are four meters running independently, and each has a different fix: cut verbosity at the source for ingestion, set retention for storage, narrow scope for queries, bound your dimensions for metrics. Applying the wrong fix to the wrong meter is why so many cost-reduction efforts produce a bill that barely moves.

So attribute first. Split by usage type, rank log groups by IncomingBytes, count custom metrics per namespace. Ten minutes of that tells you which lever you are actually pulling, and stops you deleting the log group that turns out to be the only place a production alarm was getting its data.

Need a hand with your CloudWatch spend?

This is well-defined work with a measurable outcome, which makes it a good fit for a short engagement. Things I can help with:

• Breaking your CloudWatch bill down by meter and by log group, so you know what you are paying for before anything changes
• Auditing custom metric cardinality and identifying which dimensions are multiplying your series count
• Building a retention policy that satisfies your compliance requirement without paying CloudWatch rates for archival data
• Enforcing retention and log class in Terraform or CloudFormation so the fix survives the next deployment
• Designing a subscription filter and S3 archive path for logs you need to keep but rarely query
• Reviewing alarms and dashboards for anomaly detection and API polling costs that are not earning their keep

If you want a second opinion, send me your Cost Explorer breakdown grouped by usage type, or the output of describe-log-groups for the account. That is usually enough to say where the money is going before we talk about anything else.

The post The Real Cost of CloudWatch: Logs, Metrics, Retention and Cardinality appeared first on John Nessime.

That’s the signature failure of telemetry pipelines on AWS. Transport worked, authentication worked, the endpoint answered 200. It kept part of the payload and dropped the rest, because something in the batch broke a limit your collector has no idea exists.

Getting started with OpenTelemetry on AWS is not the hard part. AWS Distro for OpenTelemetry (ADOT) will have you shipping traces in an afternoon. The hard part is four decisions underneath it: where the collector sits, which door telemetry uses to get into AWS, what happens to your metrics on the way through, and whether a second backend later costs you a config line or a migration. This post covers those four decisions and how each one fails without announcing itself.

What ADOT actually buys you

ADOT is not a separate protocol or agent. It’s a downstream build of the upstream OpenTelemetry Collector plus AWS-flavoured SDKs, tested by AWS and covered by AWS Support. The config syntax is the upstream syntax, so a config written for vanilla OTel runs on ADOT and vice versa. The difference is which components are compiled in: ADOT ships the X-Ray exporter, the CloudWatch EMF exporter, the SigV4 extension, and ECS metric receivers already there.

So the choice is narrower than it looks. Take ADOT for a build someone else validates and supports. Take upstream Contrib if you need a component ADOT hasn’t bundled, or you run one collector image across AWS, another cloud, and bare metal. Neither choice touches your instrumentation, which is the entire point.

Decision one: where the collector runs

Three shapes, trading the same three things: blast radius, cost, and how much processing happens before data leaves your network.

Sidecar

One collector container per ECS task or pod. The application talks to localhost, so no service discovery and no network hop to get wrong, and failure stays contained to one workload. The cost: you pay for that container everywhere, batches stay small because batching is per-instance, and any config change is a redeploy of every task.

Agent plus gateway

A light collector per node (a DaemonSet on EKS) forwarding to a small pool of gateways. This is the shape I reach for first on anything past a handful of services. The agent does host-level enrichment, the gateway does the expensive work: large batches, tail sampling, redaction, fan-out. Config changes hit the gateway only.

The catch: a gateway pool is a thing you now operate. It needs autoscaling and its own alerting, and undersized it starts refusing data at exactly the moment you have an incident and volume spikes.

No collector at all

The ADOT SDKs can export straight to the CloudWatch OTLP endpoints, signing requests with credentials already available to the process. For a handful of Lambda functions this is the right answer: nothing to run, nothing to scale, no extra container in the cold start path. What you give up is control. Batching, retries, sampling policy, redaction, and backend routing all move into application processes and their environment variables, so changing any of them is a fleet redeploy rather than a config push. Good starting point, poor steady state.

Decision two: which door into AWS

Two ways to hand telemetry to AWS, and they behave very differently.

The older path uses AWS-specific exporters. awsxray converts OTLP spans into X-Ray segment documents and calls the X-Ray API. awsemf converts OTLP metrics into CloudWatch Embedded Metric Format and writes them as log events. Both work, both are well trodden, and both reshape your data into an AWS-native format on the way out.

The newer path is native OTLP. CloudWatch exposes OTLP endpoints per signal, reached with the plain otlphttp exporter and a SigV4 signer. Same exporter you’d point at Grafana Cloud or Honeycomb, different URL and authenticator. That’s what “export anywhere” means in practice, and it’s why I default to it on new work.

The endpoints follow a per-service pattern:

• Traces: https://xray.<region>.amazonaws.com/v1/traces
• Metrics: https://monitoring.<region>.amazonaws.com/v1/metrics
• Logs: https://logs.<region>.amazonaws.com/v1/logs

Three constraints there will each cost you an afternoon if you don’t know them going in.

HTTP only. No gRPC. If your collector exports over otlp on 4317 you can’t just change the URL, you have to switch exporters. Receiving gRPC from applications is fine, it’s the outbound leg that must be HTTP.

SigV4 required. That means the sigv4auth extension, configured per signal because the signing service name differs: xray, monitoring, and logs respectively. Bearer tokens are an alternative for metrics and logs, but not for traces.

Logs need headers, not just a URL. Target log group and stream travel in x-aws-log-group and x-aws-log-stream headers. Omit them and the request has nowhere to land.

A working shape for traces and logs looks like this:

extensions:
  sigv4auth/traces:
    region: "us-east-1"
    service: "xray"
  sigv4auth/logs:
    region: "us-east-1"
    service: "logs"

exporters:
  otlphttp/traces:
    compression: gzip
    traces_endpoint: https://xray.us-east-1.amazonaws.com/v1/traces
    auth:
      authenticator: sigv4auth/traces

  otlphttp/logs:
    compression: gzip
    logs_endpoint: https://logs.us-east-1.amazonaws.com/v1/logs
    headers:
      x-aws-log-group: MyApplicationLogs
      x-aws-log-stream: default
    auth:
      authenticator: sigv4auth/logs

service:
  extensions: [sigv4auth/traces, sigv4auth/logs]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp/traces]
    logs:
      receivers: [otlp]
      exporters: [otlphttp/logs]

Note traces_endpoint and logs_endpoint rather than endpoint. This one bites people. The endpoint setting is a base URL and the exporter appends the signal path itself, so endpoint: https://monitoring.us-east-1.amazonaws.com/v1/metrics actually requests /v1/metrics/v1/metrics. The per-signal settings take a full path. If you’re getting 404s from a URL you’re certain is right, this is why.

Also worth a comment in your config: upstream has renamed the component to otlp_http and otlphttp is now a deprecated alias scheduled for removal. Both work today.

Traces need Transaction Search turned on first

This is the single most common reason a correctly configured trace pipeline produces nothing. The X-Ray OTLP endpoint requires Transaction Search to be enabled on the account, which redirects span ingestion into CloudWatch Logs:

aws xray update-trace-segment-destination --destination CloudWatchLogs

Spans then land in a log group named aws/spans, with a percentage indexed in X-Ray as trace summaries for search. The default index rate is one percent, enough to find traces while all spans stay queryable as structured logs. The caller needs xray:UpdateTraceSegmentDestination and xray:UpdateIndexingRule plus log group creation rights.

Two consequences to plan for. Span ingestion is billed separately from log ingestion, so it’s a line item rather than a rounding error. And AWS recommends always_on sampling in the SDK on this path, because the indexing rule already handles reduction. Sample in both places and your service map goes patchy.

The X-Ray SDK clock is running

The dates matter here, so plainly: the X-Ray SDKs and daemon entered maintenance mode on 25 February 2026 (security fixes only, no new instrumentation support), with end of support on 25 February 2027. The X-Ray service is fine and still gaining features. It’s the client libraries and the UDP daemon winding down. If you run aws-xray-sdk and a daemon sidecar, that is technical debt with a published expiry, and either the collector or the CloudWatch agent replaces the daemon.

Decision three: metrics, temporality, and the cardinality bill

Metrics are where the silent drops live, because collector defaults and endpoint limits disagree.

The metrics endpoint caps a single request at 1 MB uncompressed and 1,000 datapoints, counted across resource, scope, and metric levels combined. The upstream batch processor’s default is far larger. Leave it alone and you build oversized requests, and the response to an oversized or partially invalid request is not always a clean failure: it can come back 200 with some metrics accepted and others rejected or throttled. So set the batch size deliberately. AWS’s own examples use a conservative value:

processors:
  batch:
    send_batch_size: 200
    timeout: 10s

The other limits shape attribute design more than config:

• 150 labels maximum across resource, scope, and datapoint attributes per datapoint
• 40 KB combined label and value size per series per datapoint
• One million new series creatable per ten-minute window, per account
• Timestamps no more than ten minutes in the future or fourteen days in the past

That new-series ceiling catches teams out. Attach a request ID, a session ID, or a raw URL path to a metric attribute and every request mints a fresh series. You’ll hit a million faster than you expect, and it presents as metrics randomly going missing rather than a quota error. Unbounded identifiers belong on spans and logs. Metric attributes should be values you could enumerate on a whiteboard.

Temporality is the other conscious decision. The OpenTelemetry SDK spec defaults to cumulative, while CloudWatch’s metric model is delta-shaped: it wants what happened this period, not the total since your process booted. Counters that look like ever-climbing staircases instead of rates are that mismatch. The cumulativetodelta processor converts in the pipeline, keeping the decision in the collector rather than scattered across SDK environment variables in every service.

Decide once and centrally either way. Mixed temporality across one account produces dashboards that are subtly wrong for months before anyone notices. And once volume climbs, pointing a cost tool such as Vantage or CloudZero at the CloudWatch line items earns its setup time, because telemetry spend grows in steps nobody approved.

Decision four: a second backend without a second instrumentation

Here’s the payoff. A collector pipeline holds multiple exporters, and adding one is a config change:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/traces, otlphttp/vendor]

The same spans now reach CloudWatch and Grafana Cloud, Honeycomb, Datadog, or a self-hosted Tempo, with no application redeploy and no second agent. That’s the cashable value of instrumenting once against an open protocol: the switching cost of a backend drops from a migration project to a pull request.

Be honest about the price. You pay ingest twice, and egress from the gateway is real money at volume. Collector memory scales with exporter queue count, so a fan-out gateway needs headroom. And if one backend slows, backpressure can reach the pipeline feeding the other, which argues for separate pipelines per destination when their reliability differs.

It works in reverse too. Because the metrics and logs endpoints accept bearer tokens, a machine with no AWS credentials at all can ship into CloudWatch: a CI runner, another cloud, or a VPS at a provider like Contabo or InterServer running part of your stack. One collector config, one destination, regardless of who owns the hardware. Traces are the exception and still require SigV4.

Why OpenTelemetry on AWS fails quietly

Most breakages here don’t throw. How to recognise the common ones:

Data is missing but nothing errors

Almost always a limit breach inside an accepted request. Check batch sizes against the per-signal caps first, then attribute counts. Turn on the collector’s internal telemetry and compare the exporter’s sent counters against its failed counters. If sent looks healthy and data is still missing, the loss is happening server-side after acceptance, which narrows it to limits.

403 with a signature mismatch

A SigV4 problem, not an IAM problem. The usual cause is the wrong signing service name in the extension, since xray, monitoring, and logs are not interchangeable. If you sign requests yourself rather than using the extension, note that the traces endpoint is stricter about which headers land in the signed set, so sign a minimal stable set rather than everything the HTTP client added.

Backfilled data vanishes

Every endpoint enforces a timestamp window, and fourteen days in the past is the outer edge for all three signals. Replaying an old queue past that boundary gets rejected. Not a bug, but it looks like one when the replay job appears to succeed and produces nothing.

Traces arrive but the service map is empty

Usually a missing or inconsistent service.name resource attribute, or double sampling between the SDK and the indexing rule. Confirm what’s actually leaving the collector before hunting in the console: add the debug exporter to a copy of the pipeline, set verbosity: detailed, and read the resource attributes on real spans.

Common mistakes

• Pointing an OTLP gRPC exporter at a CloudWatch endpoint. They’re HTTP only, and the error won’t say so clearly.
• Using endpoint with a full signal path instead of the per-signal traces_endpoint, metrics_endpoint, or logs_endpoint.
• Leaving the batch processor at its default size on the metrics pipeline.
• Sampling in the SDK and relying on Transaction Search indexing rules, halving visibility twice over.
• Putting request IDs, user IDs, or raw paths into metric attributes.
• Running a gateway pool with no alerting on the collector’s own health, so saturation is invisible until you need it.

Best practices

• Keep instrumentation vendor-neutral. Plain OpenTelemetry SDK APIs, with the collector owning every AWS-specific decision.
• Set service.name, service.version, and deployment.environment as resource attributes everywhere. Almost every correlation feature depends on them.
• Alert on exporter failure counters and queue depth, and route those alerts somewhere that doesn’t depend on the pipeline being healthy.
• Version the collector config in Git and deploy it like application code, with a staging pipeline you can break safely.
• Redact in a processor before export. Once telemetry reaches a backend, removing it is a support ticket.
• Test a second exporter early, even to a throwaway account. A vendor migration is the wrong time to learn the fan-out path doesn’t work.

Frequently asked questions

Do I still need a collector if CloudWatch accepts OTLP directly?

Not to get data in, no. You need one to control what happens before it leaves: batching to fit endpoint limits, tail sampling, redaction, and fan-out. Small serverless estates can reasonably skip it. Anything with a dozen services will want one.

Can I send OTLP to CloudWatch over gRPC?

No. The CloudWatch OTLP endpoints are HTTP 1.1 only, accept binary or JSON payloads, and support gzip or no compression. Your collector can still receive gRPC from applications, it just can’t forward over it.

What’s the difference between ADOT and the upstream collector?

Same codebase, different build. ADOT is AWS’s tested distribution with AWS components bundled and AWS Support behind it; upstream Contrib carries a wider component set and moves faster. Configuration is compatible either way, so it’s a support and packaging choice, not an architectural one.

Do I have to enable Transaction Search to send traces?

For the X-Ray OTLP endpoint, yes. It’s a prerequisite, and enabling it routes span ingestion through CloudWatch Logs. Using the awsxray exporter against the classic X-Ray API instead doesn’t require it, but you give up the span-level analytics Transaction Search provides.

Can I ship telemetry to CloudWatch from outside AWS?

For metrics and logs, yes, using bearer token authentication instead of SigV4, which removes the need for AWS credentials on the host. Traces still require SigV4. Never hardcode the token in the config; read it from a mounted secret file or an injected environment variable.

Is it urgent to migrate off the X-Ray SDK?

Not an emergency, but it’s on a clock: maintenance mode now, end of support 25 February 2027. Existing applications keep working, they just won’t get new library instrumentation. Plan the migration on your schedule rather than someone else’s.

The one thing worth remembering

A pipeline that returns 200 is not a pipeline that works. Judge every design decision in OpenTelemetry on AWS against one question: when this drops data, do I find out from a dashboard or from a colleague asking why a service vanished off the map?

Size batches against the published limits, keep unbounded identifiers off metric attributes, decide temporality in one place, and instrument with plain OpenTelemetry so the export target stays a config line. Do that and “instrument once, export anywhere” stops being a slogan and becomes a property you can test.

Need help with your OpenTelemetry pipeline on AWS?

Most of this work is unglamorous and specific, which is why it gets deferred. Things I can help with:

• Auditing an ADOT or upstream collector config for silent drops, oversized batches, and limit breaches
• Designing collector topology for ECS, EKS, or Lambda, including gateway sizing and autoscaling
• Migrating X-Ray SDK and daemon workloads to OpenTelemetry ahead of end of support
• Cutting telemetry spend through sampling policy, cardinality control, and attribute pruning
• Dual export to CloudWatch and a third-party backend, so a future switch is a config change
• Meta-monitoring that tells you the pipeline is broken before your users do

Send me a collector config, an exporter log, or a screenshot of the gap in your dashboard and I’ll tell you what I’d look at first.

The post Instrument Once, Export Anywhere: OpenTelemetry on AWS With ADOT appeared first on John Nessime.

That gap is the entire problem. Monitoring n8n workflow failures in Grafana is not hard because the tooling is missing. It’s hard because the built-in metrics endpoint answers a question you didn’t ask. It tells you the process is alive and busy. It does not tell you that a workflow stopped producing correct output.

This post covers what n8n’s Prometheus endpoint actually exposes, the three families of failure it handles badly, and the specific PromQL and alert rules that close each gap. There’s also a troubleshooting section for the case where /metrics returns nothing at all, which is the first wall most people hit.

What the n8n metrics endpoint actually gives you

n8n uses the prom-client library and serves metrics on the same port as the editor, at /metrics. The endpoint is off by default, and almost everything interesting inside it is off by default too, behind its own environment variable.

N8N_METRICS=true
N8N_METRICS_INCLUDE_MESSAGE_EVENT_BUS_METRICS=true
N8N_METRICS_INCLUDE_WORKFLOW_ID_LABEL=true
N8N_METRICS_INCLUDE_NODE_TYPE_LABEL=true
N8N_METRICS_INCLUDE_QUEUE_METRICS=true

The first line opens the endpoint. The second turns on the workflow and node event counters, which is the part people assume is included and is not. The third attaches a workflow_id label to workflow metrics, without which you get a single instance-wide number and no way to tell which automation broke.

Those labels are opt-in for a reason worth understanding rather than working around. Every distinct label value creates a separate time series. Turn on workflow_id and you get one series per workflow per counter; add node_type and node counters fan out across every integration you use. On a few dozen workflows that’s fine. On several hundred with per-node labels, you’re signing up for a cardinality problem that surfaces later as slow queries and a Prometheus process that keeps getting OOM-killed.

With the event bus metrics enabled, the counters you get include n8n_workflow_started_total, n8n_workflow_success_total, n8n_node_started_total and n8n_node_finished_total. Queue mode adds n8n_scaling_mode_queue_jobs_active, _completed, _failed and _waiting, gathered from Bull and exposed on the main instance only.

Prometheus side, the scrape job is unremarkable:

scrape_configs:
  - job_name: n8n
    static_configs:
      - targets:
          - n8n-main:5678
          - n8n-worker-1:5678
    metrics_path: /metrics

Note both main and workers. Each process exposes its own endpoint and its own counters. Scrape only the main and you lose everything the workers did, which in queue mode is nearly all the real execution work.

One thing the n8n docs are emphatic about, and they’re right: don’t put /metrics on the public internet. It leaks workflow IDs, execution volumes and instance topology to anyone who asks. Keep the scrape path on a private network segment, a WireGuard tunnel, or your provider’s internal networking. If you’re running the whole stack on a single VPS from somewhere like Contabo or InterServer, binding n8n to the Docker network and scraping it by container name is enough. Publishing port 5678 to the host and firewalling it afterwards is not, because you will forget.

Failure family one: the execution that never happened

This is the one that costs the most and shows up the least. A schedule trigger gets deactivated during a deploy. A credential expires and the workflow is switched off rather than fixed. An upstream service stops calling a webhook. Nothing errors. Nothing increments. The workflow simply stops existing as far as your metrics are concerned.

Every error-rate alert you write is blind to this, because an error rate needs a denominator. Zero executions divided by zero executions is not an alert, it’s a NaN.

The instinct is to write this:

sum by (workflow_id) (increase(n8n_workflow_started_total[6h])) == 0

It looks correct and it does not work. When a workflow stops running, its series stops being exported. Prometheus marks it stale after a few missed scrapes and instant queries stop returning it. There is no series left for == 0 to match against. The alert never fires, because the thing you wanted to alert on is exactly the thing that removed the data.

What you want is a function that returns something when a series is absent:

absent_over_time(
  n8n_workflow_started_total{workflow_id="AbC123XyZ"}[6h]
)

absent_over_time returns 1 if the matched series produced no samples in the window and nothing at all if it did. That inversion is the point. The alert fires on silence.

The cost is that absent_over_time needs an exact matcher, so you can’t write it once for everything. That’s more acceptable than it sounds. You don’t need a heartbeat on all two hundred workflows, you need one on the six a human would notice within an hour. Write those six by hand and treat the list as part of the runbook.

One Grafana-specific trap. When a query returns nothing, Grafana’s alert rule enters a No Data state, which is configurable separately from the alert condition and defaults to firing in most versions. Set it deliberately. For an absent_over_time rule, No Data means the whole scrape target is gone, which is a different incident from one workflow going quiet, and you probably want it routed differently.

Failure family two: the failure you can see but can’t attribute

You have the counters. Something is failing. Which workflow?

Without N8N_METRICS_INCLUDE_WORKFLOW_ID_LABEL, the answer is: no idea. The counters are instance-wide totals. You know the number moved and that’s it. That’s a fine signal for capacity planning and useless during an incident.

Turn the label on and you get IDs, which are opaque strings like AbC123XyZ. A dashboard full of those is technically attributable and practically unreadable at 3am. Newer n8n versions solve this with a gauge that maps IDs to names, enabled with N8N_METRICS_INCLUDE_WORKFLOW_INFO:

n8n_workflow_info{workflow_id="VaQPuPmx9tPpo6BX",workflow_name="Invoice sync"} 1

It’s a constant-1 gauge that exists only to be joined against. The pattern is a standard Prometheus vector match:

sum by (workflow_name) (
  rate(n8n_workflow_started_total[5m])
  * on(workflow_id) group_left(workflow_name)
  n8n_workflow_info
)

The multiplication is arithmetically a no-op because the gauge is always 1. Its only job is to carry the workflow_name label across with group_left. Do this once in a Grafana variable or a recording rule rather than in every panel, or you’ll be copy-pasting the join for the rest of your life.

On an older build without the info gauge, the fallback is a hand-maintained relabel config or a Grafana value mapping. Both drift, and the relabel version drifts silently.

Failure family three: the failure that counts as a success

This is the invisible one, and it’s why a green n8n dashboard is not evidence of anything.

n8n gives you several deliberate ways to make a node’s failure not be an execution failure. They’re all reasonable features. Together they mean the success counter and “the automation worked” are different statements.

• Continue on error. A node set to continue passes its error downstream instead of stopping the run. The execution finishes. It finishes as a success.
• Error output branches. A node with a separate error output routes failures to a different path. That path might log and stop. The execution still completes normally.
• Never Error on HTTP responses. The HTTP Request node can be configured to treat any status code as a valid response. A run of 500s from an upstream API becomes a workflow that processed zero useful records, successfully.
• Empty result sets. A query that returns nothing is not an error. A sync that syncs nothing, every night, looks exactly like a sync that has nothing to do.

There are also user reports on the n8n community forum of started and success counters matching even when a node failed outright, in queue mode. I have not reproduced that and I’d treat it as unconfirmed rather than a known behavior. But it’s the right instinct: don’t assume the success counter is a strict complement of failure. Verify it on your own instance before you build alerting on the assumption.

For webhook-triggered workflows, newer n8n versions give you a real answer here. The n8n_webhook_request_duration_seconds histogram carries a status_code label, so you can alert on what the caller actually received rather than on what n8n thought about the run:

sum by (workflow_id) (
  rate(n8n_webhook_request_duration_seconds_count{status_code!~"2.."}[5m])
)

Worth flagging: the n8n documentation shows this example with status_code!="2..", using the equality operator. In PromQL, != compares against a literal string, so that matcher excludes only a status code whose literal value is the three characters 2.., which never occurs. It matches everything, including your 200s. Use !~ for the regex form. Test any label matcher you copy from documentation by running it in Prometheus and checking the series count against what you expect.

Getting a second source of truth

For everything that isn’t a webhook, the metrics endpoint alone won’t close this gap. You need a second signal that reads execution outcomes from n8n’s own store.

The public REST API exposes exactly that. Executions can be filtered by status, and the failed ones are one request away:

curl -s -H "X-N8N-API-KEY: $N8N_API_KEY" 
  "https://n8n.example.com/api/v1/executions?status=error&limit=50"

Two ways to get that into Grafana. Write a small exporter that polls the API on a timer and serves its own /metrics, or run a scheduled job that pushes counts to a Prometheus Pushgateway. The exporter is more work and behaves correctly. Pushgateway is faster to build and carries the usual caveat: pushed metrics persist until deleted, so a job that stops running leaves a stale value sitting there looking healthy. That reintroduces failure family one through the back door.

Two caveats on the API. There’s an open report that the status=crashed list filter returns executions whose stored status is something else entirely when re-fetched by ID, which would make crash alerting built on the list endpoint unreliable. Check the behavior on your version before depending on it. And n8n’s log streaming feature, which forwards execution events to syslog or a webhook in real time, is an Enterprise capability. If you’re on the community edition, the API poll is your path.

The third option, and often the best one for a small setup, is an Error Trigger workflow. n8n lets you designate a workflow that runs whenever another workflow fails. Point it at Pushgateway, at Loki, or at a Grafana webhook contact point. It’s native, it needs no API key, and it gives you the error message rather than just a count. The trade-off is that it only fires on real execution failures, so it inherits every blind spot in this section.

The Grafana panels that earn their space

Four panels cover most of what you need. Everything past that is usually decoration.

1. Executions per workflow, by name. A time series of rate(n8n_workflow_started_total[5m]) joined to n8n_workflow_info. Its real job is showing you the lines that flatten to zero.
2. Success ratio. rate(n8n_workflow_success_total[15m]) / rate(n8n_workflow_started_total[15m]). Anything below 1 is worth a look. Anything above 1 means your window is catching a restart and the counters reset.
3. Queue depth and failed jobs. n8n_scaling_mode_queue_jobs_waiting next to increase(n8n_scaling_mode_queue_jobs_failed[1h]). A waiting count that climbs and never drains means workers are dead or starved, which looks identical to “no failures” on every other panel.
4. Webhook non-2xx rate. The query from the previous section. This is the only panel that reports what the outside world experienced.

Skip the node-level breakdown unless you have a specific reason. It is the most expensive thing you can enable in cardinality terms and it answers a question you can answer faster by opening the execution in the n8n UI.

Alert rules worth paging on

1. Named workflow silent. absent_over_time on each business-critical workflow, window set to roughly twice its expected interval. This is the rule that catches the invoice sync.
2. Success ratio below threshold. Sustained for 15 minutes, not instantaneous. A single failed run is not a page.
3. Queue waiting count rising. Compare the current value to fifteen minutes ago rather than to a fixed number, so the threshold survives traffic growth.
4. Webhook error rate. Per workflow, so a single noisy endpoint doesn’t mask the rest.
5. Scrape target down. The up metric. Obvious, and routinely forgotten, and it invalidates every other rule on this list.

Troubleshooting: the endpoint returns nothing useful

• “Cannot GET /metrics”. N8N_METRICS isn’t set on the process you’re hitting, or it’s set as a string that n8n doesn’t read as true. In Docker Compose, quote it as "true" and confirm with docker compose exec n8n env | grep N8N_METRICS.
• Endpoint responds but has no n8n_ counters. You have the default Node.js process metrics and nothing else. Set N8N_METRICS_INCLUDE_MESSAGE_EVENT_BUS_METRICS=true and restart.
• Counters exist but no workflow_id label. N8N_METRICS_INCLUDE_WORKFLOW_ID_LABEL is off. Restart is required; it’s read at startup.
• Queue metrics are zero or missing. They’re gathered from Bull and exposed on the main instance, not the workers. Querying the worker endpoint gives you nothing. Confirm you’re looking at the right target.
• Counters look wrong after a deploy. They reset to zero when the process restarts. Always wrap them in rate() or increase(), which handle counter resets. Raw counter values on a dashboard will lie to you every time you deploy.
• Prometheus can’t reach the target. Usually a Docker network mismatch. Scrape by container name on the shared network rather than by published host port, and check the target page in Prometheus for the actual error rather than guessing.

Common mistakes

• Treating the success counter as proof the automation worked. It proves the run finished.
• Building only error-rate alerts, so the workflows that stop entirely go unmonitored.
• Enabling every N8N_METRICS_INCLUDE_* variable on day one and discovering the cardinality bill a month later.
• Scraping only the main instance in queue mode and losing all worker-side execution data.
• Copying label matchers out of documentation without running them and checking the returned series count.
• Exposing /metrics publicly because it’s “just numbers”.

Best practices

• Decide which workflows are business-critical and write a silence alert for each one by name. Six good heartbeats beat two hundred generic panels.
• Run a second source of truth. Metrics tell you about volume, the executions API tells you about outcomes. You need both.
• Push the ID-to-name join into a recording rule so panels stay readable and cheap.
• Break a staging workflow on purpose and confirm the alert fires. An untested alert is a decoration.
• If you’d rather not run Prometheus and Grafana yourself, Grafana Cloud will scrape a remote endpoint and gives you alerting without a control plane to maintain. Weigh that against your metrics leaving your network.

FAQ

Does n8n expose a failed-executions counter?

Not a reliable per-workflow one. You get started and success counters from the event bus, and a failed-jobs counter for queue mode. Deriving failures by subtracting success from started works only if you’ve verified on your own instance that a failing run genuinely doesn’t increment success, which is exactly the assumption that breaks when nodes are set to continue on error.

Can I monitor n8n Cloud this way?

No. The /metrics endpoint isn’t available on n8n Cloud. On Cloud you’re limited to the public API and the built-in insights views, so the executions-API approach in this post is the only one that transfers.

Why does my workflow error rate look like zero when I know things are failing?

Almost always one of three things: the event bus metrics aren’t enabled so you’re only seeing Node.js process metrics, the failures are being absorbed by continue-on-error settings and recorded as successes, or you’re scraping the main instance while the work happens on workers.

How do I alert when a scheduled workflow stops running entirely?

Use absent_over_time against that workflow’s started counter with an exact workflow_id matcher, and set the window to about twice the expected interval. A threshold comparison won’t work, because a workflow that stops running stops producing the series you’d compare against.

Is enabling the workflow_id label safe on a large instance?

It multiplies your n8n series count by the number of workflows. On a few dozen it’s negligible. Past a few hundred, combined with node-type labels, watch your Prometheus memory and head series count before and after. Enable the workflow label, leave the node label off, and only turn the node one on when you have a specific question it answers.

Should I use OpenTelemetry instead of Prometheus for n8n?

They solve different problems. Metrics answer “how many, how often, how fast” cheaply and forever. Traces answer “where did this run spend its time and what did it call”, which is better for debugging a slow multi-service workflow. Tracing n8n usually means a custom image with the OpenTelemetry SDK loaded before startup, so it’s a project rather than an environment variable. Start with metrics; add tracing when you have a latency question metrics can’t answer.

Where do the n8n counters go when the container restarts?

They reset to zero. They’re in-process counters with no persistence. Prometheus handles this correctly as long as your queries use rate() or increase(). Any panel showing a raw counter value will appear to lose all its history on every deploy.

The one thing to take away

Monitoring n8n workflow failures in Grafana works, but only once you stop asking the metrics endpoint a question it can’t answer. It reports activity. Activity and correctness are different things, and the space between them is where the expensive outages live.

So build for absence first. A silence alert on the handful of workflows a human would miss will catch more real incidents than any error-rate panel, because errors are loud and stopped schedules are not. Then add attribution with the workflow label, then close the false-success gap with the executions API or an Error Trigger workflow. In that order, because each one is cheap and the value drops off fast after the third.

Need help wiring this up?

I build and maintain observability stacks for self-hosted automation platforms, and n8n is a common one. If any of the above sounds like a week you’d rather not spend, here’s what I can take off your plate:

• Enabling and securing the n8n metrics endpoint on a private network path, including queue-mode setups with multiple workers
• Building Prometheus scrape configs, recording rules and the ID-to-name join so your Grafana panels are readable rather than a wall of hashes
• Writing silence and error-rate alert rules that survive counter resets, Grafana No Data states and traffic growth
• Adding a second source of truth from the n8n executions API, either as a small exporter or an Error Trigger workflow feeding Loki or Pushgateway
• Auditing an existing setup for cardinality problems, stale Pushgateway series and alerts that quietly stopped being able to fire
• Running a failure drill against staging so you know which alerts actually work before you need them

Send me your Compose file, your prometheus.yml, or a screenshot of the panel that looks fine while things are on fire, and I’ll tell you what’s missing.

The post Monitoring n8n Workflow Failures in Grafana: Why Your Green Dashboard Is Lying appeared first on John Nessime.