The rule was fine. The query was fine. The evaluation ran on schedule and the state transition happened exactly when it should have. What failed was everything after that, and none of it made a sound.

That is the failure mode worth designing around. Rule evaluation and notification delivery are two different systems, and a red rule only proves the first one works. Grafana alert notifications run against SMTP servers, Slack’s API and Telegram’s bot endpoint, all of which fail in their own quiet ways. A rejected login, a revoked webhook, a message over a length limit: each produces a log line and nothing else. The alert list still shows the rule firing, so everything looks healthy from the one place you’d think to check.

This post covers the practical setup for the three channels most small and mid-sized teams actually use: email, Slack and Telegram. It covers what breaks in each of them, how to route alerts so the right ones interrupt people and the rest don’t, how to provision the whole thing as files instead of clicking through the UI, and how to detect the case where delivery has stopped entirely.

Where a firing alert actually goes

Before touching any integration, it helps to know how many hops sit between “the rule fired” and “my phone buzzed”.

• The rule evaluates on its own schedule and produces alert instances, each carrying labels.
• Those instances go to an Alertmanager. Grafana ships its own and can also use an external Prometheus Alertmanager.
• The Alertmanager applies mute timings, groups instances, and walks the notification policy tree for a match.
• The matched policy names a contact point, which holds one or more integrations.
• Each integration makes its own outbound call to a third-party service.

Steps one and two are visible in the Grafana UI. Steps three through five are where things go wrong, and they are visible only in logs. That asymmetry is the whole problem.

One thing worth knowing early: a contact point can hold several integrations at once, so a single contact point named critical can fire email, Slack and Telegram together. Building one contact point per channel instead makes routing much harder than it needs to be.

Email: the channel everyone assumes works

Email is the one people configure first and verify least. The catch is that the email contact point holds only the recipient addresses. The actual SMTP credentials live in grafana.ini, so a perfectly valid contact point can sit there doing nothing because the server underneath was never configured.

[smtp]
enabled = true
host = smtp.example.com:587
user = alerts@example.com
# Wrap the password in triple quotes if it contains # or ;
password = """your-smtp-password"""
from_address = alerts@example.com
from_name = Grafana Alerts
skip_verify = false
# Default is OpportunisticStartTLS
startTLS_policy = MandatoryStartTLS

Several details in that block matter more than they look.

• The port is part of host. There is no separate port setting. Port 587 with STARTTLS is the normal choice; 465 is implicit TLS and behaves differently, which is usually why one fails cleanly and the other hangs.
• startTLS_policy decides how hard Grafana insists on encryption. The default is opportunistic: it upgrades the connection if the server offers it and quietly continues in the clear if not. On a relay you control, mandatory turns a silent downgrade into a visible failure.
• from_address has to be something the relay accepts. Managed relays reject mail from unverified domains after the connection succeeds, which is why the Grafana side looks fine.
• Leave skip_verify false. It exists for internal relays with self-signed certificates. Turning it on to clear an error removes the check that was telling you the certificate is wrong.

Changes to this file need a Grafana restart. Nothing in the UI reloads SMTP settings.

Deliverability is a separate problem from delivery

Sending directly from a VPS is the fastest way to get alerts filed as spam. A fresh IP range from a provider like Contabo or InterServer has no sending reputation, and a mail server that emits one message a week during incidents never builds one.

Point Grafana at a transactional relay instead. Amazon SES, Postmark, Mailgun and Brevo all do the same job at this volume. Whichever you pick, publish SPF and DKIM records for the sending domain and set a DMARC policy, then send a real alert to a Gmail address and a Microsoft 365 address and check the spam folder. That is the only test that tells you anything.

Slack: webhook or bot token

Grafana supports two ways of posting to Slack, and the choice has consequences beyond setup time.

Incoming webhooks are the quick path. Slack generates a URL under hooks.slack.com/services/ that is bound to a single channel, and Grafana posts to it. There’s no OAuth flow and no scopes to reason about. The trade-off is that the URL is the credential and it maps to exactly one channel, so a team with six alert channels ends up managing six secrets with no central view of them.

A Slack app with a bot token takes longer. You create the app, add the chat:write.public scope so it can post to public channels without being invited to each one, then copy the bot user OAuth token, which starts with xoxb-. In Grafana you supply that token plus the channel ID as the recipient. One credential covers every channel, permissions are visible in one place, and revoking it kills all posting at once.

Webhooks win when you have one or two channels and want to be done in five minutes. The bot token wins as soon as channels start multiplying or someone asks what the integration is permitted to do. I reach for the webhook on small setups and regret it around the fourth channel.

Two constraints either way. A Slack integration targets one channel, so several channels means several integrations. And notification templates control the message title and body text, not its visual structure, so rich block layouts aren’t on the table.

Telegram: cheap paging, one nasty edge

Telegram is genuinely useful for out-of-hours alerts because it pushes to a phone without anyone paying for a paging product. The setup is two values: a bot API token and a chat ID.

1. Message BotFather and send /newbot. The username has to end in bot or _bot.
2. Copy the HTTP API token it returns. That’s the bot API token Grafana wants.
3. Create a group for alerts and add the bot to it. A bot cannot message a chat it has never seen.
4. Send a slash command in the group, such as /start@your_bot_name. An explicit command is the reliable way to make the group visible to the bot.
5. Fetch the chat ID from the Telegram API.

curl -s "https://api.telegram.org/bot<YOUR_BOT_TOKEN>/getUpdates" | jq '.result[].message.chat'

The id field in that output is what goes in Grafana’s Chat ID field. Group chat IDs are negative. Copy the minus sign; leaving it off is the single most common reason a Telegram contact point tests as failing.

Now the edge case, and it’s the reason Telegram belongs in a post about invisible failures. Telegram caps messages at 4096 UTF-8 characters. If you’re using a parse mode other than none, truncating at that boundary can leave the markup unbalanced, and Telegram rejects the whole message as malformed. You don’t get a shortened alert. You get nothing.

This is exactly backwards, because the message most likely to blow the limit is a large grouped alert during a real outage. The channel works perfectly for months of single alerts, then goes quiet the first time forty instances fire together. Three ways to deal with it, in the order I’d try them:

• Write a deliberately short notification template for Telegram: alert name, severity, instance count and a dashboard link. Treat it as a pager, not a report.
• Narrow group_by on the policy that feeds Telegram so a single notification covers fewer instances.
• Set the parse mode to none, which makes truncation harmless at the cost of formatting.

Routing: split by severity, not by tool

The most common design mistake is a policy tree organised around channels: a Slack policy, an email policy, a Telegram policy. That gives you no way to express the thing you actually care about, which is that some alerts should wake someone up and most should not.

Put severity on the alert rule as a label and route on that instead. Contact points then become bundles of channels chosen to match urgency.

• severity=critical goes to a contact point holding Telegram and Slack. Something is broken and someone needs to look now.
• severity=warning goes to Slack only, in a channel people read during working hours.
• Everything else falls through to the default policy and lands in email, where it accumulates without interrupting anyone.

Four timing settings control how noisy this is, and each one does something specific:

• group_by decides what counts as the same notification. Grouping by alertname and a service label means one cascading failure produces one message instead of forty.
• group_wait (default 30 seconds) is how long to hold the first notification for a new group, so related instances that fire a few seconds apart arrive together.
• group_interval (default 5 minutes) is the minimum gap before a group that has already notified sends again because new instances joined it.
• repeat_interval (default 4 hours) is how often an unresolved alert re-notifies with nothing new to say.

The temptation is to drop repeat_interval to fifteen minutes so nothing gets forgotten. Resist it. Teams that do this stop reading the channel within a month, which converts a routing problem into a much worse cultural one. Longer intervals plus a real escalation path beats short intervals every time.

Grafana also offers a simpler path: pick a contact point directly on the alert rule form and skip label matching entirely. Reasonable when a rule has one obvious destination. The trade-off is that routing knowledge moves into individual rules, so changing where a class of alerts goes means editing every rule instead of one policy.

Provision it as files, not clicks

Contact points built through the UI are invisible to review, easy to delete by accident, and impossible to recreate after a bad restore. If Grafana alert notifications matter enough to build, they matter enough to keep in Git. Grafana reads alerting configuration from files in the provisioning/alerting directory under its provisioning path at startup.

Before writing any YAML by hand, build one contact point in the UI, test it, then use the export option on the contact points list to dump it as YAML. That gives you the exact setting keys for that integration type instead of the ones you half-remember from a blog post. Setting names are not consistent between integration types, and a guessed key produces a config that loads without error and sends nothing.

apiVersion: 1

contactPoints:
  - orgId: 1
    name: critical-oncall
    receivers:
      # Telegram receiver goes here. Export it from the UI first:
      # the setting key names differ from what you would guess.
      - uid: critical-slack
        type: slack
        settings:
          url: https://hooks.slack.com/services/REPLACE/WITH/YOURS
        disableResolveMessage: false

  - orgId: 1
    name: warnings-slack
    receivers:
      - uid: warnings-slack
        type: slack
        settings:
          url: https://hooks.slack.com/services/REPLACE/WITH/YOURS

  - orgId: 1
    name: email-digest
    receivers:
      - uid: email-digest
        type: email
        settings:
          addresses: ops@example.com
          singleEmail: false

And the policy tree that uses them:

apiVersion: 1

policies:
  - orgId: 1
    receiver: email-digest
    group_by: [grafana_folder, alertname]
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - receiver: critical-oncall
        matchers:
          - severity = critical
        group_wait: 10s
        repeat_interval: 1h
      - receiver: warnings-slack
        matchers:
          - severity = warning
        repeat_interval: 12h

Two things to know about provisioned resources. By default they become read-only in the UI, which is the point but surprises people the first time. And a contact point that a policy references cannot be deleted until the policy stops referencing it, so removals have to happen in the right order.

Keep tokens out of the files themselves. Grafana supports environment variable substitution in provisioning files, which means the YAML can live in Git and the secrets can come from your existing secret store.

Detecting the silence

Everything above still leaves the original problem unsolved. If the Slack app gets revoked or the SMTP password rotates, notifications stop and Grafana does not tell you.

The fix is a heartbeat alert, and it’s worth the twenty minutes. Create a rule whose condition is always true, give it a low severity label, route it to each contact point with a short repeat_interval, and point the resulting notification at an external dead-man’s-switch service. Healthchecks.io, Cronitor and Better Stack all offer this, and Healthchecks.io can be self-hosted if you’d rather not add a dependency. The service expects a signal on a schedule and alerts you when it stops arriving.

The logic is inverted on purpose. You aren’t monitoring infrastructure here, you’re monitoring the path that tells you about your infrastructure, and it has to be watched by something that isn’t itself.

As a second layer, Grafana exposes its own internal metrics on a Prometheus-compatible endpoint, including counters for notification attempts and failures. If you’re already scraping Grafana, alert on that failure counter increasing. It won’t catch a credential that works but delivers to a dead mailbox, which is why the heartbeat comes first.

Troubleshooting: the rule fires, nothing arrives

Work down this list in order. Each step rules out one hop, and stopping early is how people spend an afternoon on the wrong layer.

1. Test the contact point in isolation. The Test button sends a notification without touching rules or policies. If it fails, the problem is credentials or network and nothing else matters yet.
2. Watch the Grafana logs during the test. On a systemd install, run journalctl -u grafana-server -f while you click Test. SMTP, Slack and Telegram delivery errors all surface here, usually specific enough to fix directly.
3. Compare the instance’s labels against your matchers. A policy matching severity = critical never sees severity = Critical. Matchers are exact and case-sensitive. Read the firing instance’s real labels, not the ones you think the rule sets.
4. Look for a policy earlier in the tree that swallowed it. Evaluation stops at the first match unless the policy continues. A broad route near the top silently absorbs everything below it.
5. Check active silences and mute timings. A silence created during last month’s maintenance with a generous matcher is the classic. So is a mute timing in the wrong timezone.
6. Check whether you’re inside repeat_interval. A red alert now does not mean a notification is due now. This looks identical to a broken integration and isn’t.
7. For email, read the relay’s own logs. Grafana handing the message off is not delivery. Bounces and spam filing happen after Grafana’s involvement ends.

Common mistakes

• Configuring the email contact point and never touching grafana.ini. The contact point is only a recipient list. Without the SMTP block, it has nothing to send with.
• Routing by channel instead of severity. It feels tidy and it makes urgency impossible to express.
• Dropping the minus sign from a Telegram group chat ID. Group IDs are negative and the field takes the value literally.
• Sending verbose templates to Telegram. The 4096-character limit turns your biggest incident into your quietest one.
• Testing with the Test button and calling it done. Test bypasses the policy tree entirely. It proves the credential works, not that a real alert would ever reach it.
• Leaving resolved notifications on for every channel. Resolution messages in a pager channel double the volume and train people to ignore it.
• Storing webhook URLs and bot tokens in the UI only. They exist in one place, are invisible to review, and disappear with the database.

Best practices for Grafana alert notifications

• Set a consistent severity label on every rule from day one. Retrofitting labels across dozens of rules is miserable work.
• Use one contact point per urgency level, holding as many integrations as that level needs.
• Put a dashboard or runbook link in an annotation your templates render. An alert that says what broke but not where to look costs the responder five minutes every time.
• Write channel-specific templates: short for Telegram, richer for email, in between for Slack.
• Provision contact points and policies from files, with secrets injected from the environment, and run a heartbeat through every channel.
• Re-test every integration after credential rotation, Slack workspace changes or a Grafana upgrade. Those are the three events that quietly break delivery.

FAQ

Can one contact point send to email, Slack and Telegram at the same time?

Yes. A contact point holds multiple integrations and every one of them fires when that contact point is selected. This is the right way to build an escalation tier: one contact point per severity, with the channel mix chosen to match how urgently that severity needs attention.

Why does the Test button work but real alerts never arrive?

Test sends straight to the contact point and skips the policy tree. If it works and real alerts don’t, the break is in routing: matchers that don’t match, a broader policy earlier in the tree absorbing the alert, an active silence, or a mute timing. Open the firing instance, read its real labels, and compare them character by character against your matchers.

Do I need Grafana Cloud for any of this?

No. Contact points, notification policies, templates and all three integrations ship in the open source build. Grafana Cloud handles hosting and adds on-call tooling on top, which matters if you need rotations and escalation chains. For a team routing alerts to three channels, a self-hosted instance on a modest VPS does everything described here.

Should I use a Slack webhook or a bot token?

Webhook for one or two channels where speed of setup matters. Bot token once you have several channels, want one revocable credential, or need to see in one place what the integration is allowed to do. Both deliver identically; the difference is entirely in credential management.

Why did my Telegram alerts stop during a big outage?

Almost certainly the 4096-character limit. A large grouped notification exceeded it, truncation broke the message markup, and Telegram rejected the send outright. Shorten the template for that channel, group more narrowly, or drop the parse mode. It’s the failure mode most worth guarding against, because it only shows up when you need the channel most.

Can I keep using the Prometheus Alertmanager I already run?

Yes. Grafana can send to an external Alertmanager instead of its built-in one, and the contact point selector at the top of the alerting pages lets you choose which one you’re configuring. If you already have a mature Alertmanager routing config, keeping it and pointing Grafana at it is usually less work than migrating the tree.

The one thing to remember

A firing rule proves evaluation works. It proves nothing about whether anyone was told. Grafana alert notifications cross three or four systems after the rule turns red, and every one of them fails quietly by default.

So build the channels carefully, route by severity rather than by tool, then run a heartbeat through every path and let something outside Grafana watch it. The alternative is finding out during the incident, which is the one time you have no attention to spare.

Need alerting that reaches people?

Most of the Grafana setups I get asked to look at have good dashboards and alerting nobody trusts. Work I take on here:

• Fixing email delivery end to end: SMTP config, relay selection, SPF, DKIM and DMARC so alerts stop landing in spam.
• Setting up Slack and Telegram contact points properly, including bot apps, scoped tokens and channel-appropriate templates.
• Designing a policy tree around severity labels, with grouping and timing tuned so people keep reading the channel.
• Converting click-built alerting into provisioned YAML under version control, with secrets kept out of Git.
• Adding heartbeat monitoring so a broken notification path announces itself instead of waiting for an outage.
• Auditing alert rules for missing labels, missing runbook links, and rules that fire constantly and are therefore ignored.

Send me your notification policy export and the output of journalctl -u grafana-server from a period when an alert should have gone out, and I’ll tell you which hop is dropping it.

The post Grafana Alert Notifications That Actually Arrive: Email, Slack and Telegram appeared first on John Nessime.

Then you check the Glue job history and the last successful run was six days ago. The job has not failed. It has not run at all. Because it never ran, it never emitted a metric, so CloudWatch recorded nothing and Grafana drew nothing. The panel is not green because the pipeline is healthy. It is green because the pipeline is silent.

That gap is the most common problem I see with Grafana monitoring for AWS data pipelines, and it is not a Grafana bug. It follows from how AWS batch services emit metrics and how alert engines read an empty result set. This post covers that failure and the three that follow close behind: wrong signal, cost, and access. Then the dashboard layout, the alerting rules, and a troubleshooting list for empty panels.

Failure family one: the pipeline stopped and nothing turned red

Application metrics are continuous. A web server emits request counts every minute forever, so a gap in the line genuinely means something is wrong.

Pipeline metrics are not like that. Glue reports job metrics only while a job executes, and only if job metrics are enabled and the script initialises a GlueContext. Step Functions publishes execution metrics when executions happen. MWAA publishes DAG metrics when DAGs run. Between runs the metric does not exist. A daily job produces a few minutes of data and twenty-three hours of nothing.

So the question that decides whether your monitoring works is: what does your alert engine do with an empty result?

No Data and MissingSeries are not the same thing

• No Data means the query returned no data points at all. Grafana handles this with a per-rule setting and by default raises a separate alert named DatasourceNoData. That alert does not inherit your rule’s labels and it fires immediately, skipping the pending period. If your notification policy routes on a team label, it will not match.
• MissingSeries means the query still returns data but one series has disappeared. That is the dead-job case exactly: a query for failed tasks across all Glue jobs still returns rows for the jobs that ran, and the dead one just stops existing. Grafana holds the last state for a couple of evaluation cycles, then evicts the instance with a transition to Normal, annotated with grafana_state_reason set to MissingSeries.

Read that again, because it is the whole problem. A job that stops running produces an alert that resolves itself and sends a cheerful “back to normal” notification. Nobody investigates a resolution.

The fix: stop inferring health from absence

Do not build the “did it run” signal out of metrics that only exist when it runs. Publish a heartbeat: a custom metric your pipeline writes once, deliberately, at the point you consider the run complete.

aws cloudwatch put-metric-data 
  --namespace "Pipelines" 
  --metric-name "RunCompleted" 
  --dimensions PipelineName=orders_daily 
  --value 1 
  --unit Count

Put that at the end of the Glue script, in the final Step Functions state, or in the last task of the DAG. Now “did it run” has a positive signal instead of an inferred one.

The heartbeat is still sparse, so you also need to tell the evaluator that absence means failure. In a CloudWatch alarm, that is the missing-data setting: the options are missing, ignore, breaching and notBreaching, and for a heartbeat you want breaching.

aws cloudwatch put-metric-alarm 
  --alarm-name "orders_daily-heartbeat-missing" 
  --namespace "Pipelines" 
  --metric-name "RunCompleted" 
  --dimensions Name=PipelineName,Value=orders_daily 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 26 
  --threshold 1 
  --comparison-operator LessThanThreshold 
  --treat-missing-data breaching 
  --alarm-actions arn:aws:sns:REGION:ACCOUNT_ID:pipeline-alerts

One subtlety here bites people. CloudWatch looks back over an evaluation range wider than your evaluation periods, and if it finds enough real data points in that wider range, the missing-data setting is ignored entirely. For a once-a-day metric that behaviour can suppress the alarm you just built. The reliable way around it is metric math: fill the gaps before the alarm evaluates anything.

FILL(m1, 0)

Now every period has a value, zero means “no run completed”, and a plain threshold behaves the way you expected. FILL(m1, REPEAT) and FILL(m1, LINEAR) carry the last known value or interpolate instead, which suits gauges better than heartbeats.

On the Grafana side, set the no-data state explicitly on each rule and build a notification policy that catches alertname=DatasourceNoData, otherwise those alerts go nowhere. The heartbeat still matters even with that in place, because it turns a disappearing series into a present series with a bad value, and every rule engine handles that correctly.

Failure family two: you are monitoring the compute, not the data

The second failure is subtler. The job runs, succeeds, takes its usual time, and writes nothing, because an upstream API returned an empty page or a partition filter matched no files. Executor CPU and heap will not tell you that. Two signals will.

Freshness is time since the last successful completion, per pipeline. It maps directly onto “can I trust the report on my screen”, which is the only question your stakeholders are asking. Derive it from the heartbeat.

Volume is rows or bytes written per run, compared against the same run last week. A drop to zero with a successful exit code is the classic empty-load bug. For Spark jobs on Glue the built-in counters give you a usable start:

• glue.driver.aggregate.recordsRead and glue.driver.aggregate.bytesRead for input volume
• glue.driver.aggregate.numFailedTasks for partial failures that do not fail the job
• glue.driver.s3.filesystem.write_bytes and the glue.ALL equivalents for output across executors

Glue reports these roughly every thirty seconds as deltas from the previous report, so Sum over the run window is usually what you want rather than Average. Beyond those, the signals I put on a board first are ExecutionsStarted minus ExecutionsSucceeded from the AWS/States namespace, since aborted and timed-out executions land in separate counters and failure count alone misses them; DAG metrics from the AmazonMWAA namespace, remembering that metrics.metrics_allow_list and metrics.metrics_block_list control what reaches CloudWatch at all; consumer lag rather than throughput on streaming ingestion, because a stream will happily accept records nobody is reading; and message age on dead letter queues, which is a near-perfect proxy for “something is failing repeatedly and nobody noticed”.

Failure family three: the dashboard costs more than the pipeline

Grafana’s CloudWatch data source uses GetMetricData to fetch samples and ListMetrics when you pick dimensions in the editor. GetMetricData does not qualify for the CloudWatch API free tier, and it is billed per metric requested rather than per API call. One request pulling five hundred metrics is five hundred billable metric requests.

Now multiply: panels, times series per panel, times statistics per series, times refresh rate, times every user with the board open, times every alert rule on its own schedule. A wildcard query matching every Glue job in the account looks tidy in the editor and quietly becomes the largest line item in your monitoring bill. Asking for more than five statistics on one metric counts as an extra metric request too.

The levers, roughly in order of return:

1. Set a sane refresh interval. Metrics that arrive every thirty seconds do not need a five-second refresh, and a daily batch board does not need auto-refresh at all.
2. Replace wildcard dimension matching with explicit resource lists on always-open boards. Keep wildcards for ad hoc investigation.
3. Query only the regions you run in. It is easy to leave a data source pointed at a region you migrated out of.
4. Use template variables instead of one duplicated board per team.
5. If you pull a large metric catalogue continuously, evaluate CloudWatch metric streams. Streams push through Amazon Data Firehose to a destination you control, and AWS recommends that path for third-party tools with heavy GetMetricData usage. The trade-off is genuine: you lose metric math and Metrics Insights against CloudWatch directly, and you pick up storage and Firehose ingestion costs instead.

To find which client is driving the spend, enable CloudTrail data events on GetMetricData. It surfaces the calling IAM principal and source address, which is usually enough to identify the forgotten dashboard.

Failure family four: access, accounts and regions

Grafana ships CloudWatch support natively, so there is no plugin to install. What there is, is an IAM policy people habitually make too broad or too narrow. This is the read-only set the data source needs:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReadingMetricsFromCloudWatch",
      "Effect": "Allow",
      "Action": [
        "cloudwatch:DescribeAlarmsForMetric",
        "cloudwatch:DescribeAlarmHistory",
        "cloudwatch:DescribeAlarms",
        "cloudwatch:ListMetrics",
        "cloudwatch:GetMetricData",
        "cloudwatch:GetMetricStatistics"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingRegions",
      "Effect": "Allow",
      "Action": "ec2:DescribeRegions",
      "Resource": "*"
    },
    {
      "Sid": "AllowReadingResourcesForTags",
      "Effect": "Allow",
      "Action": "tag:GetResources",
      "Resource": "*"
    }
  ]
}

Prefer an assumed role over long-lived access keys. If Grafana runs on EC2 or ECS in the same account, the instance or task role is cleanest and there is no secret to rotate.

Two boundaries to plan for. CloudWatch quotas are defined per account and per region, so a multi-account estate can hit limits in one account while the rest are fine, and each needs its own quota increase. And metrics do not cross regions: a pipeline that reads in one region and writes in another needs both on the board, and the missing half is a common reason a panel looks empty for no obvious cause.

Where to run Grafana for this

Three options, and the choice is mostly about viewer count and who carries the operational load.

• Amazon Managed Grafana bills per active user license per workspace, split into editor and viewer tiers, with at least one editor license required per workspace even if nobody logs in. An active user is anyone who logged in or made an API request during the billing cycle. Excellent for a small engineering team, expensive if you hand read-only boards to a large business audience.
• Grafana Cloud from Grafana Labs bills on usage rather than seats, which flips that calculation. It also brings the AWS observability integrations and Grafana Alloy if you want CloudWatch metrics in a Prometheus-compatible store.
• Self-hosted is cheapest per seat and most work. A small VPS from a provider like Contabo or InterServer runs Grafana for a handful of pipelines comfortably, and you own upgrades, TLS renewal and backups. One licensing note: Grafana’s open source core moved from Apache 2.0 to AGPLv3, while plugins and agents stayed Apache-licensed. If your legal team has a policy on AGPL, have that conversation before deployment.

Laying out a board people actually read

Organise by question, not by service. Whoever opens this board on a Monday morning has one question, and the answer should be visible without scrolling.

1. Freshness table. One row per pipeline, hours since last successful completion, threshold colour tied to that pipeline’s actual SLA. Most viewers read only this row.
2. Volume against baseline. Records or bytes written per run, overlaid with the same period a week ago.
3. Failure counts and durations, grouped by stage so a slow extract is distinguishable from a slow load.
4. Resource detail at the bottom. Executor memory, DPU usage, throttling. Useful during an incident, noise the rest of the time.

Use a template variable for pipeline name rather than one board per pipeline. It keeps the query count down and new pipelines appear automatically. For cross-pipeline aggregation without fanning out into individual series, CloudWatch Metrics Insights runs a SQL-style query inside CloudWatch and the Grafana data source supports it:

SELECT SUM(ExecutionsFailed)
FROM SCHEMA("AWS/States", StateMachineArn)
GROUP BY StateMachineArn

Each GetMetricData operation can carry only one Metrics Insights query, so treat it as a way to consolidate one fan-out, not to run several at once.

Troubleshooting: why the panel is empty

1. Check the region on the query, not the data source. Grafana allows a per-query region override and it is easy to leave one on default.
2. Widen the time range to seven days. If data appears, the metric is sparse and you have a heartbeat problem, not a query problem.
3. Check the statistic. Sum and Average behave very differently on delta counters, and empty-looking panels are often Average over a window that is mostly gaps.
4. Check dimensions match exactly. With Match Exact on, every dimension must be specified, so a metric published with two dimensions returns nothing when queried with one. Turning Match Exact off makes Grafana generate a search expression instead, which is the usual fix.
5. Check job metrics are enabled. For Glue you need both the job metrics option and a GlueContext in the script. Without the GlueContext, nothing is emitted regardless of the console setting.
6. Check IAM. A role with ListMetrics but not GetMetricData populates every dropdown perfectly and returns no data, which is confusing until you have seen it once.
7. Check for throttling. Panels that are intermittently empty under load rather than consistently empty usually mean a CloudWatch API quota for that account and region.

For alert rules behaving oddly rather than panels, treat the Error state separately from No Data. Grafana raises DatasourceError on evaluation timeouts or repeated query failures, governed by the evaluation_timeout and max_attempts settings. Slow CloudWatch queries under quota pressure trip this, and the resulting alert looks nothing like the one you wrote.

Common mistakes

• Alerting on failure counts only. A job that never starts cannot fail, so failure-count alerting is blind to the most common outage.
• Leaving no-data handling on the default and never routing DatasourceNoData. The alert fires into the void.
• Treating a MissingSeries resolution as good news. If alerts resolve without anyone acting, check the reason annotation.
• Importing a large community dashboard pack wholesale. Most of those panels use wildcards, and you have just subscribed to a permanent GetMetricData bill for metrics nobody reads.
• Monitoring the orchestrator but not the destination. Step Functions reports a clean success while the load that ran inside it wrote zero rows.

Best practices worth the effort

• Emit a heartbeat from every scheduled pipeline using one consistent namespace and dimension scheme. Consistency is what lets a single dashboard cover everything.
• Write down a freshness SLA per pipeline, then set the threshold from it. “Alert if stale” only means something once someone says how stale is too stale.
• Provision dashboards and alert rules as code in the same repository as the pipeline. A pipeline that ships without its monitoring is not finished.
• Route by severity, not by service. Freshness breaches page; executor memory warnings go to a channel.
• Break something on purpose to test the alert. Disable the schedule on a non-production pipeline and confirm you get paged. An untested alert is a guess.

Frequently asked questions

Do I need a plugin to connect Grafana to CloudWatch?

No. Grafana includes native support for the Amazon CloudWatch data source. Configure credentials or an assumed role, set a default region, and query. The same data source also reaches CloudWatch Logs and X-Ray.

Why does my Grafana alert resolve itself when the pipeline is still broken?

Almost certainly MissingSeries eviction. When one series in a multi-series query disappears, Grafana holds the last state briefly and then transitions the instance to Normal with grafana_state_reason set to MissingSeries. It is not treated as No Data. A heartbeat metric with gaps filled to zero stops the series disappearing in the first place.

CloudWatch dashboards or Grafana for AWS data pipelines?

CloudWatch dashboards are the shorter path if everything sits in one account and the visualisation options suit you. Grafana earns its place when you need pipeline metrics correlated with a warehouse query, a database, a trace or a business KPI on one screen, and when alerting has to route to several destinations. Most data platforms end up wanting the correlation.

How do I monitor a Glue job that succeeds but writes no data?

Alert on output volume rather than exit status. Use the Glue write byte counters, or better, emit your own row-count metric after the write so it reflects what your code believes it wrote. Compare against the equivalent run a week earlier, because fixed thresholds age badly as data grows.

Does Grafana querying CloudWatch cost money?

Yes. Grafana uses GetMetricData, which is billed per metric requested and excluded from the CloudWatch API free tier. Cost scales with panels multiplied by series multiplied by refresh frequency multiplied by viewers, so wildcards on an auto-refreshing board are the expensive pattern. Check current rates on the CloudWatch pricing page and do the arithmetic for your own board.

Amazon Managed Grafana or self-hosted?

It comes down to viewer count. Managed Grafana bills per active user per workspace, which suits a small engineering team and gets costly when read-only dashboards go out widely. Self-hosting removes the per-seat cost and adds upgrade, TLS and backup work. Grafana Cloud sits between them on usage-based billing. Count likely viewers first, then choose.

What is the minimum useful monitoring for a new pipeline?

A heartbeat proving the run completed, a row or byte count proving it moved something, and an alert that fires when the heartbeat is missing for longer than the agreed freshness window. Everything else is refinement.

The one thing to take away

Effective Grafana monitoring for AWS data pipelines is not about drawing more metrics. It is about refusing to infer health from absence. Batch pipelines are silent most of the time, and every monitoring system in existence reads silence as calm unless you tell it otherwise.

Publish a heartbeat. Fill the gaps with zero. Set your no-data behaviour deliberately. Then alert on freshness and volume, because those are the two things people downstream will notice before you do. Do that, and a green panel means something.

Need help building this on your AWS estate?

I design and build monitoring for AWS data platforms. This is the kind of work I take on:

• Heartbeat and freshness instrumentation across Glue, Step Functions, MWAA and Lambda pipelines, with one consistent namespace scheme so a single dashboard covers everything
• Grafana dashboards built around freshness and volume rather than executor charts nobody reads
• Alert rules and notification policies that handle No Data and MissingSeries correctly, routed to Slack, Teams, Telegram or email by severity
• CloudWatch query cost reviews, tracing GetMetricData spend back to the dashboard or rule causing it
• Cross-account and multi-region data source setup with least-privilege IAM roles instead of long-lived keys
• Grafana deployment and hardening on Amazon Managed Grafana, Grafana Cloud or a self-hosted instance behind TLS

If you have a pipeline that failed quietly, send me the job definition, the alert rule, or a screenshot of the panel that stayed green. I would rather look at the actual thing than talk in generalities.

The post Grafana Monitoring for AWS Data Pipelines: The Green Dashboard Problem appeared first on John Nessime.