That gap is the entire problem. Run status tells you the code executed. It tells you nothing about whether data arrived. Data pipeline freshness monitoring is the signal that closes the gap, and most teams add it only after an analyst finds the stale table first.

This post covers the three signals worth instrumenting for a batch or streaming pipeline: how current the data is, how long it took to get there, and how often the thing breaks. For each one I’ll cover what to measure, where to measure it, the queries and config that produce it, and the specific ways the measurement lies to you.

The signal that lies to you first: run status

Job success is a statement about your code path, not about your data. A pipeline that reads an empty S3 prefix, transforms zero rows, writes zero rows and exits cleanly has succeeded. So has one whose upstream API silently started returning an empty page after a token rotation. So has one whose incremental watermark got stuck and now re-reads the same already-loaded slice on every run.

All three are green. All three are producing stale data. This is why the ordering matters: freshness is the signal that fires first in a real incident, and run status is usually the last one to notice anything at all.

So keep run status. It’s cheap and it catches crashes. Just stop treating it as your top-line health indicator.

Freshness: measure the table, not the job

Freshness is the age of the newest record in a dataset. It’s measured against the dataset itself, not against the process that filled it, and that distinction is what makes it useful. If your job disappears entirely, freshness keeps climbing and keeps alerting. If your job succeeds while doing nothing, freshness keeps climbing and keeps alerting. It’s the one signal that survives both failure modes.

Two clocks, and you need both

Every row usually carries two timestamps: when the event happened in the source system, and when your pipeline wrote it. Track the maximum of each.

• Max event time tells you whether the upstream system is still producing. If this stops moving, the problem is upstream of you.
• Max load time tells you whether your pipeline is still writing. If this stops moving while event time is fine, the problem is yours.

Watching only one of them means every stale-data page starts with twenty minutes of figuring out which side of the boundary the fault sits on. Watching both answers that in the alert body.

A freshness probe is a small scheduled query. This one is PostgreSQL syntax, and it returns both clocks plus a volume check in a single round trip:

select
  'orders' as dataset,
  extract(epoch from max(event_ts))  as max_event_ts,
  extract(epoch from max(loaded_at)) as max_loaded_at,
  count(*) filter (where loaded_at >= now() - interval '1 hour') as rows_last_hour
from analytics.orders;

The row count matters. A pipeline can advance its load timestamp while writing almost nothing, which is what a partially broken source looks like. Freshness alone will not catch that; freshness plus volume will.

Run that probe on a schedule that’s independent of the pipeline, push the results as gauges, and alert on age in PromQL:

# Newest row is older than 90 minutes
time() - max by (dataset) (dataset_max_loaded_timestamp_seconds) > 5400

# The probe itself has stopped reporting: a deadman check
absent(dataset_max_loaded_timestamp_seconds{dataset="orders"})

That second rule is the one people forget. A freshness metric that disappears looks identical to a healthy silence on a graph. absent() is what turns “no data” into a page.

Where dbt fits

If you’re already running dbt, source freshness is built in and worth using before you write anything custom. You declare thresholds per source, and dbt queries the maximum of your timestamp column and compares it to now.

sources:
  - name: raw_shop
    schema: raw
    config:
      loaded_at_field: _loaded_at
      freshness:
        warn_after: {count: 2, period: hour}
        error_after: {count: 6, period: hour}
    tables:
      - name: orders
      - name: refunds
        config:
          freshness:
            warn_after: {count: 12, period: hour}
            error_after: {count: 24, period: hour}

Three details that trip people up. First, dbt source freshness is a separate command; dbt build does not run it, so a green build says nothing about source staleness. Second, it exits non-zero when a source hits its error_after threshold, which makes it a natural gate at the top of a job: fail fast rather than building models on stale input. Third, results land in target/sources.json, which is the artifact you parse if you want to distinguish warn from error, or ship the numbers into Prometheus rather than just failing the run.

dbt source freshness --select source:raw_shop --output target/freshness.json

One caveat on the YAML above: dbt moved these keys under a config: block in recent releases, and loaded_at_field followed later. Older projects nest them directly under the source. Check what your project’s version expects before copying, because a misplaced key fails quietly by simply not calculating freshness at all.

Latency: name the clock before you name the number

“Our pipeline latency is twelve minutes” is meaningless until you say which two points you measured between. There are at least three plausible definitions, and teams routinely argue past each other because they’re each using a different one.

• Run duration. Start to end of the job. Easy, and mostly useless for anyone downstream.
• Ingestion latency. Source extract to target load. This is what you control.
• End-to-end latency. Event time to the moment the row is queryable. This is what the business actually feels, and it includes queue wait, scheduler delay, and every upstream hop you don’t own.

Publish end-to-end as the headline and keep the others as breakdown. If you only track run duration, a scheduler backlog that delays every run by forty minutes is completely invisible to you: each individual run still takes eleven minutes.

Instrument per stage with a histogram so you can ask percentile questions later without re-instrumenting:

histogram_quantile(
  0.95,
  sum by (le, stage) (rate(pipeline_stage_duration_seconds_bucket[6h]))
)

Use the median for capacity planning and p95 or p99 for the SLO. Averages hide the exact tail that generates the complaints, and on a pipeline that runs a few dozen times a day, a single pathological run is a real fraction of your day.

For streaming, the equivalent is consumer lag expressed in time rather than offsets. Offset lag of fifty thousand messages means nothing without a rate; two hundred seconds of lag means something to everyone. Kafka Lag Exporter popularised this by interpolating a time estimate from observed offset and timestamp samples, exposing kafka_consumergroup_group_max_lag_seconds alongside the offset-based kafka_consumergroup_group_lag. Worth knowing: that project’s repository has been archived and is read-only, so if you’re starting fresh, check whether your broker vendor or a maintained fork covers it before you deploy something unmaintained into the critical path.

Failure rate: decide what counts as a failure

Failure rate is trivially easy to compute and surprisingly easy to compute wrongly. The denominator and the definition both need a decision.

Count runs, not tasks. A DAG with sixty tasks where one flaps on a transient network error looks catastrophic at task level and fine at run level. The run is the unit the consumer cares about.

Count a run that succeeded on its third retry as a success for availability and a failure for a separate reliability metric. Both are true and they answer different questions. If your only metric folds retries into success, you will never see the slow degradation of an upstream API until it stops responding entirely.

sum by (pipeline) (rate(pipeline_runs_total{result="failure"}[6h]))
/
sum by (pipeline) (rate(pipeline_runs_total[6h]))

And add a category label for the failure reason at push time: source unavailable, schema mismatch, permission denied, timeout, validation failed. The rate tells you something is wrong. The category tells you who to wake up. Without it, every failure alert costs you a log dive before you can even route the incident.

Getting metrics out of a job that exits

Prometheus scrapes. Batch jobs finish and vanish. Pushgateway bridges that: the job pushes before exiting, and Pushgateway holds the values for Prometheus to scrape on its own schedule.

#!/usr/bin/env bash
set -euo pipefail

JOB="orders_load"
PGW="http://pushgateway.internal:9091"

start=$(date +%s)
if python /opt/pipelines/load_orders.py; then
  result=0
else
  result=1
fi
end=$(date +%s)

{
  cat <<EOF
# TYPE pipeline_run_duration_seconds gauge
pipeline_run_duration_seconds $((end - start))
# TYPE pipeline_last_run_timestamp_seconds gauge
pipeline_last_run_timestamp_seconds $end
# TYPE pipeline_last_run_success gauge
pipeline_last_run_success $((1 - result))
EOF
  if [ "$result" -eq 0 ]; then
    cat <<EOF
# TYPE pipeline_last_success_timestamp_seconds gauge
pipeline_last_success_timestamp_seconds $end
EOF
  fi
} | curl --fail --data-binary @- "$PGW/metrics/job/$JOB"

exit $result

The conditional block is doing real work. curl’s --data-binary issues a POST, and a POST to Pushgateway replaces only the metrics whose names appear in the payload, leaving the rest of the group intact. So a failing run updates the run timestamp and the success flag while leaving the previous pipeline_last_success_timestamp_seconds exactly where it was. That’s what lets time() - pipeline_last_success_timestamp_seconds keep climbing across consecutive failures. Send a PUT instead and you replace the whole group, wiping the value you needed.

On the Prometheus side, one setting is not optional:

scrape_configs:
  - job_name: pushgateway
    honor_labels: true
    static_configs:
      - targets: ['pushgateway.internal:9091']

Without honor_labels: true, Prometheus overwrites the job label your pipeline pushed with the scrape job’s own name, and every pipeline in your estate collapses into one indistinguishable series called pushgateway.

The trap worth internalising: Pushgateway never forgets. Metrics persist until something explicitly deletes them or the process restarts. A pipeline you decommissioned last quarter is still cheerfully reporting a success timestamp and a duration, and it looks alive on every dashboard. Treat Pushgateway as a cache of past executions, not a picture of current state, and use the push_time_seconds gauge that Pushgateway attaches to each group to tell the difference between a fresh push and a fossil.

If you’re on Airflow, the SLA feature is gone

This one will bite anyone upgrading. The sla and sla_miss_callback parameters were removed in Airflow 3.0, and the replacement, Deadline Alerts, arrived in 3.1. DAGs carrying the old configuration need manual migration; they don’t quietly keep working.

from datetime import timedelta

from airflow.sdk import AsyncCallback, DAG, DeadlineAlert, DeadlineReference
from airflow.providers.slack.notifications.slack_webhook import SlackWebhookNotifier

with DAG(
    dag_id="orders_load",
    deadline=DeadlineAlert(
        reference=DeadlineReference.DAGRUN_QUEUED_AT,
        interval=timedelta(minutes=45),
        callback=AsyncCallback(
            SlackWebhookNotifier,
            kwargs={"text": "orders_load has not finished 45 minutes after queuing."},
        ),
    ),
):
    ...

Note the reference point. Measuring from when the run was queued, rather than from its logical date, means scheduler backlog counts against the deadline. That’s usually what you want, because a run that sat in a queue for an hour is late to its consumers regardless of how fast it executed once it started.

There’s a structural weakness here worth naming: any alert that lives inside the orchestrator dies with the orchestrator. If the scheduler is down, nothing evaluates your deadline and nothing notifies anyone. The freshness probe from earlier is the answer, and it needs to run somewhere else. A small VPS from a provider like Contabo or InterServer running Prometheus and Alertmanager, or a hosted option like Grafana Cloud, gives you a watcher outside the blast radius of the thing being watched. This is the single highest-value piece of monitoring most data teams are missing.

Alerting without burning your on-call

The fastest way to make all of this worthless is to alert on every threshold crossing. Three rules keep it survivable.

1. Set thresholds from observed behaviour, not from wishes. Look at a month of actual freshness values and set warn_after a comfortable margin above the normal worst case. A source that habitually passes at eleven hours against a twelve-hour threshold is not healthy, it’s one upstream hiccup from paging you.
2. Page on consumer impact, not internal events. A task retry is not an incident. A dataset breaching the freshness commitment its consumers rely on is. Route everything else to a channel someone reads in the morning.
3. Use burn rate over multiple windows for SLOs. A short window catches fast breakage, a long window catches slow erosion, and requiring both to fire filters out the transient spikes that generate most false pages.

One more: alert on the absence of your own telemetry. A freshness gauge that stops updating is indistinguishable from one that’s fine, right up until someone asks about the numbers.

Troubleshooting

Freshness alert fires but the data looks current. Almost always a timezone problem. Your loaded_at column is in local time, now() is in UTC, and the offset shows up as a constant bias in the age. Store load timestamps in UTC and cast explicitly at read time.

Freshness flaps in and out of breach. Your check runs too close to the expected arrival. If loads land around six and your probe runs at five past, a fifteen-minute upstream delay produces an intermittent failure that trains everyone to ignore the alert. Move the probe later or widen the threshold.

Every pipeline reports as one series. honor_labels: true is missing from the Pushgateway scrape config.

A decommissioned pipeline still shows healthy. Stale group in Pushgateway. Delete the group and add a check on push_time_seconds so the next one surfaces on its own.

Latency looks fine but consumers say data is late. You’re measuring run duration and they’re feeling end-to-end. Add queue wait and event-time-to-load and the gap will be obvious.

Freshness passes, row count is near zero. The load timestamp advanced without meaningful data. This is a broken source or a stuck watermark, and it’s the reason the volume check belongs in the same probe.

Common mistakes

• Treating job success as the health signal and discovering staleness through a human.
• Running the freshness check inside the same pipeline it’s meant to police.
• Tracking freshness without volume, so a zero-row load reads as healthy.
• Reporting latency without saying which two timestamps it spans.
• Putting run IDs, batch IDs or timestamps into Prometheus labels, which multiplies your series count without bound. Those belong in traces or logs.
• Alerting on task-level failures instead of run-level outcomes, then muting the whole channel a week later.
• Assuming Pushgateway reflects current state rather than the last thing anyone pushed.

Best practices for data pipeline freshness monitoring

• Start with your three most-used tables. Full coverage is a project; three tables is an afternoon and catches most of the pain.
• Write down the freshness commitment per dataset in plain language, then encode it. “Yesterday’s orders are complete by 07:00” converts directly into a threshold.
• Name metrics consistently across pipelines. A shared prefix and a stable label set is what makes one dashboard work for all of them.
• Keep labels low-cardinality: pipeline, dataset, stage, environment, result. Nothing unbounded.
• Emit a failure category alongside every failure so alerts route themselves.
• Run the deadman check on infrastructure that doesn’t share a failure domain with the pipeline.
• Link every alert to a runbook that names the owner and the first three things to check.

Frequently asked questions

What is the difference between pipeline latency and data freshness?

Latency measures how long a specific batch of data took to travel from source to destination. Freshness measures how old the newest available record is right now, regardless of whether anything is currently running. A pipeline can have excellent latency and terrible freshness if it stopped being triggered.

How often should freshness checks run?

Frequently enough that you find out before your consumers do. A useful rule is roughly a quarter of your tolerance window: if data may be up to four hours old, check hourly. Freshness probes are cheap single-aggregate queries, so the limiting factor is usually warehouse billing rather than load.

Do I need a data observability platform for this?

Not to start. Freshness, latency and failure rate for a handful of critical datasets is a scheduled query, a push, and a few alert rules. Commercial platforms earn their cost when you need automatic column-level lineage, anomaly detection across hundreds of tables, or coverage of assets nobody has explicitly instrumented. That’s a real problem at scale, and a genuinely expensive one to build yourself. It’s just not the problem you have on day one.

How do I monitor freshness for a table that only updates weekly?

Set the threshold from the schedule plus a delivery margin, and add a deadman rule so a missing metric alerts on its own. For genuinely static reference tables, disable freshness explicitly rather than leaving a check that always warns; in dbt that means setting freshness to null for the table.

Should freshness checks fail the pipeline or just warn?

Both, in different places. Checking source freshness at the start of a job and failing hard prevents you building models on stale input, which is the cheapest bug to prevent and the most expensive to unwind. Checking output freshness after the fact should alert rather than fail, because the run is already over.

What percentile should I use for a latency SLO?

p95 for most internal analytics pipelines, p99 where downstream systems make automated decisions on the data. Track the median separately for capacity planning. Never use the average as the SLO number, because it hides exactly the tail that produces complaints.

Does this work for streaming pipelines too?

Yes, with different plumbing. Freshness becomes consumer lag measured in time, latency becomes event-time to availability-time, and failure rate becomes connector and task state plus dead-letter volume. The reasoning is identical; only the source of the numbers changes.

Conclusion

If you remember one thing, make it this: measure the data, not the job. Run status, task counts and duration all describe your code. Only freshness describes what your consumers actually receive, and it’s the one signal that stays honest when the pipeline succeeds at doing nothing.

Good data pipeline freshness monitoring is not a platform purchase. It’s a scheduled query that reports the age and volume of your most important tables, pushed somewhere durable, with a deadman rule so that silence is treated as a failure rather than as health. Add latency broken down by stage and failure rate categorised by reason, and you can answer “is the data good right now” without opening a single log file.

Need help instrumenting your pipelines?

Most of my consulting work in this area is retrofitting observability onto pipelines that already exist and can’t be paused. Specifically:

• Defining freshness and latency commitments per dataset, then translating them into thresholds and alert rules that hold up on-call.
• Adding freshness and volume probes to existing warehouses without touching the pipelines themselves.
• Wiring batch jobs into Prometheus through Pushgateway, including the grouping-key and stale-metric problems that bite six months later.
• Migrating Airflow DAGs off the removed SLA feature onto Deadline Alerts, or onto external checks that survive a scheduler outage.
• Building the Grafana dashboard that answers “is the data good right now” in one screen, with drill-down by stage and failure category.
• Cutting alert noise on pipelines where the channel has already been muted, by moving from task-level events to consumer-impact SLOs.

If you want a concrete starting point, send me a DAG file, a scrape config, or a screenshot of the dashboard you don’t trust, and I’ll tell you what I’d instrument first and why.

The post Green Pipeline, Stale Data: Tracking Latency, Freshness and Failure Rates appeared first on John Nessime.

That gap is the entire problem. Monitoring n8n workflow failures in Grafana is not hard because the tooling is missing. It’s hard because the built-in metrics endpoint answers a question you didn’t ask. It tells you the process is alive and busy. It does not tell you that a workflow stopped producing correct output.

This post covers what n8n’s Prometheus endpoint actually exposes, the three families of failure it handles badly, and the specific PromQL and alert rules that close each gap. There’s also a troubleshooting section for the case where /metrics returns nothing at all, which is the first wall most people hit.

What the n8n metrics endpoint actually gives you

n8n uses the prom-client library and serves metrics on the same port as the editor, at /metrics. The endpoint is off by default, and almost everything interesting inside it is off by default too, behind its own environment variable.

N8N_METRICS=true
N8N_METRICS_INCLUDE_MESSAGE_EVENT_BUS_METRICS=true
N8N_METRICS_INCLUDE_WORKFLOW_ID_LABEL=true
N8N_METRICS_INCLUDE_NODE_TYPE_LABEL=true
N8N_METRICS_INCLUDE_QUEUE_METRICS=true

The first line opens the endpoint. The second turns on the workflow and node event counters, which is the part people assume is included and is not. The third attaches a workflow_id label to workflow metrics, without which you get a single instance-wide number and no way to tell which automation broke.

Those labels are opt-in for a reason worth understanding rather than working around. Every distinct label value creates a separate time series. Turn on workflow_id and you get one series per workflow per counter; add node_type and node counters fan out across every integration you use. On a few dozen workflows that’s fine. On several hundred with per-node labels, you’re signing up for a cardinality problem that surfaces later as slow queries and a Prometheus process that keeps getting OOM-killed.

With the event bus metrics enabled, the counters you get include n8n_workflow_started_total, n8n_workflow_success_total, n8n_node_started_total and n8n_node_finished_total. Queue mode adds n8n_scaling_mode_queue_jobs_active, _completed, _failed and _waiting, gathered from Bull and exposed on the main instance only.

Prometheus side, the scrape job is unremarkable:

scrape_configs:
  - job_name: n8n
    static_configs:
      - targets:
          - n8n-main:5678
          - n8n-worker-1:5678
    metrics_path: /metrics

Note both main and workers. Each process exposes its own endpoint and its own counters. Scrape only the main and you lose everything the workers did, which in queue mode is nearly all the real execution work.

One thing the n8n docs are emphatic about, and they’re right: don’t put /metrics on the public internet. It leaks workflow IDs, execution volumes and instance topology to anyone who asks. Keep the scrape path on a private network segment, a WireGuard tunnel, or your provider’s internal networking. If you’re running the whole stack on a single VPS from somewhere like Contabo or InterServer, binding n8n to the Docker network and scraping it by container name is enough. Publishing port 5678 to the host and firewalling it afterwards is not, because you will forget.

Failure family one: the execution that never happened

This is the one that costs the most and shows up the least. A schedule trigger gets deactivated during a deploy. A credential expires and the workflow is switched off rather than fixed. An upstream service stops calling a webhook. Nothing errors. Nothing increments. The workflow simply stops existing as far as your metrics are concerned.

Every error-rate alert you write is blind to this, because an error rate needs a denominator. Zero executions divided by zero executions is not an alert, it’s a NaN.

The instinct is to write this:

sum by (workflow_id) (increase(n8n_workflow_started_total[6h])) == 0

It looks correct and it does not work. When a workflow stops running, its series stops being exported. Prometheus marks it stale after a few missed scrapes and instant queries stop returning it. There is no series left for == 0 to match against. The alert never fires, because the thing you wanted to alert on is exactly the thing that removed the data.

What you want is a function that returns something when a series is absent:

absent_over_time(
  n8n_workflow_started_total{workflow_id="AbC123XyZ"}[6h]
)

absent_over_time returns 1 if the matched series produced no samples in the window and nothing at all if it did. That inversion is the point. The alert fires on silence.

The cost is that absent_over_time needs an exact matcher, so you can’t write it once for everything. That’s more acceptable than it sounds. You don’t need a heartbeat on all two hundred workflows, you need one on the six a human would notice within an hour. Write those six by hand and treat the list as part of the runbook.

One Grafana-specific trap. When a query returns nothing, Grafana’s alert rule enters a No Data state, which is configurable separately from the alert condition and defaults to firing in most versions. Set it deliberately. For an absent_over_time rule, No Data means the whole scrape target is gone, which is a different incident from one workflow going quiet, and you probably want it routed differently.

Failure family two: the failure you can see but can’t attribute

You have the counters. Something is failing. Which workflow?

Without N8N_METRICS_INCLUDE_WORKFLOW_ID_LABEL, the answer is: no idea. The counters are instance-wide totals. You know the number moved and that’s it. That’s a fine signal for capacity planning and useless during an incident.

Turn the label on and you get IDs, which are opaque strings like AbC123XyZ. A dashboard full of those is technically attributable and practically unreadable at 3am. Newer n8n versions solve this with a gauge that maps IDs to names, enabled with N8N_METRICS_INCLUDE_WORKFLOW_INFO:

n8n_workflow_info{workflow_id="VaQPuPmx9tPpo6BX",workflow_name="Invoice sync"} 1

It’s a constant-1 gauge that exists only to be joined against. The pattern is a standard Prometheus vector match:

sum by (workflow_name) (
  rate(n8n_workflow_started_total[5m])
  * on(workflow_id) group_left(workflow_name)
  n8n_workflow_info
)

The multiplication is arithmetically a no-op because the gauge is always 1. Its only job is to carry the workflow_name label across with group_left. Do this once in a Grafana variable or a recording rule rather than in every panel, or you’ll be copy-pasting the join for the rest of your life.

On an older build without the info gauge, the fallback is a hand-maintained relabel config or a Grafana value mapping. Both drift, and the relabel version drifts silently.

Failure family three: the failure that counts as a success

This is the invisible one, and it’s why a green n8n dashboard is not evidence of anything.

n8n gives you several deliberate ways to make a node’s failure not be an execution failure. They’re all reasonable features. Together they mean the success counter and “the automation worked” are different statements.

• Continue on error. A node set to continue passes its error downstream instead of stopping the run. The execution finishes. It finishes as a success.
• Error output branches. A node with a separate error output routes failures to a different path. That path might log and stop. The execution still completes normally.
• Never Error on HTTP responses. The HTTP Request node can be configured to treat any status code as a valid response. A run of 500s from an upstream API becomes a workflow that processed zero useful records, successfully.
• Empty result sets. A query that returns nothing is not an error. A sync that syncs nothing, every night, looks exactly like a sync that has nothing to do.

There are also user reports on the n8n community forum of started and success counters matching even when a node failed outright, in queue mode. I have not reproduced that and I’d treat it as unconfirmed rather than a known behavior. But it’s the right instinct: don’t assume the success counter is a strict complement of failure. Verify it on your own instance before you build alerting on the assumption.

For webhook-triggered workflows, newer n8n versions give you a real answer here. The n8n_webhook_request_duration_seconds histogram carries a status_code label, so you can alert on what the caller actually received rather than on what n8n thought about the run:

sum by (workflow_id) (
  rate(n8n_webhook_request_duration_seconds_count{status_code!~"2.."}[5m])
)

Worth flagging: the n8n documentation shows this example with status_code!="2..", using the equality operator. In PromQL, != compares against a literal string, so that matcher excludes only a status code whose literal value is the three characters 2.., which never occurs. It matches everything, including your 200s. Use !~ for the regex form. Test any label matcher you copy from documentation by running it in Prometheus and checking the series count against what you expect.

Getting a second source of truth

For everything that isn’t a webhook, the metrics endpoint alone won’t close this gap. You need a second signal that reads execution outcomes from n8n’s own store.

The public REST API exposes exactly that. Executions can be filtered by status, and the failed ones are one request away:

curl -s -H "X-N8N-API-KEY: $N8N_API_KEY" 
  "https://n8n.example.com/api/v1/executions?status=error&limit=50"

Two ways to get that into Grafana. Write a small exporter that polls the API on a timer and serves its own /metrics, or run a scheduled job that pushes counts to a Prometheus Pushgateway. The exporter is more work and behaves correctly. Pushgateway is faster to build and carries the usual caveat: pushed metrics persist until deleted, so a job that stops running leaves a stale value sitting there looking healthy. That reintroduces failure family one through the back door.

Two caveats on the API. There’s an open report that the status=crashed list filter returns executions whose stored status is something else entirely when re-fetched by ID, which would make crash alerting built on the list endpoint unreliable. Check the behavior on your version before depending on it. And n8n’s log streaming feature, which forwards execution events to syslog or a webhook in real time, is an Enterprise capability. If you’re on the community edition, the API poll is your path.

The third option, and often the best one for a small setup, is an Error Trigger workflow. n8n lets you designate a workflow that runs whenever another workflow fails. Point it at Pushgateway, at Loki, or at a Grafana webhook contact point. It’s native, it needs no API key, and it gives you the error message rather than just a count. The trade-off is that it only fires on real execution failures, so it inherits every blind spot in this section.

The Grafana panels that earn their space

Four panels cover most of what you need. Everything past that is usually decoration.

1. Executions per workflow, by name. A time series of rate(n8n_workflow_started_total[5m]) joined to n8n_workflow_info. Its real job is showing you the lines that flatten to zero.
2. Success ratio. rate(n8n_workflow_success_total[15m]) / rate(n8n_workflow_started_total[15m]). Anything below 1 is worth a look. Anything above 1 means your window is catching a restart and the counters reset.
3. Queue depth and failed jobs. n8n_scaling_mode_queue_jobs_waiting next to increase(n8n_scaling_mode_queue_jobs_failed[1h]). A waiting count that climbs and never drains means workers are dead or starved, which looks identical to “no failures” on every other panel.
4. Webhook non-2xx rate. The query from the previous section. This is the only panel that reports what the outside world experienced.

Skip the node-level breakdown unless you have a specific reason. It is the most expensive thing you can enable in cardinality terms and it answers a question you can answer faster by opening the execution in the n8n UI.

Alert rules worth paging on

1. Named workflow silent. absent_over_time on each business-critical workflow, window set to roughly twice its expected interval. This is the rule that catches the invoice sync.
2. Success ratio below threshold. Sustained for 15 minutes, not instantaneous. A single failed run is not a page.
3. Queue waiting count rising. Compare the current value to fifteen minutes ago rather than to a fixed number, so the threshold survives traffic growth.
4. Webhook error rate. Per workflow, so a single noisy endpoint doesn’t mask the rest.
5. Scrape target down. The up metric. Obvious, and routinely forgotten, and it invalidates every other rule on this list.

Troubleshooting: the endpoint returns nothing useful

• “Cannot GET /metrics”. N8N_METRICS isn’t set on the process you’re hitting, or it’s set as a string that n8n doesn’t read as true. In Docker Compose, quote it as "true" and confirm with docker compose exec n8n env | grep N8N_METRICS.
• Endpoint responds but has no n8n_ counters. You have the default Node.js process metrics and nothing else. Set N8N_METRICS_INCLUDE_MESSAGE_EVENT_BUS_METRICS=true and restart.
• Counters exist but no workflow_id label. N8N_METRICS_INCLUDE_WORKFLOW_ID_LABEL is off. Restart is required; it’s read at startup.
• Queue metrics are zero or missing. They’re gathered from Bull and exposed on the main instance, not the workers. Querying the worker endpoint gives you nothing. Confirm you’re looking at the right target.
• Counters look wrong after a deploy. They reset to zero when the process restarts. Always wrap them in rate() or increase(), which handle counter resets. Raw counter values on a dashboard will lie to you every time you deploy.
• Prometheus can’t reach the target. Usually a Docker network mismatch. Scrape by container name on the shared network rather than by published host port, and check the target page in Prometheus for the actual error rather than guessing.

Common mistakes

• Treating the success counter as proof the automation worked. It proves the run finished.
• Building only error-rate alerts, so the workflows that stop entirely go unmonitored.
• Enabling every N8N_METRICS_INCLUDE_* variable on day one and discovering the cardinality bill a month later.
• Scraping only the main instance in queue mode and losing all worker-side execution data.
• Copying label matchers out of documentation without running them and checking the returned series count.
• Exposing /metrics publicly because it’s “just numbers”.

Best practices

• Decide which workflows are business-critical and write a silence alert for each one by name. Six good heartbeats beat two hundred generic panels.
• Run a second source of truth. Metrics tell you about volume, the executions API tells you about outcomes. You need both.
• Push the ID-to-name join into a recording rule so panels stay readable and cheap.
• Break a staging workflow on purpose and confirm the alert fires. An untested alert is a decoration.
• If you’d rather not run Prometheus and Grafana yourself, Grafana Cloud will scrape a remote endpoint and gives you alerting without a control plane to maintain. Weigh that against your metrics leaving your network.

FAQ

Does n8n expose a failed-executions counter?

Not a reliable per-workflow one. You get started and success counters from the event bus, and a failed-jobs counter for queue mode. Deriving failures by subtracting success from started works only if you’ve verified on your own instance that a failing run genuinely doesn’t increment success, which is exactly the assumption that breaks when nodes are set to continue on error.

Can I monitor n8n Cloud this way?

No. The /metrics endpoint isn’t available on n8n Cloud. On Cloud you’re limited to the public API and the built-in insights views, so the executions-API approach in this post is the only one that transfers.

Why does my workflow error rate look like zero when I know things are failing?

Almost always one of three things: the event bus metrics aren’t enabled so you’re only seeing Node.js process metrics, the failures are being absorbed by continue-on-error settings and recorded as successes, or you’re scraping the main instance while the work happens on workers.

How do I alert when a scheduled workflow stops running entirely?

Use absent_over_time against that workflow’s started counter with an exact workflow_id matcher, and set the window to about twice the expected interval. A threshold comparison won’t work, because a workflow that stops running stops producing the series you’d compare against.

Is enabling the workflow_id label safe on a large instance?

It multiplies your n8n series count by the number of workflows. On a few dozen it’s negligible. Past a few hundred, combined with node-type labels, watch your Prometheus memory and head series count before and after. Enable the workflow label, leave the node label off, and only turn the node one on when you have a specific question it answers.

Should I use OpenTelemetry instead of Prometheus for n8n?

They solve different problems. Metrics answer “how many, how often, how fast” cheaply and forever. Traces answer “where did this run spend its time and what did it call”, which is better for debugging a slow multi-service workflow. Tracing n8n usually means a custom image with the OpenTelemetry SDK loaded before startup, so it’s a project rather than an environment variable. Start with metrics; add tracing when you have a latency question metrics can’t answer.

Where do the n8n counters go when the container restarts?

They reset to zero. They’re in-process counters with no persistence. Prometheus handles this correctly as long as your queries use rate() or increase(). Any panel showing a raw counter value will appear to lose all its history on every deploy.

The one thing to take away

Monitoring n8n workflow failures in Grafana works, but only once you stop asking the metrics endpoint a question it can’t answer. It reports activity. Activity and correctness are different things, and the space between them is where the expensive outages live.

So build for absence first. A silence alert on the handful of workflows a human would miss will catch more real incidents than any error-rate panel, because errors are loud and stopped schedules are not. Then add attribution with the workflow label, then close the false-success gap with the executions API or an Error Trigger workflow. In that order, because each one is cheap and the value drops off fast after the third.

Need help wiring this up?

I build and maintain observability stacks for self-hosted automation platforms, and n8n is a common one. If any of the above sounds like a week you’d rather not spend, here’s what I can take off your plate:

• Enabling and securing the n8n metrics endpoint on a private network path, including queue-mode setups with multiple workers
• Building Prometheus scrape configs, recording rules and the ID-to-name join so your Grafana panels are readable rather than a wall of hashes
• Writing silence and error-rate alert rules that survive counter resets, Grafana No Data states and traffic growth
• Adding a second source of truth from the n8n executions API, either as a small exporter or an Error Trigger workflow feeding Loki or Pushgateway
• Auditing an existing setup for cardinality problems, stale Pushgateway series and alerts that quietly stopped being able to fire
• Running a failure drill against staging so you know which alerts actually work before you need them

Send me your Compose file, your prometheus.yml, or a screenshot of the panel that looks fine while things are on fire, and I’ll tell you what’s missing.

The post Monitoring n8n Workflow Failures in Grafana: Why Your Green Dashboard Is Lying appeared first on John Nessime.

So you open the orchestrator. Every task is green. The run started on schedule, finished inside its usual window, exit code zero across the board. Nothing paged. And the numbers are still wrong.

That gap is the whole reason to build a data pipeline operations dashboard. Not to watch DAGs go green, which the orchestrator UI already does perfectly well, but to answer the questions the orchestrator structurally cannot answer. This post covers the four signals worth putting on a screen, where each one actually comes from, how to wire them into Prometheus and Grafana without inventing a metrics platform, and how to turn the panels into alerts that fire before a human notices.

Why green pipelines still ship bad data

An orchestrator tracks process outcomes. Did the Python exit cleanly, did the SQL statement return without an error, did the container terminate zero. That is a real signal and you should keep it. It is just answering a different question than the one your users care about.

Consider what a perfectly successful run looks like when the upstream source has quietly changed:

• An API starts paginating differently and your extractor pulls the first page only. It ran fine. It loaded 500 rows instead of 500,000.
• A source column gets renamed and your transform selects it with a coalesce default. It ran fine. Every row now says “unknown”.
• A partition arrives late, so the incremental filter finds nothing to process. It ran fine. It did nothing.
• An upstream team backfills, and your dedupe key no longer holds. It ran fine. Everything is doubled.

None of those raise an exception. All four are visible in under a second if you are tracking row counts and freshness at the table level. This is the failure family that eats the most incident time, because the clock does not start when the pipeline breaks. It starts when a human notices, which can be days later, and by then the bad data has been copied into reports, cached in a BI extract, and quoted in a meeting.

The four questions a data pipeline operations dashboard has to answer

Resist the instinct to start from what your tools can emit. Start from the questions you ask during an incident, in the order you ask them. Everything that does not answer one of these is a second-page panel.

Did it run at all?

Sounds trivial, and it is the one most homegrown dashboards get wrong. A panel showing “last run: success” does not distinguish between a pipeline that succeeded ten minutes ago and one that succeeded eleven days ago and has not been scheduled since. You want time since last successful completion, as a number that grows, not a status pill that sits there looking calm.

Did it finish in time?

Duration matters less than duration against a deadline. A load that takes forty minutes is fine if the report is read at 9am and the run starts at 6am. The same forty minutes is an incident if someone moved the schedule to 8:40. Put the deadline on the panel as a threshold line, not in a runbook. Trending duration also catches slow decay: a stage that creeps from five minutes to fifty usually means data growth, a bad query plan, or resource contention, and it will breach your window eventually.

Did the right amount of data arrive?

Rows in versus rows out, per stage. This is the highest value panel per unit of effort on the entire dashboard. A drop from a million rows to twenty thousand is not subtle once you plot it, but it is completely invisible in an exit code. Compare against the same weekday from the previous week rather than against yesterday, because most business data has a weekly shape and Monday-versus-Sunday comparisons generate noise you will learn to ignore.

Is the data still the shape you expect?

Null rates, distinct counts on key columns, and schema changes. This is the panel almost nobody builds, and it is where the expensive failures live. A gradual rise in null rate on a joining key means an upstream system is degrading and your joins are silently dropping rows. You do not need statistical anomaly detection to catch that. You need the number plotted over time next to the others.

There is a fifth question that belongs on a different surface: what breaks downstream when this table is wrong. That is lineage, and it answers blast radius rather than health. Keep it one click away, not on the operations screen.

Where each signal actually comes from

Three sources, three different collection mechanisms. This is the part that takes an afternoon rather than a sprint, provided you do not try to build a unified agent.

The orchestrator, for run and timing signals

Airflow has emitted StatsD metrics for a long time, and that path is still the one most deployments run. Airflow speaks StatsD, so you put a statsd_exporter in front of it to translate into Prometheus format. Enable it in the metrics section of your config, or with the equivalent AIRFLOW__METRICS__ environment variables if you are on containers:

[metrics]
statsd_on = True
statsd_host = statsd-exporter
statsd_port = 9125
statsd_prefix = airflow
metrics_allow_list = scheduler,executor,dagrun,pool,triggerer

Two things worth knowing before you roll this out. First, metrics_allow_list is not optional at any real scale. Airflow emits per-DAG and per-task metric names, so without a prefix filter your cardinality grows with every DAG anyone adds, and you find out when the time series database starts struggling. Second, Airflow expects the StatsD destination to be resolvable at startup, so a scheduler that boots before the exporter can fail to come up. Order your dependencies accordingly.

StatsD has no concept of labels, so the exporter has to reconstruct them from the dotted metric name. That mapping file is the fiddly part:

mappings:
  - match: "*.dag.*.*.duration"
    match_metric_type: observer
    name: "airflow_task_duration"
    labels:
      airflow_id: "$1"
      dag_id: "$2"
      task_id: "$3"

The wildcards map positionally onto $1, $2, $3, and match_metric_type has to match what Airflow actually sends, timer versus counter versus gauge. Get that wrong and the metric silently never appears at the exporter, which is a genuinely annoying half hour of debugging. Check the exporter’s own metrics endpoint before you go looking at Prometheus, because if it is not there it never left Airflow.

Airflow also has a native OpenTelemetry path for both metrics and traces, which skips the StatsD translation entirely and gives you real labels rather than reconstructed ones:

[metrics]
otel_on = True
otel_host = otel-collector
otel_port = 4318
otel_prefix = airflow
otel_interval_milliseconds = 30000

The port there is the collector’s OTLP HTTP receiver, and you need an OpenTelemetry Collector or compatible endpoint in the path. If you are standing up a new stack, this is the one I would reach for, because the label problem disappears and traces give you per-task spans under a DAG run parent span, which is worth a lot during a slow-pipeline investigation. If you already have working StatsD mappings and dashboards built on them, migrating is real work for a modest gain, and the honest answer is that it can wait. Metric names do shift between Airflow major versions either way, so pin your dashboard queries to the metrics reference for the version you actually run.

Managed Airflow changes the plumbing but not the shape. On Amazon MWAA the metrics land in CloudWatch instead, and Google Cloud Composer and Astronomer each have their own export path. The four questions do not change.

The transformation layer, for data-level signals

If you use dbt, most of what you need is already being written to disk and thrown away. Every invocation produces run_results.json in the target directory, containing per-node status and timing, and dbt source freshness produces its own sources.json. The manifest.json holds the dependency graph, which is your lineage.

The cheapest useful thing you can do is parse the failures out at the end of the run:

jq -r '.results[]
  | select(.status != "success")
  | [.unique_id, .status, (.execution_time | tostring)]
  | @tsv' target/run_results.json

One trap if you run dbt in ephemeral Kubernetes pods: the target directory dies with the pod. Upload the artifacts to object storage as the last step inside the same task that ran dbt, not as a downstream task, or you will find your history is empty exactly when you want it.

Elementary is the low-friction option here. It installs as a dbt package with an on-run-end hook that loads the artifacts into your warehouse as ordinary tables, which means your dashboard can query test results and model timings in SQL alongside everything else. Great Expectations and Soda cover the same validation ground with more expressive checks and more setup. The commercial platforms, Monte Carlo and the vendors around it, add automated anomaly detection and column-level lineage across the whole stack, and the genuine case for them is that they find the checks you did not think to write. The trade-off is that you are paying per monitored table for statistical inference, and on a stack of twenty tables that inference is not telling you much that a row-count panel and a null-rate panel would not.

Without dbt, you write the check yourself and it is still short. One query per critical table, run on a schedule:

SELECT
  COUNT(*)                                        AS row_count,
  MAX(updated_at)                                 AS last_event,
  SUM(CASE WHEN customer_id IS NULL THEN 1 ELSE 0 END) AS null_keys
FROM analytics.orders
WHERE loaded_at >= CURRENT_DATE - 1;

Batch jobs, and the Pushgateway caveat

Prometheus pulls. Batch jobs finish and vanish long before a scrape interval comes round, so there is nothing to pull from. The Pushgateway exists for exactly this: short-lived service-level jobs push their results to it, and Prometheus scrapes the gateway instead.

cat <<EOF | curl --data-binary @- 
  http://pushgateway:9091/metrics/job/orders_sync
# TYPE pipeline_last_success_timestamp_seconds gauge
pipeline_last_success_timestamp_seconds $(date +%s)
# TYPE pipeline_rows_loaded gauge
pipeline_rows_loaded ${ROWS}
EOF

Now the caveat, which is the single most common way a pipeline dashboard lies to you. The Pushgateway is a cache, not an aggregator, and it never forgets. Push a success metric, then have the job stop running entirely, and that success sits there being scraped forever. Your dashboard stays green because the last known value is green, and the fact that nothing has updated it in six days is not something a status panel can see.

Two defences, and you want both. First, always push a timestamp gauge rather than a boolean status, so staleness is arithmetic rather than inference. Second, alert on absence as well as on value, because the Pushgateway also exposes push_time_seconds per group, which tells you when anything last arrived regardless of what it said. Prometheus’s usual safety net, the up metric going to zero when a target disappears, does not apply here: Prometheus is scraping the gateway successfully, and the gateway is happily serving stale data.

Also keep instance identifiers out of the grouping labels. Push per service, not per worker, or you accumulate a permanent time series for every container that ever ran.

Laying out the dashboard so it works during an incident

The layout constraint is that someone is reading this at 7am on a phone with a stakeholder waiting. Three rows, top to bottom, in the order you need them:

1. Freshness across everything. One panel, one row per critical table, showing time since last successful load against its threshold. If this row is clean, the pipelines are fine and the problem is elsewhere. That is a genuinely useful thing to establish in five seconds.
2. Volume and duration per pipeline. Row counts with a week-over-week comparison, and stage duration with the deadline drawn on. This is where you localise a problem to a specific pipeline and stage.
3. Data quality detail. Null rates, failed test counts, schema change events. This is where you work out what actually went wrong once you know where.

The freshness query in PromQL, given the timestamp gauge above:

time() - max by (job) (pipeline_last_success_timestamp_seconds)

Grafana renders that nicely as a table with per-row thresholds, which is what makes the top row scannable. If you are self-hosting, Prometheus plus Grafana on a modest VPS from Hetzner, InterServer or DigitalOcean handles a surprising amount of pipeline metrics, because pipeline telemetry is low-volume compared to application metrics. Grafana Cloud’s free tier is a reasonable alternative if you do not want to own the storage, and Datadog or New Relic make sense if your organisation is already paying for one of them and you would rather have one pane than a better one.

Keep it to one screen. Every panel you add past that dilutes the ones that matter, and a dashboard nobody can read at a glance is a dashboard nobody opens.

Turning panels into alerts

A dashboard that requires someone to look at it is a dashboard that finds problems the morning after. The freshness panel converts into an alert almost directly:

groups:
  - name: pipelines
    rules:
      - alert: PipelineStale
        expr: time() - max by (job) (pipeline_last_success_timestamp_seconds) > 5400
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.job }} has not completed successfully in over 90 minutes"

      - alert: PipelineMetricsMissing
        expr: absent(pipeline_last_success_timestamp_seconds{job="orders_sync"})
        for: 30m
        labels:
          severity: page
        annotations:
          summary: "No metrics received from orders_sync"

The first catches a pipeline that is late or broken. The second catches the one that stopped reporting entirely, which the first cannot see, because an expression over a series that no longer exists returns nothing and an empty result does not fire. That pairing is the whole trick, and it applies to every push-based metric you collect.

Set the threshold from the deadline the data has to meet, not from how long the job usually takes. Ninety minutes is not “the run takes an hour so give it some room”. It should be the point past which someone downstream is materially affected.

Troubleshooting a dashboard that shows nothing useful

The metric never appears in Prometheus. Work backwards along the path, one hop at a time. Curl the statsd_exporter’s own metrics endpoint first. If the metric is not there, it never left the orchestrator, so the problem is the config or an allow-list that filtered it out. If it is there but not in Prometheus, the problem is your scrape config. Do not start in Grafana.

The metric appears with no labels, or with the DAG name baked into the metric name. Your statsd_exporter mapping did not match. Check the wildcard count against the actual dotted name, and check match_metric_type, which is the usual culprit.

Everything is green but the data is old. Classic Pushgateway staleness. Query push_time_seconds for the affected group and compare it to now. If it has not moved, the job is not running and you are looking at a cached value.

Volume alerts fire every Monday. You are comparing against the previous day on data with a weekly cycle. Compare against the same day last week using an offset, or exclude weekends from the baseline.

Gaps in the duration graph. If the metric is a timer sent only on completion, a run that is still going or that failed early sends nothing. That is correct behaviour, and it is why liveness needs its own panel rather than being inferred from the duration series.

Prometheus memory climbing after rollout. Cardinality. Per-DAG and per-task metrics multiply, and any label carrying a run identifier, timestamp or UUID creates a permanent new series. Tighten the allow list and drop the offending labels.

Common mistakes

• Treating job success as data correctness. This is the entire problem, and every other mistake is a variation of it.
• Pushing a boolean status instead of a timestamp, so staleness becomes invisible.
• Alerting on threshold breaches but never on metric absence.
• Building the dashboard before agreeing what “healthy” means for each table with the people who consume it.
• Instrumenting every pipeline at once instead of the three that matter, then abandoning the project when the noise gets unmanageable.
• Putting instance or pod identifiers into Pushgateway grouping labels and accumulating dead series forever.
• Alerting on things nobody will act on at 3am. An alert that gets acknowledged and ignored twice has trained your team to ignore it permanently.

Best practices

• Baseline before you threshold. Collect for two or three weeks and look at the actual shape of the data before deciding what abnormal means.
• Start with freshness on your three most critical tables. It catches more real problems per line of code than anything else you can instrument.
• Define freshness thresholds from the consumer’s deadline, and write the consumer’s name next to the threshold so the number has an owner.
• Version dashboards and alert rules in Git alongside the pipelines. A dashboard edited live in the UI is a dashboard that will be silently different next quarter.
• Emit row counts at stage boundaries as a habit, not as a special case. In versus out at each hop is what turns “the numbers are wrong” into “stage three drops rows”.
• Review alerts monthly and delete the ones that never fired usefully. Alert rules are code and they rot the same way.
• Keep lineage accessible but off the main screen. It answers a different question and it competes for attention with the ones you need first.

Frequently asked questions

What is the difference between pipeline monitoring and data observability?

Monitoring watches the process: did the job run, did it succeed, how long did it take. Observability watches the data the process produced: is it fresh, is there the right amount, has its shape changed. You need both, and the second is the one most teams are missing when they say their dashboards did not catch something.

Do I need a commercial data observability platform?

Not to start. Freshness, volume and null-rate checks on your critical tables cover most real incidents and cost an afternoon. Commercial platforms earn their keep when you have hundreds of tables, several teams changing them independently, and nobody with time to hand-write checks. The honest signal that you have outgrown the homegrown version is when maintaining the checks becomes its own recurring task.

Should I use StatsD or OpenTelemetry for Airflow metrics?

New stack, OpenTelemetry: you get real labels instead of reconstructing them from dotted names, and traces come along with it. Existing stack with working StatsD mappings, stay put until you have another reason to touch it. Metric coverage and naming differ between the two paths and between Airflow versions, so verify against the metrics reference for your version rather than copying a dashboard from a blog post.

How do I monitor a pipeline that only runs weekly?

Same timestamp gauge, much larger threshold, and lean harder on the absence alert. Infrequent pipelines are where Pushgateway staleness does the most damage, because the last successful push can sit there for a fortnight looking entirely healthy. Check push_time_seconds for those groups specifically.

What should a data pipeline operations dashboard show first?

Time since last successful load, per critical table, with its threshold visible. That one panel answers the question you will be asked most often, and when it is clean it rules out the entire pipeline layer in a glance.

How many alerts is the right number?

Few enough that every page gets investigated. If someone can name an alert they routinely dismiss without looking, you have too many, and the cost is not the noise itself but the habit it builds. Start with freshness and absence per critical pipeline, add volume, and only go further when a real incident shows you a gap.

Can I build this without an orchestrator like Airflow?

Yes. Cron jobs, Step Functions, GitHub Actions and shell scripts can all push a timestamp and a row count to a Pushgateway in three lines. The orchestrator gives you the run and timing signals for free, but it is the data-level signals that do the heavy lifting, and those come from querying the warehouse regardless of what triggered the load.

The one thing to take away

A data pipeline operations dashboard is not a nicer view of your orchestrator. It exists because the orchestrator can only tell you that a process finished, and the question you are actually being asked is whether the data is right.

If you build one panel this week, build time since last successful load, sourced from a timestamp gauge, alerted on both threshold and absence. It costs an afternoon, it catches the failure mode that hides longest, and it turns “is the dashboard broken or just late?” into a question you can answer before anyone has to ask it.

Need help building or fixing this?

I work with teams on pipeline observability and the plumbing underneath it. Things I regularly help with:

• Designing freshness, volume and quality checks for your critical tables, with thresholds derived from real consumer deadlines rather than guesses
• Wiring Airflow into Prometheus and Grafana, including the statsd_exporter mappings and the cardinality controls that keep it affordable
• Migrating an existing StatsD setup to the OpenTelemetry path without losing your dashboard history
• Getting dbt artifacts out of ephemeral pods and into somewhere you can query them, with Elementary or with plain object storage
• Auditing an existing dashboard and alert set to find what is stale, what will never fire, and what is quietly training your team to ignore pages
• Standing up a self-hosted Prometheus and Grafana stack on a VPS, or sizing a managed alternative if owning the storage is not worth it to you

If you have a dashboard that went green through an incident, send me the panel queries and the alert rules and I will tell you where the gap is.

The post Building a Data Pipeline Operations Dashboard That Catches Silent Failures appeared first on John Nessime.