That gap is the entire problem. Run status tells you the code executed. It tells you nothing about whether data arrived. Data pipeline freshness monitoring is the signal that closes the gap, and most teams add it only after an analyst finds the stale table first.

This post covers the three signals worth instrumenting for a batch or streaming pipeline: how current the data is, how long it took to get there, and how often the thing breaks. For each one I’ll cover what to measure, where to measure it, the queries and config that produce it, and the specific ways the measurement lies to you.

The signal that lies to you first: run status

Job success is a statement about your code path, not about your data. A pipeline that reads an empty S3 prefix, transforms zero rows, writes zero rows and exits cleanly has succeeded. So has one whose upstream API silently started returning an empty page after a token rotation. So has one whose incremental watermark got stuck and now re-reads the same already-loaded slice on every run.

All three are green. All three are producing stale data. This is why the ordering matters: freshness is the signal that fires first in a real incident, and run status is usually the last one to notice anything at all.

So keep run status. It’s cheap and it catches crashes. Just stop treating it as your top-line health indicator.

Freshness: measure the table, not the job

Freshness is the age of the newest record in a dataset. It’s measured against the dataset itself, not against the process that filled it, and that distinction is what makes it useful. If your job disappears entirely, freshness keeps climbing and keeps alerting. If your job succeeds while doing nothing, freshness keeps climbing and keeps alerting. It’s the one signal that survives both failure modes.

Two clocks, and you need both

Every row usually carries two timestamps: when the event happened in the source system, and when your pipeline wrote it. Track the maximum of each.

• Max event time tells you whether the upstream system is still producing. If this stops moving, the problem is upstream of you.
• Max load time tells you whether your pipeline is still writing. If this stops moving while event time is fine, the problem is yours.

Watching only one of them means every stale-data page starts with twenty minutes of figuring out which side of the boundary the fault sits on. Watching both answers that in the alert body.

A freshness probe is a small scheduled query. This one is PostgreSQL syntax, and it returns both clocks plus a volume check in a single round trip:

select
  'orders' as dataset,
  extract(epoch from max(event_ts))  as max_event_ts,
  extract(epoch from max(loaded_at)) as max_loaded_at,
  count(*) filter (where loaded_at >= now() - interval '1 hour') as rows_last_hour
from analytics.orders;

The row count matters. A pipeline can advance its load timestamp while writing almost nothing, which is what a partially broken source looks like. Freshness alone will not catch that; freshness plus volume will.

Run that probe on a schedule that’s independent of the pipeline, push the results as gauges, and alert on age in PromQL:

# Newest row is older than 90 minutes
time() - max by (dataset) (dataset_max_loaded_timestamp_seconds) > 5400

# The probe itself has stopped reporting: a deadman check
absent(dataset_max_loaded_timestamp_seconds{dataset="orders"})

That second rule is the one people forget. A freshness metric that disappears looks identical to a healthy silence on a graph. absent() is what turns “no data” into a page.

Where dbt fits

If you’re already running dbt, source freshness is built in and worth using before you write anything custom. You declare thresholds per source, and dbt queries the maximum of your timestamp column and compares it to now.

sources:
  - name: raw_shop
    schema: raw
    config:
      loaded_at_field: _loaded_at
      freshness:
        warn_after: {count: 2, period: hour}
        error_after: {count: 6, period: hour}
    tables:
      - name: orders
      - name: refunds
        config:
          freshness:
            warn_after: {count: 12, period: hour}
            error_after: {count: 24, period: hour}

Three details that trip people up. First, dbt source freshness is a separate command; dbt build does not run it, so a green build says nothing about source staleness. Second, it exits non-zero when a source hits its error_after threshold, which makes it a natural gate at the top of a job: fail fast rather than building models on stale input. Third, results land in target/sources.json, which is the artifact you parse if you want to distinguish warn from error, or ship the numbers into Prometheus rather than just failing the run.

dbt source freshness --select source:raw_shop --output target/freshness.json

One caveat on the YAML above: dbt moved these keys under a config: block in recent releases, and loaded_at_field followed later. Older projects nest them directly under the source. Check what your project’s version expects before copying, because a misplaced key fails quietly by simply not calculating freshness at all.

Latency: name the clock before you name the number

“Our pipeline latency is twelve minutes” is meaningless until you say which two points you measured between. There are at least three plausible definitions, and teams routinely argue past each other because they’re each using a different one.

• Run duration. Start to end of the job. Easy, and mostly useless for anyone downstream.
• Ingestion latency. Source extract to target load. This is what you control.
• End-to-end latency. Event time to the moment the row is queryable. This is what the business actually feels, and it includes queue wait, scheduler delay, and every upstream hop you don’t own.

Publish end-to-end as the headline and keep the others as breakdown. If you only track run duration, a scheduler backlog that delays every run by forty minutes is completely invisible to you: each individual run still takes eleven minutes.

Instrument per stage with a histogram so you can ask percentile questions later without re-instrumenting:

histogram_quantile(
  0.95,
  sum by (le, stage) (rate(pipeline_stage_duration_seconds_bucket[6h]))
)

Use the median for capacity planning and p95 or p99 for the SLO. Averages hide the exact tail that generates the complaints, and on a pipeline that runs a few dozen times a day, a single pathological run is a real fraction of your day.

For streaming, the equivalent is consumer lag expressed in time rather than offsets. Offset lag of fifty thousand messages means nothing without a rate; two hundred seconds of lag means something to everyone. Kafka Lag Exporter popularised this by interpolating a time estimate from observed offset and timestamp samples, exposing kafka_consumergroup_group_max_lag_seconds alongside the offset-based kafka_consumergroup_group_lag. Worth knowing: that project’s repository has been archived and is read-only, so if you’re starting fresh, check whether your broker vendor or a maintained fork covers it before you deploy something unmaintained into the critical path.

Failure rate: decide what counts as a failure

Failure rate is trivially easy to compute and surprisingly easy to compute wrongly. The denominator and the definition both need a decision.

Count runs, not tasks. A DAG with sixty tasks where one flaps on a transient network error looks catastrophic at task level and fine at run level. The run is the unit the consumer cares about.

Count a run that succeeded on its third retry as a success for availability and a failure for a separate reliability metric. Both are true and they answer different questions. If your only metric folds retries into success, you will never see the slow degradation of an upstream API until it stops responding entirely.

sum by (pipeline) (rate(pipeline_runs_total{result="failure"}[6h]))
/
sum by (pipeline) (rate(pipeline_runs_total[6h]))

And add a category label for the failure reason at push time: source unavailable, schema mismatch, permission denied, timeout, validation failed. The rate tells you something is wrong. The category tells you who to wake up. Without it, every failure alert costs you a log dive before you can even route the incident.

Getting metrics out of a job that exits

Prometheus scrapes. Batch jobs finish and vanish. Pushgateway bridges that: the job pushes before exiting, and Pushgateway holds the values for Prometheus to scrape on its own schedule.

#!/usr/bin/env bash
set -euo pipefail

JOB="orders_load"
PGW="http://pushgateway.internal:9091"

start=$(date +%s)
if python /opt/pipelines/load_orders.py; then
  result=0
else
  result=1
fi
end=$(date +%s)

{
  cat <<EOF
# TYPE pipeline_run_duration_seconds gauge
pipeline_run_duration_seconds $((end - start))
# TYPE pipeline_last_run_timestamp_seconds gauge
pipeline_last_run_timestamp_seconds $end
# TYPE pipeline_last_run_success gauge
pipeline_last_run_success $((1 - result))
EOF
  if [ "$result" -eq 0 ]; then
    cat <<EOF
# TYPE pipeline_last_success_timestamp_seconds gauge
pipeline_last_success_timestamp_seconds $end
EOF
  fi
} | curl --fail --data-binary @- "$PGW/metrics/job/$JOB"

exit $result

The conditional block is doing real work. curl’s --data-binary issues a POST, and a POST to Pushgateway replaces only the metrics whose names appear in the payload, leaving the rest of the group intact. So a failing run updates the run timestamp and the success flag while leaving the previous pipeline_last_success_timestamp_seconds exactly where it was. That’s what lets time() - pipeline_last_success_timestamp_seconds keep climbing across consecutive failures. Send a PUT instead and you replace the whole group, wiping the value you needed.

On the Prometheus side, one setting is not optional:

scrape_configs:
  - job_name: pushgateway
    honor_labels: true
    static_configs:
      - targets: ['pushgateway.internal:9091']

Without honor_labels: true, Prometheus overwrites the job label your pipeline pushed with the scrape job’s own name, and every pipeline in your estate collapses into one indistinguishable series called pushgateway.

The trap worth internalising: Pushgateway never forgets. Metrics persist until something explicitly deletes them or the process restarts. A pipeline you decommissioned last quarter is still cheerfully reporting a success timestamp and a duration, and it looks alive on every dashboard. Treat Pushgateway as a cache of past executions, not a picture of current state, and use the push_time_seconds gauge that Pushgateway attaches to each group to tell the difference between a fresh push and a fossil.

If you’re on Airflow, the SLA feature is gone

This one will bite anyone upgrading. The sla and sla_miss_callback parameters were removed in Airflow 3.0, and the replacement, Deadline Alerts, arrived in 3.1. DAGs carrying the old configuration need manual migration; they don’t quietly keep working.

from datetime import timedelta

from airflow.sdk import AsyncCallback, DAG, DeadlineAlert, DeadlineReference
from airflow.providers.slack.notifications.slack_webhook import SlackWebhookNotifier

with DAG(
    dag_id="orders_load",
    deadline=DeadlineAlert(
        reference=DeadlineReference.DAGRUN_QUEUED_AT,
        interval=timedelta(minutes=45),
        callback=AsyncCallback(
            SlackWebhookNotifier,
            kwargs={"text": "orders_load has not finished 45 minutes after queuing."},
        ),
    ),
):
    ...

Note the reference point. Measuring from when the run was queued, rather than from its logical date, means scheduler backlog counts against the deadline. That’s usually what you want, because a run that sat in a queue for an hour is late to its consumers regardless of how fast it executed once it started.

There’s a structural weakness here worth naming: any alert that lives inside the orchestrator dies with the orchestrator. If the scheduler is down, nothing evaluates your deadline and nothing notifies anyone. The freshness probe from earlier is the answer, and it needs to run somewhere else. A small VPS from a provider like Contabo or InterServer running Prometheus and Alertmanager, or a hosted option like Grafana Cloud, gives you a watcher outside the blast radius of the thing being watched. This is the single highest-value piece of monitoring most data teams are missing.

Alerting without burning your on-call

The fastest way to make all of this worthless is to alert on every threshold crossing. Three rules keep it survivable.

1. Set thresholds from observed behaviour, not from wishes. Look at a month of actual freshness values and set warn_after a comfortable margin above the normal worst case. A source that habitually passes at eleven hours against a twelve-hour threshold is not healthy, it’s one upstream hiccup from paging you.
2. Page on consumer impact, not internal events. A task retry is not an incident. A dataset breaching the freshness commitment its consumers rely on is. Route everything else to a channel someone reads in the morning.
3. Use burn rate over multiple windows for SLOs. A short window catches fast breakage, a long window catches slow erosion, and requiring both to fire filters out the transient spikes that generate most false pages.

One more: alert on the absence of your own telemetry. A freshness gauge that stops updating is indistinguishable from one that’s fine, right up until someone asks about the numbers.

Troubleshooting

Freshness alert fires but the data looks current. Almost always a timezone problem. Your loaded_at column is in local time, now() is in UTC, and the offset shows up as a constant bias in the age. Store load timestamps in UTC and cast explicitly at read time.

Freshness flaps in and out of breach. Your check runs too close to the expected arrival. If loads land around six and your probe runs at five past, a fifteen-minute upstream delay produces an intermittent failure that trains everyone to ignore the alert. Move the probe later or widen the threshold.

Every pipeline reports as one series. honor_labels: true is missing from the Pushgateway scrape config.

A decommissioned pipeline still shows healthy. Stale group in Pushgateway. Delete the group and add a check on push_time_seconds so the next one surfaces on its own.

Latency looks fine but consumers say data is late. You’re measuring run duration and they’re feeling end-to-end. Add queue wait and event-time-to-load and the gap will be obvious.

Freshness passes, row count is near zero. The load timestamp advanced without meaningful data. This is a broken source or a stuck watermark, and it’s the reason the volume check belongs in the same probe.

Common mistakes

• Treating job success as the health signal and discovering staleness through a human.
• Running the freshness check inside the same pipeline it’s meant to police.
• Tracking freshness without volume, so a zero-row load reads as healthy.
• Reporting latency without saying which two timestamps it spans.
• Putting run IDs, batch IDs or timestamps into Prometheus labels, which multiplies your series count without bound. Those belong in traces or logs.
• Alerting on task-level failures instead of run-level outcomes, then muting the whole channel a week later.
• Assuming Pushgateway reflects current state rather than the last thing anyone pushed.

Best practices for data pipeline freshness monitoring

• Start with your three most-used tables. Full coverage is a project; three tables is an afternoon and catches most of the pain.
• Write down the freshness commitment per dataset in plain language, then encode it. “Yesterday’s orders are complete by 07:00” converts directly into a threshold.
• Name metrics consistently across pipelines. A shared prefix and a stable label set is what makes one dashboard work for all of them.
• Keep labels low-cardinality: pipeline, dataset, stage, environment, result. Nothing unbounded.
• Emit a failure category alongside every failure so alerts route themselves.
• Run the deadman check on infrastructure that doesn’t share a failure domain with the pipeline.
• Link every alert to a runbook that names the owner and the first three things to check.

Frequently asked questions

What is the difference between pipeline latency and data freshness?

Latency measures how long a specific batch of data took to travel from source to destination. Freshness measures how old the newest available record is right now, regardless of whether anything is currently running. A pipeline can have excellent latency and terrible freshness if it stopped being triggered.

How often should freshness checks run?

Frequently enough that you find out before your consumers do. A useful rule is roughly a quarter of your tolerance window: if data may be up to four hours old, check hourly. Freshness probes are cheap single-aggregate queries, so the limiting factor is usually warehouse billing rather than load.

Do I need a data observability platform for this?

Not to start. Freshness, latency and failure rate for a handful of critical datasets is a scheduled query, a push, and a few alert rules. Commercial platforms earn their cost when you need automatic column-level lineage, anomaly detection across hundreds of tables, or coverage of assets nobody has explicitly instrumented. That’s a real problem at scale, and a genuinely expensive one to build yourself. It’s just not the problem you have on day one.

How do I monitor freshness for a table that only updates weekly?

Set the threshold from the schedule plus a delivery margin, and add a deadman rule so a missing metric alerts on its own. For genuinely static reference tables, disable freshness explicitly rather than leaving a check that always warns; in dbt that means setting freshness to null for the table.

Should freshness checks fail the pipeline or just warn?

Both, in different places. Checking source freshness at the start of a job and failing hard prevents you building models on stale input, which is the cheapest bug to prevent and the most expensive to unwind. Checking output freshness after the fact should alert rather than fail, because the run is already over.

What percentile should I use for a latency SLO?

p95 for most internal analytics pipelines, p99 where downstream systems make automated decisions on the data. Track the median separately for capacity planning. Never use the average as the SLO number, because it hides exactly the tail that produces complaints.

Does this work for streaming pipelines too?

Yes, with different plumbing. Freshness becomes consumer lag measured in time, latency becomes event-time to availability-time, and failure rate becomes connector and task state plus dead-letter volume. The reasoning is identical; only the source of the numbers changes.

Conclusion

If you remember one thing, make it this: measure the data, not the job. Run status, task counts and duration all describe your code. Only freshness describes what your consumers actually receive, and it’s the one signal that stays honest when the pipeline succeeds at doing nothing.

Good data pipeline freshness monitoring is not a platform purchase. It’s a scheduled query that reports the age and volume of your most important tables, pushed somewhere durable, with a deadman rule so that silence is treated as a failure rather than as health. Add latency broken down by stage and failure rate categorised by reason, and you can answer “is the data good right now” without opening a single log file.

Need help instrumenting your pipelines?

Most of my consulting work in this area is retrofitting observability onto pipelines that already exist and can’t be paused. Specifically:

• Defining freshness and latency commitments per dataset, then translating them into thresholds and alert rules that hold up on-call.
• Adding freshness and volume probes to existing warehouses without touching the pipelines themselves.
• Wiring batch jobs into Prometheus through Pushgateway, including the grouping-key and stale-metric problems that bite six months later.
• Migrating Airflow DAGs off the removed SLA feature onto Deadline Alerts, or onto external checks that survive a scheduler outage.
• Building the Grafana dashboard that answers “is the data good right now” in one screen, with drill-down by stage and failure category.
• Cutting alert noise on pipelines where the channel has already been muted, by moving from task-level events to consumer-impact SLOs.

If you want a concrete starting point, send me a DAG file, a scrape config, or a screenshot of the dashboard you don’t trust, and I’ll tell you what I’d instrument first and why.

The post Green Pipeline, Stale Data: Tracking Latency, Freshness and Failure Rates appeared first on John Nessime.

Nobody broke anything. Shopify, Stripe, the CRM and the helpdesk each created a customer record using whatever identifier they had at the moment they needed one, and none of those identifiers agreed. The systems are all correct in isolation. The person is the thing that got lost.

This post covers what actually breaks when you unify Shopify, Stripe, CRM and support data, and how to build the join so it survives guest checkouts, duplicate webhooks, refunds, currency conversion and deletion requests. It’s organized by failure family rather than by tool, because the tools get acquired and renamed every couple of years and the failures never change.

Failure family one: the join key is not the email address

Every one of these systems will happily give you an email address, which is exactly why so many pipelines join on it and quietly produce garbage.

Start with where identity actually lives in each system:

• Shopify has a customer ID, but an order does not have to be attached to one. Guest checkout produces an order with an email on it and no durable customer record behind it.
• Stripe has a customer ID, but only if a Customer object was created. Checkout Sessions that don’t create one are associated with guest customers instead, which Stripe groups in the Dashboard based on the same card, email or phone. That grouping is read-only and it is not an ID you can join on.
• The CRM keys on email in almost every default configuration, and cheerfully creates a second contact when the email differs by one character.
• The helpdesk keys on the requester’s email, which is whatever the person typed into a contact form or sent the ticket from. It is very often not the address on the order.

Here’s the part that stays invisible until someone finally checks: nothing errors. Every dashboard still returns a number. Lifetime value gets silently split across three profiles, so your LTV looks lower and your repeat-purchase rate looks worse than it is. Support metrics look better than they are, because a customer coming back with the same unresolved problem shows up as a brand new contact. You can run like that for a year without a single failed job.

Build an identifier table, not a fuzzy match

The fix is boring and it works: a surrogate person key, and a separate table of every identifier ever observed for that person, with the source that reported it.

-- One row per real human, generated by you, owned by you.
CREATE TABLE person (
  person_key      uuid PRIMARY KEY,
  created_at      timestamptz NOT NULL DEFAULT now(),
  merged_into     uuid NULL REFERENCES person(person_key)
);

-- One row per identifier per source. This is the whole trick.
CREATE TABLE person_identifier (
  person_key      uuid NOT NULL REFERENCES person(person_key),
  source          text NOT NULL,   -- shopify | stripe | crm | helpdesk
  id_type         text NOT NULL,   -- email | phone | shopify_customer_id | stripe_customer_id
  id_value        text NOT NULL,   -- already normalized
  first_seen_at   timestamptz NOT NULL,
  last_seen_at    timestamptz NOT NULL,
  PRIMARY KEY (source, id_type, id_value)
);

Normalization happens before the row is written, not in the join. Lowercase and trim emails, store phone numbers in E.164, strip whitespace from external IDs. Resist the urge to get clever with Gmail’s dot and plus handling. Stripping +tags is a business policy decision with real consequences, not a technical detail, and the same rule applied to a domain that treats plus addresses as distinct mailboxes will merge two genuinely different people.

Deterministic matching first, always. Exact matches on shared identifiers, transitively closed. Probabilistic matching on name plus address plus device is worth reaching for only when you have exhausted the deterministic keys, and it has an asymmetric failure mode: a household sharing one card gets collapsed into one person, and unmerging afterwards is expensive because every downstream system has already copied the merged key into its own records. Merges are cheap to make and painful to undo. Bias toward leaving two records separate.

One thing worth doing early on the Stripe side: when you create a Customer, write your own identifier into metadata. Stripe allows fifty key-value pairs per object, with keys up to forty characters and values up to five hundred, and it only returns metadata for requests made with a secret key. That’s enough room for a person key and an order reference, which turns a future fuzzy join into an exact one. Never put anything sensitive there.

Failure family two: webhooks arrive twice, out of order, or not at all

Both platforms are explicit about this, and both get treated as if they weren’t.

Stripe delivers at least once, does not guarantee ordering, and retries failed deliveries with exponential backoff for up to three days. The event id (the one starting evt_) stays identical across every retry, which makes it your deduplication key. Shopify sends X-Shopify-Webhook-Id for the same purpose, signs the raw body into X-Shopify-Hmac-Sha256, and states plainly that ordering isn’t guaranteed within a topic or across topics for the same resource. Shopify’s own documentation goes further and tells you not to rely on webhooks as your only source of data.

For a customer-data pipeline specifically, out-of-order delivery is nastier than duplication. Consider a customers/update that changes someone’s email arriving before the orders/create that was placed under the old address. Process those in arrival order and you attach the order to a person who doesn’t exist yet, or worse, to the wrong one. Order your writes by the event’s own timestamp, using X-Shopify-Triggered-At or the updated_at in the payload, and make profile updates last-writer-wins on that timestamp rather than on insertion time.

Do almost nothing in the handler

Verify the signature, claim the event ID, write the raw body to durable storage, return 2xx. That’s it. Identity resolution, CRM lookups and enrichment calls all belong in a worker reading from that store. A handler that tries to resolve identity inline will eventually be slow enough to time out, which triggers a retry, which arrives while the first attempt is still running.

Make the claim atomic rather than check-then-act, because two retries genuinely can land concurrently:

-- Atomic claim. If this inserts zero rows, someone already has it.
INSERT INTO webhook_event (source, event_id, topic, received_at, payload)
VALUES ($1, $2, $3, now(), $4)
ON CONFLICT (source, event_id) DO NOTHING
RETURNING event_id;

Return a 2xx for a duplicate. A duplicate is a success from the sender’s point of view, and returning an error just schedules more of them.

Signature verification has one recurring trap on the Shopify side: it runs against the raw body. Any JSON body parser that runs first will change the bytes and every signature will fail. Capture the raw body before parsing, and compare in constant time.

import base64, hashlib, hmac

def verify_shopify(raw_body: bytes, header_value: str, client_secret: str) -> bool:
    digest = hmac.new(
        client_secret.encode("utf-8"),
        raw_body,
        hashlib.sha256,
    ).digest()
    expected = base64.b64encode(digest).decode("utf-8")
    return hmac.compare_digest(expected, header_value)

Worth knowing before you spend an afternoon on it: after rotating an app’s client secret, Shopify notes it can take up to an hour before signatures are generated with the new one. If verification starts failing right after a rotation, that’s usually why.

And because delivery isn’t guaranteed, webhooks alone are never enough. Run a reconciliation job that pulls orders and customers modified since the last watermark and fills the gaps. This is the job that saves you during an outage on your side, and it’s the one people skip. If you’d rather not build the buffering and replay layer, managed webhook infrastructure like Hookdeck exists for exactly this, and self-hosting a small queue and worker on a modest VPS from somewhere like Contabo or InterServer is perfectly reasonable too. The point is that something has to catch what the webhook missed.

Failure family three: “revenue” means four different things

This is the one that turns into a meeting. Finance says the number is wrong. Analytics says the number is right. Both are correct, because they’re measuring different things and nobody wrote down which.

• Shopify reports gross sales at the moment the order is placed, in the store’s currency and the store’s timezone.
• Stripe pays out in batches, net of processing fees, refunds and disputes, on its own schedule, in the settlement currency of the account.
• A refund issued in Shopify and a refund issued in Stripe are two different events that may or may not be the same underlying money movement.
• A dispute lands days or weeks after the order and reverses revenue in a period that has already been reported.

There’s a structural trap here too. If the store runs Shopify Payments, there is no Stripe account for you to query. Shopify Payments is built on Stripe’s processing infrastructure, but it’s Shopify’s gateway, and payouts, fees and disputes live in the Shopify admin rather than behind a Stripe API key. Plenty of businesses end up with Shopify Payments handling storefront orders and a separate Stripe account handling subscriptions or invoicing, which means two payment systems, two payout schedules and one customer who has no idea any of this exists.

Do not build a single revenue column. Build separate fact tables at their natural grain (orders, order line items, payments, refunds, disputes, payouts) plus a bridging table that links a payout back to the transactions inside it. Then let each team define its own metric on top. Finance gets net settled cash. Marketing gets gross booked revenue. Neither has to be wrong for the other to be right.

Store every timestamp in UTC and keep the original timezone as a separate column. Store every amount in minor units as an integer alongside its currency code, and store the FX rate that was actually applied rather than recalculating it later from a rates table. Recalculated FX is the single most common source of “the number moved and nobody changed anything.”

On ingestion, Stripe Data Pipeline can push Stripe data directly into Snowflake, Redshift or Databricks as a data share, which removes a connector from the picture if you’re already on one of those. Otherwise a managed ELT tool like Fivetran or a self-hosted Airbyte will handle Shopify, Stripe, the CRM and the helpdesk with off-the-shelf connectors. Both routes are fine. The connector is genuinely the easy part.

Failure family four: deletion is the requirement nobody models

If you’re building this as a Shopify app, three compliance webhooks are mandatory: customers/data_request, customers/redact and shop/redact. You acknowledge with a 200 and complete the action within thirty days. shop/redact fires forty-eight hours after uninstall and covers the shop’s data. These are enforced at app review, so you’ll find out about them either way.

The interesting problem isn’t receiving the webhook. It’s that by the time it arrives, that person’s email exists in your raw landing zone, your staging tables, your identity table, your customer view, a BI extract, the CRM, the helpdesk, and a Slack message from a debugging session six months ago. A delete that only touches the golden record is not a delete.

The design decision that makes this tractable is to hold personal data in exactly one place. Every downstream table carries person_key and nothing else. Facts, aggregates, models, extracts and syncs all reference the key. When a redaction request lands, you redact one row set in the identifier table and the rest of the warehouse is already anonymous. Getting this right at schema-design time costs an afternoon; retrofitting it costs a quarter.

Practical pieces that make it real:

• Tag every table and column that can contain an identifier, in your catalog or in dbt metadata, and treat the tag list as the deletion scope.
• Partition the raw landing zone so you can rewrite a partition rather than scan an entire bucket looking for one email.
• Keep a deletion queue with a status per downstream system, including the CRM and the helpdesk, which have their own APIs and their own idea of what deletion means.
• Write down what you retain and why. Financial records generally have to survive a deletion request for tax and dispute reasons. Say so rather than promising a hard delete you can’t perform.

Failure family five: pushing the unified view back out

A customer view nobody can act on is a report. The value shows up when a support agent opens a ticket and can see subscription status and lifetime order count without switching tabs.

That’s reverse ETL, and the market is well served: Hightouch, Census (now part of Fivetran), Polytomic, RudderStack, and iPaaS-style tools like Workato if you need branching logic rather than field syncs. The warehouse-native argument is straightforward and mostly correct: your models already live there, and you avoid maintaining a second copy of the customer profile inside a vendor’s rigid schema. The counter-argument is also real. Reverse ETL is batch by nature, so if you need sub-minute reaction to an event, a packaged CDP or a direct event pipeline will serve you better, and you’ll be running several vendors plus warehouse compute plus the people to maintain the models.

Three things go wrong here more or less every time.

The write loop. You compute a churn-risk field and sync it into the CRM. Your ELT connector reads the CRM back into the warehouse. Now the computed field is an input to the model that computes it. This is genuinely hard to spot because nothing errors, the value just drifts. Namespace every synced field with a consistent prefix and explicitly exclude that namespace from ingestion.

The resync stampede. Incremental syncs push diffs and stay small. Change a model definition and the next run wants to update every record, which will meet a helpdesk API rate limit at speed. Cap the batch size, and treat a full resync as a planned operation rather than something a merge to main can trigger.

Syncing logic instead of keys. Push person_key into a custom field on the CRM contact and the helpdesk user. It makes the key searchable by agents, it gives the next pipeline a deterministic join, and it stops three teams from each rebuilding their own version of your matching rules.

A pipeline shape that holds up

Put together, the order of operations matters more than the tool choice:

1. Land raw. Every webhook body and every API page goes to immutable storage first, keyed by source and event ID, before anything interprets it.
2. Deduplicate on the provider’s event ID, atomically, with a retention window longer than the provider’s retry window.
3. Normalize identifiers into the identifier table as rows arrive, with the source recorded.
4. Resolve identity deterministically into person_key, as a scheduled job, never inside a request handler.
5. Build fact tables per system at their own grain, keyed on person_key, with no raw PII.
6. Model the customer view on top, in dbt or whatever your team already uses. The Fivetran dbt_customer360 package is a reasonable starting point if you want to see how someone else structured the identity step.
7. Activate a small, named, namespaced set of fields back into the CRM and helpdesk.
8. Run reconciliation and freshness checks on a schedule, and alert on them like you would any other production job.

Troubleshooting: symptom to root cause

These are the ones that come up repeatedly, and the check that usually settles them.

• Warehouse customer count is much higher than Shopify’s. Guest orders are minting a new person on every purchase. Check how many person rows have exactly one identifier and one order.
• Warehouse count is much lower. An over-eager merge rule. Look for person keys with an implausible number of distinct emails or shipping addresses attached.
• BI revenue doesn’t match Stripe payouts. Grain mismatch, not a data loss. Reconcile a single payout against its constituent transactions before touching anything else.
• A CRM field keeps flipping between two values. Write loop. Check whether the field’s namespace is excluded from your ELT source configuration.
• Recent orders missing, older ones fine. Webhook delivery gap. Confirm the reconciliation job ran, and check its watermark rather than trusting its exit code.
• Every Shopify signature suddenly fails. Either a body parser is running before verification, or the client secret was rotated within the last hour.
• Tickets attached to the wrong customer after a merge. The merge happened in the warehouse but the synced key in the helpdesk still points at the old person. Merges need to propagate outward, not just inward.

Common mistakes

• Joining on email in the model layer instead of resolving identity once, upstream.
• Treating webhooks as the system of record instead of as a low-latency hint, with a reconciliation pull behind them.
• Doing identity resolution, enrichment or CRM writes inside the webhook handler.
• Ordering profile updates by arrival time rather than by the event’s own timestamp.
• One revenue column that finance and marketing are both told to use.
• Storing amounts as floats, or recalculating historical FX from a rates table.
• Copying PII into every downstream table because it was convenient at the time.
• Reaching for probabilistic matching before the deterministic keys have been exhausted.
• Assuming a Shopify store on Shopify Payments has a Stripe account you can query.

Best practices worth the effort

• Own the surrogate key. Never let a vendor’s internal ID become the primary identity of a person in your systems.
• Write your person_key into Stripe metadata and Shopify customer metafields at creation time, so future joins are exact rather than inferred.
• Keep raw payloads immutable and replayable. Every model bug you’ll ever have is cheaper to fix if you can reprocess from raw.
• Version your matching rules and record which version produced each merge, so you can audit and reverse a bad rule.
• Alert on freshness and on row-count deltas per source, not just on job success. A connector returning zero rows exits cleanly.
• Reconcile one payout end to end by hand before you trust the automated version. It’s the fastest way to find a grain error.
• Document the metric definitions next to the models. Most “the data is wrong” tickets are definition disputes.

FAQ

Do I need a CDP to unify Shopify, Stripe, CRM and support data?

Not necessarily. If you already run a warehouse and have someone comfortable with dbt, ELT plus modelling plus reverse ETL covers the same ground with more control over the data model. A packaged CDP earns its cost when you need real-time behavioural segmentation, when nobody on the team can maintain models, or when marketing needs to ship changes without waiting on data engineering.

How do I handle guest checkout customers?

Create a person from the order’s email anyway, and mark the record as identity-weak. When that email later appears attached to a real Shopify customer or Stripe Customer object, the deterministic match links them and the order history follows. The mistake is discarding guest orders because they have no customer ID, which is exactly the revenue you most want attributed.

Should I sync everything into the CRM or keep it in the warehouse?

Keep the full picture in the warehouse and sync a deliberately small set of fields outward. The test is whether an agent or a rep will act on the field during a conversation. Lifetime value, subscription status, open ticket count and churn risk pass. A hundred-column profile does not, and it turns every model change into a rate-limit incident.

Why don’t my Shopify sales match my Stripe payouts?

Because they’re measuring different things at different times. Shopify shows gross sales at order time; Stripe pays out batched amounts net of fees, refunds and disputes, and the timing differs by days. Add currency conversion and a timezone mismatch and the numbers will never tie without an explicit bridging table linking payouts to the transactions inside them.

Can I use email as the primary key if my store requires account creation?

You can get away with it longer, but it’s still a bad primary key. People change email addresses, use one address for orders and another for support, and share addresses within a household. A surrogate key with an identifier table costs very little up front and means an email change is a new row rather than a broken join and a duplicate customer.

How do I test the identity resolution before trusting it?

Run it in shadow mode. Compute person keys without activating them, then have support pick twenty customers they know are duplicates and twenty they know are distinct, and check what the resolver did. That exercise finds normalization bugs and over-eager merge rules faster than any aggregate metric, because the people who answer the tickets already know where the duplicates are.

What’s the minimum viable version of this?

Raw landing with deduplication, an identifier table with deterministic matching, one fact table per source, and a single synced field into the helpdesk. That’s a couple of weeks of work and it removes most of the pain. Everything above it is refinement.

Conclusion: unify Shopify, Stripe, CRM and support data around a key you own

If you take one thing from this: the hard part of unifying Shopify, Stripe, CRM and support data is not moving the data. Connectors are commodity. The hard part is deciding what a customer is, writing that decision down as a key you own, and then refusing to let any downstream system quietly invent its own version of it.

Do the identity work first, land everything raw so you can reprocess when you get it wrong, and keep personal data in one place so deletion stays a single operation. The dashboards will follow. Skip that order and you get exactly the screenshot this post opened with, eighteen months later and with a lot more rows behind it.

Need help wiring this together?

I work on exactly this kind of plumbing: the unglamorous layer between an ecommerce platform, a payment processor and the tools your team actually opens all day. Things I can help with here:

• Designing the identity model (surrogate keys, identifier tables, deterministic matching rules) and auditing an existing one for over-merging.
• Building idempotent, signature-verified webhook intake for Shopify and Stripe, with a dead-letter path and a reconciliation job behind it.
• Setting up ELT into Snowflake, BigQuery, Redshift or Postgres, whether with Fivetran, self-hosted Airbyte, or Stripe Data Pipeline where it fits.
• Modelling orders, payments, refunds, disputes and payouts so finance and marketing can each get a number they trust.
• Reverse ETL back into the CRM and helpdesk without write loops, rate-limit incidents or a hundred junk fields.
• Making deletion and data-subject requests actually work across the warehouse, the raw landing zone and every downstream copy.

If you’re stuck on a specific piece, send me the concrete thing: a webhook payload, a schema dump, the query that disagrees with your Stripe payout report. It’s much easier to say something useful about a real artifact than about a diagram.

The post One Customer, Four Systems: Unifying Shopify, Stripe, CRM and Support Data appeared first on John Nessime.

So you open the orchestrator. Every task is green. The run started on schedule, finished inside its usual window, exit code zero across the board. Nothing paged. And the numbers are still wrong.

That gap is the whole reason to build a data pipeline operations dashboard. Not to watch DAGs go green, which the orchestrator UI already does perfectly well, but to answer the questions the orchestrator structurally cannot answer. This post covers the four signals worth putting on a screen, where each one actually comes from, how to wire them into Prometheus and Grafana without inventing a metrics platform, and how to turn the panels into alerts that fire before a human notices.

Why green pipelines still ship bad data

An orchestrator tracks process outcomes. Did the Python exit cleanly, did the SQL statement return without an error, did the container terminate zero. That is a real signal and you should keep it. It is just answering a different question than the one your users care about.

Consider what a perfectly successful run looks like when the upstream source has quietly changed:

• An API starts paginating differently and your extractor pulls the first page only. It ran fine. It loaded 500 rows instead of 500,000.
• A source column gets renamed and your transform selects it with a coalesce default. It ran fine. Every row now says “unknown”.
• A partition arrives late, so the incremental filter finds nothing to process. It ran fine. It did nothing.
• An upstream team backfills, and your dedupe key no longer holds. It ran fine. Everything is doubled.

None of those raise an exception. All four are visible in under a second if you are tracking row counts and freshness at the table level. This is the failure family that eats the most incident time, because the clock does not start when the pipeline breaks. It starts when a human notices, which can be days later, and by then the bad data has been copied into reports, cached in a BI extract, and quoted in a meeting.

The four questions a data pipeline operations dashboard has to answer

Resist the instinct to start from what your tools can emit. Start from the questions you ask during an incident, in the order you ask them. Everything that does not answer one of these is a second-page panel.

Did it run at all?

Sounds trivial, and it is the one most homegrown dashboards get wrong. A panel showing “last run: success” does not distinguish between a pipeline that succeeded ten minutes ago and one that succeeded eleven days ago and has not been scheduled since. You want time since last successful completion, as a number that grows, not a status pill that sits there looking calm.

Did it finish in time?

Duration matters less than duration against a deadline. A load that takes forty minutes is fine if the report is read at 9am and the run starts at 6am. The same forty minutes is an incident if someone moved the schedule to 8:40. Put the deadline on the panel as a threshold line, not in a runbook. Trending duration also catches slow decay: a stage that creeps from five minutes to fifty usually means data growth, a bad query plan, or resource contention, and it will breach your window eventually.

Did the right amount of data arrive?

Rows in versus rows out, per stage. This is the highest value panel per unit of effort on the entire dashboard. A drop from a million rows to twenty thousand is not subtle once you plot it, but it is completely invisible in an exit code. Compare against the same weekday from the previous week rather than against yesterday, because most business data has a weekly shape and Monday-versus-Sunday comparisons generate noise you will learn to ignore.

Is the data still the shape you expect?

Null rates, distinct counts on key columns, and schema changes. This is the panel almost nobody builds, and it is where the expensive failures live. A gradual rise in null rate on a joining key means an upstream system is degrading and your joins are silently dropping rows. You do not need statistical anomaly detection to catch that. You need the number plotted over time next to the others.

There is a fifth question that belongs on a different surface: what breaks downstream when this table is wrong. That is lineage, and it answers blast radius rather than health. Keep it one click away, not on the operations screen.

Where each signal actually comes from

Three sources, three different collection mechanisms. This is the part that takes an afternoon rather than a sprint, provided you do not try to build a unified agent.

The orchestrator, for run and timing signals

Airflow has emitted StatsD metrics for a long time, and that path is still the one most deployments run. Airflow speaks StatsD, so you put a statsd_exporter in front of it to translate into Prometheus format. Enable it in the metrics section of your config, or with the equivalent AIRFLOW__METRICS__ environment variables if you are on containers:

[metrics]
statsd_on = True
statsd_host = statsd-exporter
statsd_port = 9125
statsd_prefix = airflow
metrics_allow_list = scheduler,executor,dagrun,pool,triggerer

Two things worth knowing before you roll this out. First, metrics_allow_list is not optional at any real scale. Airflow emits per-DAG and per-task metric names, so without a prefix filter your cardinality grows with every DAG anyone adds, and you find out when the time series database starts struggling. Second, Airflow expects the StatsD destination to be resolvable at startup, so a scheduler that boots before the exporter can fail to come up. Order your dependencies accordingly.

StatsD has no concept of labels, so the exporter has to reconstruct them from the dotted metric name. That mapping file is the fiddly part:

mappings:
  - match: "*.dag.*.*.duration"
    match_metric_type: observer
    name: "airflow_task_duration"
    labels:
      airflow_id: "$1"
      dag_id: "$2"
      task_id: "$3"

The wildcards map positionally onto $1, $2, $3, and match_metric_type has to match what Airflow actually sends, timer versus counter versus gauge. Get that wrong and the metric silently never appears at the exporter, which is a genuinely annoying half hour of debugging. Check the exporter’s own metrics endpoint before you go looking at Prometheus, because if it is not there it never left Airflow.

Airflow also has a native OpenTelemetry path for both metrics and traces, which skips the StatsD translation entirely and gives you real labels rather than reconstructed ones:

[metrics]
otel_on = True
otel_host = otel-collector
otel_port = 4318
otel_prefix = airflow
otel_interval_milliseconds = 30000

The port there is the collector’s OTLP HTTP receiver, and you need an OpenTelemetry Collector or compatible endpoint in the path. If you are standing up a new stack, this is the one I would reach for, because the label problem disappears and traces give you per-task spans under a DAG run parent span, which is worth a lot during a slow-pipeline investigation. If you already have working StatsD mappings and dashboards built on them, migrating is real work for a modest gain, and the honest answer is that it can wait. Metric names do shift between Airflow major versions either way, so pin your dashboard queries to the metrics reference for the version you actually run.

Managed Airflow changes the plumbing but not the shape. On Amazon MWAA the metrics land in CloudWatch instead, and Google Cloud Composer and Astronomer each have their own export path. The four questions do not change.

The transformation layer, for data-level signals

If you use dbt, most of what you need is already being written to disk and thrown away. Every invocation produces run_results.json in the target directory, containing per-node status and timing, and dbt source freshness produces its own sources.json. The manifest.json holds the dependency graph, which is your lineage.

The cheapest useful thing you can do is parse the failures out at the end of the run:

jq -r '.results[]
  | select(.status != "success")
  | [.unique_id, .status, (.execution_time | tostring)]
  | @tsv' target/run_results.json

One trap if you run dbt in ephemeral Kubernetes pods: the target directory dies with the pod. Upload the artifacts to object storage as the last step inside the same task that ran dbt, not as a downstream task, or you will find your history is empty exactly when you want it.

Elementary is the low-friction option here. It installs as a dbt package with an on-run-end hook that loads the artifacts into your warehouse as ordinary tables, which means your dashboard can query test results and model timings in SQL alongside everything else. Great Expectations and Soda cover the same validation ground with more expressive checks and more setup. The commercial platforms, Monte Carlo and the vendors around it, add automated anomaly detection and column-level lineage across the whole stack, and the genuine case for them is that they find the checks you did not think to write. The trade-off is that you are paying per monitored table for statistical inference, and on a stack of twenty tables that inference is not telling you much that a row-count panel and a null-rate panel would not.

Without dbt, you write the check yourself and it is still short. One query per critical table, run on a schedule:

SELECT
  COUNT(*)                                        AS row_count,
  MAX(updated_at)                                 AS last_event,
  SUM(CASE WHEN customer_id IS NULL THEN 1 ELSE 0 END) AS null_keys
FROM analytics.orders
WHERE loaded_at >= CURRENT_DATE - 1;

Batch jobs, and the Pushgateway caveat

Prometheus pulls. Batch jobs finish and vanish long before a scrape interval comes round, so there is nothing to pull from. The Pushgateway exists for exactly this: short-lived service-level jobs push their results to it, and Prometheus scrapes the gateway instead.

cat <<EOF | curl --data-binary @- 
  http://pushgateway:9091/metrics/job/orders_sync
# TYPE pipeline_last_success_timestamp_seconds gauge
pipeline_last_success_timestamp_seconds $(date +%s)
# TYPE pipeline_rows_loaded gauge
pipeline_rows_loaded ${ROWS}
EOF

Now the caveat, which is the single most common way a pipeline dashboard lies to you. The Pushgateway is a cache, not an aggregator, and it never forgets. Push a success metric, then have the job stop running entirely, and that success sits there being scraped forever. Your dashboard stays green because the last known value is green, and the fact that nothing has updated it in six days is not something a status panel can see.

Two defences, and you want both. First, always push a timestamp gauge rather than a boolean status, so staleness is arithmetic rather than inference. Second, alert on absence as well as on value, because the Pushgateway also exposes push_time_seconds per group, which tells you when anything last arrived regardless of what it said. Prometheus’s usual safety net, the up metric going to zero when a target disappears, does not apply here: Prometheus is scraping the gateway successfully, and the gateway is happily serving stale data.

Also keep instance identifiers out of the grouping labels. Push per service, not per worker, or you accumulate a permanent time series for every container that ever ran.

Laying out the dashboard so it works during an incident

The layout constraint is that someone is reading this at 7am on a phone with a stakeholder waiting. Three rows, top to bottom, in the order you need them:

1. Freshness across everything. One panel, one row per critical table, showing time since last successful load against its threshold. If this row is clean, the pipelines are fine and the problem is elsewhere. That is a genuinely useful thing to establish in five seconds.
2. Volume and duration per pipeline. Row counts with a week-over-week comparison, and stage duration with the deadline drawn on. This is where you localise a problem to a specific pipeline and stage.
3. Data quality detail. Null rates, failed test counts, schema change events. This is where you work out what actually went wrong once you know where.

The freshness query in PromQL, given the timestamp gauge above:

time() - max by (job) (pipeline_last_success_timestamp_seconds)

Grafana renders that nicely as a table with per-row thresholds, which is what makes the top row scannable. If you are self-hosting, Prometheus plus Grafana on a modest VPS from Hetzner, InterServer or DigitalOcean handles a surprising amount of pipeline metrics, because pipeline telemetry is low-volume compared to application metrics. Grafana Cloud’s free tier is a reasonable alternative if you do not want to own the storage, and Datadog or New Relic make sense if your organisation is already paying for one of them and you would rather have one pane than a better one.

Keep it to one screen. Every panel you add past that dilutes the ones that matter, and a dashboard nobody can read at a glance is a dashboard nobody opens.

Turning panels into alerts

A dashboard that requires someone to look at it is a dashboard that finds problems the morning after. The freshness panel converts into an alert almost directly:

groups:
  - name: pipelines
    rules:
      - alert: PipelineStale
        expr: time() - max by (job) (pipeline_last_success_timestamp_seconds) > 5400
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.job }} has not completed successfully in over 90 minutes"

      - alert: PipelineMetricsMissing
        expr: absent(pipeline_last_success_timestamp_seconds{job="orders_sync"})
        for: 30m
        labels:
          severity: page
        annotations:
          summary: "No metrics received from orders_sync"

The first catches a pipeline that is late or broken. The second catches the one that stopped reporting entirely, which the first cannot see, because an expression over a series that no longer exists returns nothing and an empty result does not fire. That pairing is the whole trick, and it applies to every push-based metric you collect.

Set the threshold from the deadline the data has to meet, not from how long the job usually takes. Ninety minutes is not “the run takes an hour so give it some room”. It should be the point past which someone downstream is materially affected.

Troubleshooting a dashboard that shows nothing useful

The metric never appears in Prometheus. Work backwards along the path, one hop at a time. Curl the statsd_exporter’s own metrics endpoint first. If the metric is not there, it never left the orchestrator, so the problem is the config or an allow-list that filtered it out. If it is there but not in Prometheus, the problem is your scrape config. Do not start in Grafana.

The metric appears with no labels, or with the DAG name baked into the metric name. Your statsd_exporter mapping did not match. Check the wildcard count against the actual dotted name, and check match_metric_type, which is the usual culprit.

Everything is green but the data is old. Classic Pushgateway staleness. Query push_time_seconds for the affected group and compare it to now. If it has not moved, the job is not running and you are looking at a cached value.

Volume alerts fire every Monday. You are comparing against the previous day on data with a weekly cycle. Compare against the same day last week using an offset, or exclude weekends from the baseline.

Gaps in the duration graph. If the metric is a timer sent only on completion, a run that is still going or that failed early sends nothing. That is correct behaviour, and it is why liveness needs its own panel rather than being inferred from the duration series.

Prometheus memory climbing after rollout. Cardinality. Per-DAG and per-task metrics multiply, and any label carrying a run identifier, timestamp or UUID creates a permanent new series. Tighten the allow list and drop the offending labels.

Common mistakes

• Treating job success as data correctness. This is the entire problem, and every other mistake is a variation of it.
• Pushing a boolean status instead of a timestamp, so staleness becomes invisible.
• Alerting on threshold breaches but never on metric absence.
• Building the dashboard before agreeing what “healthy” means for each table with the people who consume it.
• Instrumenting every pipeline at once instead of the three that matter, then abandoning the project when the noise gets unmanageable.
• Putting instance or pod identifiers into Pushgateway grouping labels and accumulating dead series forever.
• Alerting on things nobody will act on at 3am. An alert that gets acknowledged and ignored twice has trained your team to ignore it permanently.

Best practices

• Baseline before you threshold. Collect for two or three weeks and look at the actual shape of the data before deciding what abnormal means.
• Start with freshness on your three most critical tables. It catches more real problems per line of code than anything else you can instrument.
• Define freshness thresholds from the consumer’s deadline, and write the consumer’s name next to the threshold so the number has an owner.
• Version dashboards and alert rules in Git alongside the pipelines. A dashboard edited live in the UI is a dashboard that will be silently different next quarter.
• Emit row counts at stage boundaries as a habit, not as a special case. In versus out at each hop is what turns “the numbers are wrong” into “stage three drops rows”.
• Review alerts monthly and delete the ones that never fired usefully. Alert rules are code and they rot the same way.
• Keep lineage accessible but off the main screen. It answers a different question and it competes for attention with the ones you need first.

Frequently asked questions

What is the difference between pipeline monitoring and data observability?

Monitoring watches the process: did the job run, did it succeed, how long did it take. Observability watches the data the process produced: is it fresh, is there the right amount, has its shape changed. You need both, and the second is the one most teams are missing when they say their dashboards did not catch something.

Do I need a commercial data observability platform?

Not to start. Freshness, volume and null-rate checks on your critical tables cover most real incidents and cost an afternoon. Commercial platforms earn their keep when you have hundreds of tables, several teams changing them independently, and nobody with time to hand-write checks. The honest signal that you have outgrown the homegrown version is when maintaining the checks becomes its own recurring task.

Should I use StatsD or OpenTelemetry for Airflow metrics?

New stack, OpenTelemetry: you get real labels instead of reconstructing them from dotted names, and traces come along with it. Existing stack with working StatsD mappings, stay put until you have another reason to touch it. Metric coverage and naming differ between the two paths and between Airflow versions, so verify against the metrics reference for your version rather than copying a dashboard from a blog post.

How do I monitor a pipeline that only runs weekly?

Same timestamp gauge, much larger threshold, and lean harder on the absence alert. Infrequent pipelines are where Pushgateway staleness does the most damage, because the last successful push can sit there for a fortnight looking entirely healthy. Check push_time_seconds for those groups specifically.

What should a data pipeline operations dashboard show first?

Time since last successful load, per critical table, with its threshold visible. That one panel answers the question you will be asked most often, and when it is clean it rules out the entire pipeline layer in a glance.

How many alerts is the right number?

Few enough that every page gets investigated. If someone can name an alert they routinely dismiss without looking, you have too many, and the cost is not the noise itself but the habit it builds. Start with freshness and absence per critical pipeline, add volume, and only go further when a real incident shows you a gap.

Can I build this without an orchestrator like Airflow?

Yes. Cron jobs, Step Functions, GitHub Actions and shell scripts can all push a timestamp and a row count to a Pushgateway in three lines. The orchestrator gives you the run and timing signals for free, but it is the data-level signals that do the heavy lifting, and those come from querying the warehouse regardless of what triggered the load.

The one thing to take away

A data pipeline operations dashboard is not a nicer view of your orchestrator. It exists because the orchestrator can only tell you that a process finished, and the question you are actually being asked is whether the data is right.

If you build one panel this week, build time since last successful load, sourced from a timestamp gauge, alerted on both threshold and absence. It costs an afternoon, it catches the failure mode that hides longest, and it turns “is the dashboard broken or just late?” into a question you can answer before anyone has to ask it.

Need help building or fixing this?

I work with teams on pipeline observability and the plumbing underneath it. Things I regularly help with:

• Designing freshness, volume and quality checks for your critical tables, with thresholds derived from real consumer deadlines rather than guesses
• Wiring Airflow into Prometheus and Grafana, including the statsd_exporter mappings and the cardinality controls that keep it affordable
• Migrating an existing StatsD setup to the OpenTelemetry path without losing your dashboard history
• Getting dbt artifacts out of ephemeral pods and into somewhere you can query them, with Elementary or with plain object storage
• Auditing an existing dashboard and alert set to find what is stale, what will never fire, and what is quietly training your team to ignore pages
• Standing up a self-hosted Prometheus and Grafana stack on a VPS, or sizing a managed alternative if owning the storage is not worth it to you

If you have a dashboard that went green through an incident, send me the panel queries and the alert rules and I will tell you where the gap is.

The post Building a Data Pipeline Operations Dashboard That Catches Silent Failures appeared first on John Nessime.