The uncomfortable answer is that people did know. The superintendent knew the drywall crew had been short for two weeks. The project engineer knew a structural RFI had been sitting with the design team well past its due date. The foreman knew the weekly commitments kept slipping. None of it was hidden. It just never reached one screen, in one place, while there was still float left to spend.

This post walks through building a construction early warning dashboard that catches those signals while they are still cheap to act on. It covers why schedule delay shows up late, the one data decision that makes early warning possible, the signals worth tracking, how to pull data out of Primavera P6, Microsoft Project and Procore, and how to alert without training everyone to ignore the alerts.

Why Construction Delay Shows Up Three Weeks Late

Delay rarely arrives as one event. It accumulates in small amounts across activities that do not look critical yet. Three things keep it invisible.

1. The schedule is updated in batches. If the CPM schedule is statused monthly, or even biweekly, the finish date is only recalculated at that cadence. Everything that happened in between lands at once.
2. Progress is self-reported. Percent complete is an estimate from whoever statuses the activity. Optimism is not dishonesty, but it compounds. An activity that sits at a comfortable-looking percentage for three updates is telling you something.
3. The data is overwritten in place. This is the one that bites hardest. Most schedule tools, and most dashboards built on top of them, show the current value of float, the current forecast finish and the current status. Last month’s values are gone unless someone saved the file.

That third point is why so many project dashboards look fine right up until they don’t. A current-state dashboard tells you where you are. Delay warnings live in the rate of change. An activity with twelve days of float looks healthy. The same activity with twelve days of float, down from thirty over the last two updates, is the next critical path.

The Design Decision That Makes Early Warning Possible: Snapshot Everything

Before any chart or alert, decide that every schedule update, every RFI log pull and every weekly plan gets stored as a dated snapshot. You append. You never overwrite. Storage for this is trivial; a few hundred activities per update is nothing for any database.

PostgreSQL is what I reach for first here. It runs happily on a small managed instance like Amazon RDS or DigitalOcean Managed Databases, or on a modest VPS from Hetzner, Contabo or Interserver if you are comfortable handling backups yourself. A minimal activity snapshot table looks like this:

CREATE TABLE activity_snapshot (
    project_code     text        NOT NULL,
    data_date        date        NOT NULL,
    task_code        text        NOT NULL,
    task_name        text,
    total_float_days numeric,
    early_start      date,
    early_finish     date,
    is_complete      boolean     NOT NULL DEFAULT false,
    loaded_at        timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (project_code, data_date, task_code)
);

The primary key does the important work. Each activity appears once per data date, so loading the same export twice fails loudly instead of silently doubling the history. The data_date is the schedule’s status date, not the day you loaded the file, because trends must be plotted against when the schedule says it was true.

Keying on task_code, the activity ID the planner sees, rather than an internal database ID matters too. Internal IDs can change when a schedule is re-imported. Activity IDs usually survive, and when they don’t, you want to know (more on that in troubleshooting).

Early Warning Signals That Move Before the Finish Date Does

Each of these is a leading indicator of schedule delay. None is perfect alone. Together they give you weeks of notice instead of a surprise in the monthly update.

Float Erosion on Near-Critical Paths

Watching the critical path is necessary but late. By the time an activity is critical, its float is already gone. The useful signal is near-critical work losing float quickly. This query compares each open activity to its own previous snapshot:

WITH history AS (
    SELECT
        project_code,
        task_code,
        task_name,
        data_date,
        total_float_days,
        LAG(total_float_days) OVER (
            PARTITION BY project_code, task_code
            ORDER BY data_date
        ) AS prev_float_days
    FROM activity_snapshot
    WHERE NOT is_complete
)
SELECT
    project_code,
    task_code,
    task_name,
    prev_float_days,
    total_float_days,
    prev_float_days - total_float_days AS float_lost_days
FROM history
WHERE data_date = (SELECT max(data_date) FROM activity_snapshot)
  AND prev_float_days IS NOT NULL
  AND total_float_days <= 20
  AND prev_float_days - total_float_days >= 5
ORDER BY float_lost_days DESC;

The thresholds (twenty days remaining, five days lost) are starting points, not rules. Tune them to your update cadence and project length. What matters is that the query looks at movement, which a current-state report cannot.

Constraint Aging: RFIs and Submittals Against Need-By Dates

An overdue RFI on its own is noise. Every project has them. An overdue RFI that blocks an activity starting in ten days is a delay you can see coming.

The hard part is the link. RFI and submittal logs in Procore or Autodesk Construction Cloud rarely point at schedule activity IDs. Someone has to maintain a small mapping table from RFI or submittal to activity, or at least to area and spec section. I would not skip it. Without the link you get a list of late paperwork; with it you get “this answer is needed in eight days and the ball has been in the architect’s court for three weeks.”

The signal to chart is days of slack between the expected response and the need-by date of the linked activity. When it goes negative, the RFI is now on the schedule’s path whether the schedule knows it or not.

Commitment Reliability: PPC and Reasons for Variance

If the field runs the Last Planner System, you already have a strong signal. Percent Plan Complete (PPC) is the share of weekly commitments finished as promised. Each week’s PPC is a lagging number, but the trend across weeks leads the CPM schedule, because missed promises show up in the field long before they show up as recalculated dates.

Store reasons for variance as categories, not free text. A PPC dip caused by weather tells you something different from three straight weeks of “material not on site.” The second one is a procurement problem that will keep producing delay until someone fixes it.

Look-Ahead Churn

Count how many times an activity’s planned start has moved across snapshots of the three- or six-week look-ahead. An activity pushed out once is normal planning. One that has been pushed four updates running is being deferred, not planned, and the reason is usually a constraint nobody has named yet.

Earned Value, Read Carefully

If you track earned value, the Schedule Performance Index (SPI = EV / PV) belongs on the dashboard, with a caveat many dashboards skip. Classic SPI drifts back toward 1.0 as the project nears completion, even when the job is late, because earned value and planned value both end at the budget at completion. A late-stage SPI that looks like recovery may just be arithmetic.

Earned Schedule fixes this by measuring in time rather than money. It asks at what point in the baseline your current earned value was supposed to be reached, then divides that by the actual time elapsed to get SPI(t). It needs no new data collection, only different math on the same numbers. If you only plot one schedule index late in a job, plot SPI(t).

Claimed Progress Versus Manpower

Daily logs record who was on site. When an activity’s reported percent complete keeps rising while the manpower logged against that trade stays well below the plan, one of those numbers is wrong. Usually it is the percent complete. This comparison is crude, and it needs a consistent trade mapping, but it is one of the few checks that uses data nobody is trying to make look good.

Getting the Data Out of P6, Microsoft Project and Procore

A construction project dashboard lives or dies on boring extraction work. Plan for it.

Primavera P6 XER Exports

An XER file is plain tab-delimited text. Lines beginning with %T name a table, %F lists its fields, and %R lines are rows. Activities live in the TASK table. This reader pulls one table into a list of dictionaries:

def read_xer_table(path, table_name):
    rows, fields, current = [], None, None
    # XER exports often use a Windows code page, not UTF-8
    with open(path, encoding="cp1252", errors="replace") as fh:
        for line in fh:
            parts = line.rstrip("rn").split("t")
            tag = parts[0]
            if tag == "%T":
                current = parts[1]
            elif tag == "%F" and current == table_name:
                fields = parts[1:]
            elif tag == "%R" and current == table_name:
                rows.append(dict(zip(fields, parts[1:])))
    return rows

tasks = read_xer_table("update.xer", "TASK")
for t in tasks[:5]:
    print(t["task_code"], t["task_name"], t["total_float_hr_cnt"])

Two details catch people. Float and durations in XER are stored in hours (total_float_hr_cnt), so converting to days means dividing by the hours per day of each activity’s calendar, not a blanket eight. And an activity with a value in act_end_date has an actual finish, which is a clean way to set is_complete without relying on status labels.

Use the float P6 stored in the export rather than recomputing the critical path yourself. P6’s result reflects its scheduling options, constraints and calendars. Your recompute probably won’t.

Microsoft Project

Microsoft Project can save a schedule as XML, which carries tasks, dates and total slack. The same snapshot logic applies. Save one file per status date, load it with its data date, and never replace the previous one.

Procore and Other Project Management Platforms

Procore’s REST API exposes RFIs and submittals per project, including status, due dates and ball-in-court. Two warnings. Most useful endpoints are scoped to a single project, so a portfolio view means iterating over the project list. And submittal status values can be configured per company, so map them explicitly rather than hard-coding what you saw on one account. Autodesk Construction Cloud has its own RFI API with a similar shape.

The Spreadsheet Nobody Wants to Admit Exists

The weekly work plan is very often an Excel file or a Smartsheet. Accept that. A scheduled job that reads a fixed-layout sheet from a shared folder is less elegant than an API and far more likely to get used. Fighting the field’s tools is how early warning systems die.

Building the Dashboard and Alerts People Don’t Mute

For visualization I lean toward Grafana, self-hosted or on Grafana Cloud, because its alert rules can run SQL directly against PostgreSQL and route results to email, Slack or Microsoft Teams. You can see the style of dashboard in this live Grafana demo.

Power BI deserves a fair hearing. Many contractors already license it, project controls teams know it, and its reporting and drill-down are excellent for monthly reviews. Where it is weaker is rule-based alerting on arbitrary queries. If your organization lives in Microsoft 365, a sensible split is Power BI for the reporting pack and a small scheduled job or Grafana instance for the alerts. Metabase is another reasonable middle ground for smaller teams.

Whatever the tool, the alert design matters more than the charts:

• Send a weekly digest, not a page. Schedule risk is not an outage. A Monday summary of every signal that crossed a threshold gets read. Twenty individual notifications get filtered.
• Name the owner in the alert. “RFI 142 blocks level 3 slab pour, need-by in 8 days, ball in court: structural engineer” is actionable. “Constraint risk: high” is not.
• Compare the project to itself. Portfolio-wide thresholds punish complex jobs and let simple ones coast. A drop from its own recent average is the better trigger.
• Alert on stale data. If no schedule update has loaded in longer than your cadence, that is itself a warning. An early warning dashboard showing old data is worse than none.

Troubleshooting

Float values look wrong compared to P6

Check the hours-to-days conversion first. Then check which float definition the schedule uses. A project-level “must finish by” date or activity constraints can produce negative float that looks like a bug but is correct. Also confirm whether the planner defines critical as longest path or as float below a threshold.

Trend lines break after a re-baseline or renumbering

If activity IDs change, LAG() finds no previous row and the activity appears brand new. Add a check that counts activities present in the previous snapshot but missing from the current one. A sudden jump means the history needs a mapping table, not a fresh start.

PPC sits at or near 100 percent every week

That usually means commitments are being made at the end of the week, or tasks are so loose they cannot fail. A PPC that never moves carries no signal. Look at how commitments are recorded before trusting the number.

The dashboard is green but the field says you are behind

Check how percent complete is calculated. P6 supports duration, physical and units percent complete types, and they can tell very different stories about the same activity. Believe the field, then find which input is out of step.

Common Mistakes

• Building on the live schedule only, with no history, so every trend has to be reconstructed by hand.
• Watching only the critical path and missing the near-critical work that becomes critical next month.
• Trusting classic SPI in the final third of a project, when it drifts toward 1.0 regardless of performance.
• Tracking overdue RFIs without linking them to the activities they block.
• Recomputing float yourself instead of using what the scheduling tool calculated.
• Sending so many alerts that the project team creates an inbox rule to hide them.

Best Practices

• Append every export with its data date. Never overwrite history.
• Start with two signals, float erosion and constraint aging, and add more once those are trusted.
• Agree with the planner on update cadence. The dashboard cannot be fresher than its inputs.
• Keep the RFI-to-activity mapping small and owned by one named person.
• Review thresholds after the first month. Early numbers are guesses.
• Show each signal with its source and data date, so nobody argues about whether it is current.

Frequently Asked Questions

What are leading indicators of construction delay?

The most useful are float erosion on near-critical activities, RFIs and submittals aging toward the start of the work they block, a falling PPC trend, activities repeatedly pushed out of the look-ahead, and reported progress outpacing logged manpower.

How often should the schedule be updated for early warning to work?

More often than monthly if you can manage it. Weekly or biweekly updates give the trend queries enough points to show movement. Signals from RFIs and weekly plans can refresh daily regardless of schedule cadence.

Is SPI a good early warning metric?

Early and mid-project, yes, alongside float and critical path analysis. Late in a project, classic SPI drifts toward 1.0 even on late jobs, so the time-based SPI(t) from Earned Schedule is more trustworthy.

Can I build this with Power BI instead of Grafana?

Yes for the reporting side, and it is often the better choice where the team already uses it. The snapshot database and trend logic are the same either way. Rule-based alerting is where you may want a separate tool.

Do I need Primavera P6 for a construction early warning dashboard?

No. Any scheduling tool that exports activities with dates and float works, including Microsoft Project. What you need is a consistent export on every update and somewhere to keep the history.

How do I link RFIs to schedule activities?

Usually with a small mapping table maintained by the project engineer: RFI or submittal number, activity ID, and need-by date. Mapping by area and spec section is a workable first step if activity-level links are too much at the start.

Conclusion

The project manager who learns about delay three weeks late usually isn’t short of data. The problem is that the data is spread across a schedule, an RFI log and a weekly plan, and each one only remembers its latest state. A construction early warning dashboard works because it keeps the history and watches the movement: float draining from work that isn’t critical yet, answers arriving after the work that needs them, promises slipping week after week. Snapshot first. Everything else is a query on top.

Need Help Building a Construction Early Warning Dashboard?

I build and host the data side of this for construction teams and project controls groups. Here is where I can help:

• Setting up a snapshot database that stores every P6 or Microsoft Project update with its data date.
• Writing XER and XML loaders that handle hours-based float, calendars and renumbered activities.
• Pulling RFIs and submittals from Procore or Autodesk Construction Cloud and linking them to activities.
• Building Grafana dashboards for float erosion, constraint aging, PPC and SPI(t) trends.
• Designing weekly digest alerts with named owners, sent to email, Slack or Teams.
• Hosting and maintaining the dashboard so your team reads it instead of running it.

If you want to scope it first, the Engagement Builder lays out phases and effort. Or send me one recent XER export and your RFI log, and I will tell you which signals your data can already support.

The post The Delay Was in the Data Three Weeks Ago: Building a Construction Early Warning Dashboard appeared first on John Nessime.

The same output showed roughly 11 GB in the available column. Nothing was wrong, and nothing had been wrong for months. The only thing broken was the alert that fired.

Low free memory on Linux is one of the most reliably misread signals in server operations. It looks like an emergency, it triggers scaling decisions, and it is almost always noise. This post covers what free is really telling you, which four signals actually indicate memory trouble, the one situation where a low free number genuinely matters, and what to put in your alert rules instead.

What the free command is actually reporting

free does not measure anything. It parses /proc/meminfo and prints a handful of those fields in columns. Every number it shows comes straight from the kernel and is readable directly if you want to skip the formatting.

A typical reading on a busy server looks roughly like this:

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            15Gi       3.4Gi       231Mi       128Mi        12Gi        11Gi
Swap:          2.0Gi          0B       2.0Gi

Read that left to right and the story is boring. Applications hold a few gigabytes. The kernel filled almost everything else with page cache and buffers, because every file that gets read or written passes through memory on the way, and directory metadata lands in reclaimable slab. What is left over, genuinely untouched, is the free column.

That last part is the bit people miss. Free memory is not a reserve. It is waste. A page holding a cached copy of a file your application reads every few seconds is doing useful work. A page sitting empty is doing nothing, and the kernel converts it into cache the moment it has a reason to. A server that has been up for a week and still shows plenty of free memory is a server that is barely touching its disk.

If you want the raw source rather than the formatted view:

$ grep -E '^(MemTotal|MemFree|MemAvailable|Buffers|Cached|SReclaimable|SwapTotal|SwapFree|Dirty):' /proc/meminfo

Two options are worth knowing. free -w switches to wide mode and splits buffers and cache into separate columns, which helps when attributing memory to a specific I/O pattern. free -h -s 5 reprints every five seconds, a crude but effective trend view while you reproduce a problem.

Signal one: available memory, not free memory

The available column maps to MemAvailable in /proc/meminfo, and it is the kernel’s own estimate of how much memory a new workload could allocate without pushing the system into swap. It has been present since kernel 3.14, so every distribution you are likely to be running exposes it.

What matters is that it is not free plus cache. That naive sum is what people calculated before MemAvailable existed, and it overstates the truth. The kernel starts from free pages, subtracts the reserves it refuses to dip below, then adds back the portion of file-backed page cache and reclaimable slab it believes it could actually release. Both additions are capped, because some cache is pinned and some reclaimable slab objects are in use.

The kernel documentation is explicit that this is an estimate whose accuracy varies from system to system, so treat it as a strong hint rather than an exact figure. It is still far better than anything you can compute yourself, and it is the number your dashboards should be built on.

If you change one thing after reading this: point your memory alerts at available, not at free. Everything else in this post is refinement on top of that.

Signal two: swap activity, not swap usage

The second most common false alarm is swap. Someone sees a few hundred megabytes of swap in use and concludes the machine is thrashing. It usually isn’t.

Swap usage is a stock: pages moved to disk at some point, possibly during a backup weeks ago, and never needed since. Leaving them there is correct, and pulling them back to satisfy a dashboard would evict something useful. Swap activity is a flow, and that is the part that hurts, because a page fault serviced from disk is orders of magnitude slower than one serviced from RAM.

vmstat gives you the flow. The si and so columns are swap-in and swap-out, in blocks per second over the sampling interval:

$ vmstat 5 6

Discard the first row. vmstat reports averages since boot on its first line, which is almost never what you want. Read the rows after that. If si and so are both consistently non-zero, the machine is simultaneously reading pages back to serve current work and writing pages out to make room for new allocations. That is thrashing, and it is a real signal with no ambiguity attached.

A single non-zero sample is not that. Brief swap activity during a log rotation or a backup window is normal. Sustained activity across several consecutive samples is what to escalate.

Signal three: memory pressure, which is the one that catches things early

Pressure Stall Information is the signal I reach for first now, and it is badly underused. It has been in the kernel since 4.20 and it answers a question none of the other tools answer directly: how much time are my processes losing because memory is contended?

$ cat /proc/pressure/memory
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0

The some line is the share of wall-clock time in which at least one task was stalled waiting on memory. The full line is the share in which every non-idle task was stalled, meaning nothing useful happened at all. The averages cover ten seconds, one minute and five minutes, and total is a cumulative counter in microseconds that updates continuously, so it catches a two-second stall the running averages smooth away entirely.

Here is why this beats a percentage threshold. Utilization tells you how much of a resource is consumed; pressure tells you whether that consumption is costing you anything. A machine can sit at high utilization for months with zero pressure, which means the sizing is fine. It can also sit at moderate utilization while full climbs, which means reclaim is working hard and latency is already suffering with no threshold crossed.

With cgroup v2 mounted, the same data exists per cgroup in each directory’s memory.pressure file, so you can attribute pressure to a specific service rather than the whole machine. Prometheus node_exporter collects the system-level PSI counters, and if you already ship metrics to something like Grafana Cloud these graph well beside your latency panels.

Signal four: the OOM log, which is the only unambiguous one

When memory really does run out, the kernel writes a record. It does not do this quietly and it does not do it conditionally. If there is no OOM record, no process was killed for memory, full stop.

$ sudo journalctl -k --since "24 hours ago" | grep -i "oom"
$ sudo dmesg -T | grep -i "killed process"

The kill line names the victim and reports its memory breakdown at the moment of death: total virtual size, anonymous resident set, file-backed resident set, shared memory resident set, and the oom_score_adj that was in effect. The line above it, where a process “invoked oom-killer”, tells you which allocation triggered the whole event, and that process is frequently not the one that got killed.

That distinction is the most useful thing in the log. The kernel picks a victim by badness score, driven largely by how much memory a process holds relative to what is available to it, adjusted by oom_score_adj. A slow leak in a small helper can push the machine over the edge and get your database killed instead, because the database is simply the largest thing running. Read both lines before deciding who the culprit is.

If you have a service that should survive at almost any cost, systemd exposes OOMScoreAdjust= in the unit file, which persists across restarts in a way that writing to /proc/<pid>/oom_score_adj does not. Use it sparingly. Protecting a process that is itself leaking just moves the crash somewhere less convenient.

When low free memory on Linux really is the problem

Everything above assumes you are looking at a whole machine. Inside a container, the accounting changes, and this is the failure mode that is invisible until it isn’t.

The cgroup memory controller charges page cache to the cgroup that caused it. Clean, evictable, perfectly reclaimable cache still counts toward memory.current, and that is what gets compared against memory.max. A container reading a large file has just charged that file to its own budget.

$ cat /sys/fs/cgroup/memory.current
$ cat /sys/fs/cgroup/memory.max
$ grep -E '^(anon|file|slab_reclaimable) ' /sys/fs/cgroup/memory.stat

The split in memory.stat is what you actually want. anon is memory your process genuinely holds; file is page cache charged to you that the kernel can take back under pressure. A container at its limit on mostly file is in a very different position from one at its limit on mostly anon.

Two things follow from this that catch people out repeatedly:

• Reclaim is not instant. Under heavy pressure the reclaim path itself runs slowly, so a container can be OOM-killed while still holding clean page cache that was theoretically available.
• Running free inside a container lies to you. Unless something virtualizes /proc, you are reading the host’s /proc/meminfo, which knows nothing about your limit. The cgroup files are the only truth here.

cgroup v2 also has memory.high, a soft limit. Crossing it triggers reclaim and throttling rather than a kill, which turns a hard failure into a slow one. That is usually the better trade, and it is worth setting deliberately rather than leaving it at its default.

What to alert on

The alert on the ticket at the top of this post was watching the wrong number. Here is what a memory alert set looks like when it watches the right ones:

• Available memory below a percentage of total, sustained. Ten percent for ten minutes is a reasonable starting shape. A single thirty-second dip is noise and should never page anyone.
• Sustained swap-in and swap-out together. Rate of change on the swap counters, not the absolute amount of swap in use.
• Memory PSI rising. The some line moving off the floor is your early warning. The full line moving at all is your late one.
• Any OOM kill, ever. This one is binary and should always generate a ticket, even if the service recovered on its own.
• Per-container memory events. The cgroup memory.events file counts limit hits, and a rising count is a limit that needs revisiting.

Notice that free memory does not appear anywhere in that list. There is no threshold on the free column that produces a useful alert, because the number is a byproduct of the kernel’s caching behaviour rather than a measure of anything you care about.

Troubleshooting: which tool answers which question

When a memory question lands on you, the order you check things in matters more than the tools.

1. Did anything actually die? Check the kernel log for OOM records first. If there are none, this is a performance question and not a capacity failure, which changes everything about what you do next.
2. Is the machine stalling? Read /proc/pressure/memory. Zero pressure with low free memory means you are done. Go close the ticket.
3. Is it swapping right now? Run vmstat 5 for half a minute and watch si and so. Ignore the first row.
4. Who is holding the memory? ps aux --sort=-%mem | head -15 gets the shortlist. Resident set size double-counts shared pages, so summing the column overstates the total.
5. Is it the kernel rather than a process? If the process list does not account for the usage, check slabtop and SUnreclaim in /proc/meminfo. Unreclaimable slab growth is a different problem, usually a driver, filesystem or network workload creating enormous numbers of kernel objects.
6. Is it growing? A leak is a slope, not a level, and one reading tells you nothing. This is the whole argument for having the graph before you need it.

One more that saves time on crowded boxes: resident set size is the wrong measure of what a process really costs, because shared libraries are counted against every process that maps them. Proportional set size divides shared pages among the processes sharing them, and tools reporting PSS give a far more honest answer when you are deciding what to move off a busy VPS.

Common mistakes

• Dropping caches to make the number look better. Writing to /proc/sys/vm/drop_caches discards clean page cache and reclaimable slab. The kernel documentation says plainly that using it outside testing or debugging is not recommended, because everything dropped has to be re-read from disk. You trade a nicer-looking free output for an I/O spike and worse latency. It is a benchmarking tool, not an operations tool.
• Setting vm.swappiness to zero as a reflex. It does not disable swap and it does not make the machine faster. It makes the kernel strongly prefer evicting page cache over swapping anonymous pages, which on a database host means throwing away exactly the cache you wanted. Change it for a measured reason, not by default.
• Running with no swap at all. Swap is not just an overflow area. Without it the kernel loses the option of paging out genuinely cold anonymous memory, which means the only lever left under pressure is evicting cache or killing something. A modest swap file gives reclaim somewhere to go.
• Summing RSS across processes and comparing it to total RAM. Shared pages are counted once per process. The sum will exceed reality, sometimes dramatically.
• Resizing the instance before finding the slope. More RAM buys time against a leak. It does not fix one, and the bill arrives every month afterwards.

Best practices

• Graph available memory, swap rate and memory PSI on one panel. Together they read as a story; individually each one misleads.
• Keep the raw counters, not just derived percentages. A percentage of an instance size that has since changed cannot answer “was this happening last week”.
• Set container limits from observed anon usage plus headroom, not a round number someone picked, and put memory.high below memory.max so pressure shows up as slowness before it shows up as a kill.
• Alert on rate of change for anything that can leak. The steady climb is the signal; the level it reached is just when you happened to look.
• Right-size deliberately. On providers such as Contabo or InterServer, where RAM is a meaningful share of the monthly bill, the gap between a real requirement and a panicked upgrade is money you keep every month.
• Write the explanation down where your team will find it. Every organization rediscovers the free-versus-available confusion independently, usually at three in the morning.

Frequently asked questions

Why is free memory so low on my Linux server?

Because the kernel used it for page cache and buffers, which is the intended behaviour. Any file read or written passes through memory, and the kernel keeps those pages in case they are needed again. It reclaims them automatically the moment an application needs the space. Check the available column instead.

What is the difference between free and available in free -h?

Free is memory that is completely unused. Available is the kernel’s estimate of how much a new process could allocate without swapping: free memory plus the portion of cache and reclaimable slab it believes it could release. Available is almost always much larger, and it answers the question you are actually asking.

Should I clear the page cache to free up memory?

No. The kernel reclaims cache automatically under pressure, and more selectively than a blanket drop. Writing to drop_caches forces everything to be re-read from disk, and the kernel documentation advises against using it outside testing or debugging. Doing it on a schedule is a strong hint something else is misconfigured.

Is swap being used a bad sign?

Not by itself. Pages that were swapped out during a quiet period and never needed again are fine where they are. What matters is swap traffic: sustained non-zero si and so in vmstat at the same time means the machine is thrashing. Alert on the rate, not the amount.

How do I tell if a process was killed for running out of memory?

Search the kernel log. journalctl -k filtered for “oom”, or dmesg -T filtered for “killed process”, shows a record naming the process, its PID and its memory breakdown at the time. The kernel always logs this, so if there is no record the process died for some other reason.

Why does my container get OOM-killed when memory looks fine inside it?

Because free inside a container usually reads the host’s memory information, not the cgroup limit being enforced. Read memory.current, memory.max and memory.stat instead. Page cache charged to the container counts toward its limit, so a read-heavy workload can hit the ceiling without any process holding much anonymous memory.

What is a good memory alert threshold?

Available memory below roughly ten percent of total, sustained for ten minutes, is a defensible starting point to tune against your own workload. Pair it with memory PSI and an unconditional alert on any OOM kill. A threshold on the free column, at any value, will page you for nothing.

Conclusion

Low free memory on Linux is the expected steady state of a healthy, working server. The kernel fills memory with cache because empty memory earns nothing, and hands it back the instant an application asks. The free column is a byproduct of that behaviour, not a health metric, and alerting on it produces exactly the kind of urgent ticket that wastes an afternoon and ends in a change request nobody needed.

The one thing worth remembering: free tells you what is idle, available tells you what you can have, and pressure tells you whether any of it is costing you. Watch the second and third, log the OOM events, and read the cgroup files rather than free whenever a container is involved.

Need help reading your memory signals?

Most of the memory work I get asked for is not “add more RAM”. It is working out whether the number someone is worried about means anything. I can help with:

• Rebuilding memory alerts around available memory, swap rate and PSI so they stop paging on normal cache behaviour
• Diagnosing repeated OOM kills, including the cases where the process being killed is not the one causing the problem
• Setting container and pod memory limits from observed anonymous usage rather than guesses, with a soft limit so pressure surfaces before a kill does
• Separating a genuine leak from normal cache growth using retained metrics rather than a single reading
• Right-sizing servers and instances honestly, up or down, based on what the workload actually needs
• Building the dashboard that turns the next memory question into a five-minute conversation instead of an incident

If you have a free -h output, a vmstat capture or an OOM log line that is bothering you, send it over and I will tell you what it means.

The post Low Free Memory on Linux Isn’t a Problem. Here’s What Actually Is. appeared first on John Nessime.

The sync worker was healthy. Zero errors in the last twenty-four hours. CPU flat, memory flat, no restarts, no failed jobs. Every dashboard was green because every dashboard was measuring the wrong thing. The integration had stopped receiving anything from Zoho four days earlier, and an integration that receives nothing produces no errors at all.

That is the whole problem with monitoring Zoho integrations. The failures that hurt do not throw exceptions. They show up as silence, as slowly degrading throughput, or as HTTP 200 responses that quietly did not do what you asked. This post covers how to instrument for those three failure families using CloudWatch, Grafana and Loki, and why the obvious metrics miss all three.

The three ways Zoho integrations actually break

Before touching a dashboard, it helps to be honest about what you are defending against. In my experience these fall into three groups, and they need genuinely different instrumentation:

1. Silent stop. Zoho stops sending you events. Nothing fails, because nothing happens.
2. Silent throttle. You run out of API credits or hit a concurrency ceiling. Throughput degrades before anything hard-fails.
3. Silent drift. Calls succeed, records do not land, and the two systems diverge without a single error log.

An error-rate panel catches none of these. That is not a criticism of error-rate panels, it is just what they are for.

Failure family one: the integration that just stops

If you push data out of Zoho CRM using the Notifications API, you subscribe a notify_url to module events and Zoho posts to it when records change. It works well. It also has a detail that catches people out repeatedly.

Notification channels expire. Zoho’s documentation is explicit: the expiry can be set to a maximum of one week from the time the channel is enabled, and if you do not specify one, or you ask for longer than a week, the default expiry is one hour. So the failure mode is not “someone forgot to renew after a year.” It is “someone deployed a subscription script without an expiry field and the integration worked beautifully for sixty minutes.”

What makes this genuinely hard rather than just annoying is that there is no endpoint to list your currently active channels. You cannot ask Zoho what it thinks it is subscribed to. You can only ask whether events are still arriving. That single fact should shape your entire monitoring design: you are not monitoring for errors, you are monitoring for absence.

Emit a heartbeat, then alarm on its absence

The receiving end of the webhook should emit a metric on every delivery, including deliveries you decide to discard. Embedded Metric Format is the cheapest way to do this from Lambda or a container, because you write structured JSON to stdout and CloudWatch extracts the metrics asynchronously. No PutMetricData call, no synchronous latency in your request path.

{
  "_aws": {
    "Timestamp": 1710000000000,
    "CloudWatchMetrics": [
      {
        "Namespace": "ZohoIntegration",
        "Dimensions": [["Module", "Channel"]],
        "Metrics": [
          { "Name": "NotificationsReceived", "Unit": "Count" },
          { "Name": "ChannelSecondsRemaining", "Unit": "Seconds" }
        ]
      }
    ]
  },
  "Module": "Leads",
  "Channel": "leads-prod",
  "NotificationsReceived": 1,
  "ChannelSecondsRemaining": 486000,
  "record_id": "5725767000002166520",
  "operation": "insert"
}

Note what is a dimension and what is not. Module and Channel are bounded sets, so they are safe dimensions. record_id sits in the log body only. EMF creates a separate custom metric for every unique dimension combination, so promoting a record ID to a dimension is how you turn a monitoring project into a billing incident. AWS says this plainly in the EMF documentation and it is worth reading twice before you ship.

Now the alarm. A normal CloudWatch alarm goes to INSUFFICIENT_DATA when the metric stops reporting, and INSUFFICIENT_DATA does not notify. That is precisely backwards for this failure. You want missing data treated as a breach:

aws cloudwatch put-metric-alarm 
  --alarm-name zoho-leads-notifications-silent 
  --namespace ZohoIntegration 
  --metric-name NotificationsReceived 
  --dimensions Name=Module,Value=Leads Name=Channel,Value=leads-prod 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 2 
  --threshold 1 
  --comparison-operator LessThanThreshold 
  --treat-missing-data breaching 
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:ops-alerts

The --treat-missing-data breaching flag is the entire point of that command. Without it, an integration that goes completely dead produces an alarm that sits quietly in an unknown state forever.

On the Loki side the equivalent is absent_over_time, which returns 1 when a stream has produced no entries in the window. Run it from the Loki ruler so it fires even when nobody has a browser open:

groups:
  - name: zoho-integration
    interval: 5m
    rules:
      - alert: ZohoNotificationsSilent
        expr: absent_over_time({job="zoho-webhook", module="leads"}[2h])
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: "No Zoho notifications for Leads in 2h"
          description: "Channel may have expired. Re-subscribe and check channel_expiry."

One caveat worth knowing: absent_over_time only works for label combinations you name explicitly. It cannot tell you that a stream you never thought about has gone quiet. That is a real limitation, and it means your channel inventory has to live in your own configuration, not be discovered from Loki.

Because you cannot list channels through the API, treat renewal as a scheduled job with its own deadman alert, and emit ChannelSecondsRemaining at each renewal. Alarm when it drops below roughly a day. That gives you warning before the silence starts rather than after somebody notices.

Failure family two: throttling that is not rate limiting

Most engineers instrument Zoho as if it were a requests-per-minute API. It is not, and that mismatch produces dashboards that look fine right up until calls start failing.

Zoho CRM meters API usage in credits, not calls. Different operations deduct different amounts, and bulk operations are metered per batch of records rather than per request. The budget resets on a rolling twenty-four hour window measured from each call, not at midnight. If your dashboard has a daily reset at 00:00, it is drawing a line that does not exist.

The gauge that appears halfway through the tank

Here is the detail that makes credit monitoring awkward. Zoho returns remaining credits in an X-API-CREDITS-REMAINING response header, but only once usage has reached fifty percent or more of the daily allowance. Below that threshold the header simply is not there.

Read that again in monitoring terms. An absent header means either “you are comfortably under half your budget” or “your instrumentation is broken and you have no idea.” Those two states look identical, and one of them is fine while the other is a blind spot. So record both facts separately:

• A counter for responses where the header was present, which effectively tells you when you crossed the halfway mark.
• A gauge of the value itself, which only has meaning while the header exists.
• Your own client-side count of calls made, which works regardless and is the only thing you control.

The first crossing of that threshold is a useful early warning in its own right. Alert on the transition, not just on a low remaining value.

Concurrency is a separate ceiling

Alongside credits, Zoho applies a concurrency limit: the number of calls active simultaneously per org. There is also a sub-concurrency limit covering the expensive operations such as COQL queries, composite requests, lead conversion, and inserts or updates above a small record count. Exceeding either returns TOO_MANY_REQUESTS.

Two consequences matter for monitoring. First, since the limit is on simultaneous calls rather than calls per unit time, a requests-per-minute graph will never predict a concurrency failure. You need a gauge of in-flight requests, incremented before the call and decremented in a finally block. Second, the sub-concurrency ceiling is the same across every CRM edition, so upgrading the plan raises the general limit but does not buy headroom on the expensive calls. Batch those and queue them rather than fanning out.

One more source of consumption that surprises people: integration tasks inside Deluge functions consume credits from the same org budget as your external API calls. If your credit graph climbs while your service is idle, someone’s workflow rule is spending your budget. Zoho’s API Dashboard breaks consumption down by application and function, which is where to look before blaming your own code.

Failure family three: successful calls that did nothing

Bulk and multi-record operations return a per-record status inside the response body. The HTTP status can be perfectly healthy while individual records were rejected for validation rules, mandatory fields, duplicate checks or picklist values that do not exist in the target layout.

If your client only checks the HTTP status, you will log a success, increment a success counter, and move on. Weeks later somebody asks why the CRM is missing a slice of records that your logs insist were written.

The fix is to stop counting requests and start counting records, on both sides:

• Records submitted versus records confirmed in the per-record response status.
• A breakdown of rejection reasons as a low-cardinality label, so a spike in one specific validation error is visible.
• A periodic reconciliation job comparing source counts against Zoho counts for a bounded window, which is the only thing that catches drift you failed to anticipate.

In LogQL, the divergence between the two counters is the panel that actually earns its place on the wall:

sum(count_over_time({job="zoho-sync"} | json | status = "submitted" [1h]))
  -
sum(count_over_time({job="zoho-sync"} | json | status = "confirmed" [1h]))

A healthy integration holds that near zero. A number that grows steadily is drift, and it is visible weeks before anyone in sales notices.

Deciding which signal goes where

Running CloudWatch, Grafana and Loki together only helps if each one has a job. The split I reach for first:

• CloudWatch holds the numbers you alarm on, especially where the workload already runs in AWS and EMF gives you metrics for free out of log output.
• Loki holds the full request and response context: which record, which module, which validation error, which channel. This is what you actually read at three in the morning.
• Grafana is the single pane over both, with CloudWatch and Loki added as separate data sources so one dashboard can show the credit gauge next to the rejection reasons.

The trap that spans both backends is cardinality, and it bites in the same way for different reasons. In CloudWatch, each unique dimension combination becomes a billable custom metric. In Loki, each unique label combination becomes a separate stream, and Loki was designed for a small number of long-lived streams. Record IDs, email addresses and timestamps belong in neither.

Loki’s answer is structured metadata: high-cardinality fields attached to log lines without creating streams or entering the index. It needs enabling per tenant:

limits_config:
  allow_structured_metadata: true
  volume_enabled: true

Put module, environment and job in labels. Put the Zoho record ID and the correlation ID in structured metadata. Query with a label selector first and filter on the rest.

On collection, one scheduling note that does age: Promtail reached end of life on 2 March 2026, with its collection capabilities folded into Grafana Alloy. If you are still shipping through Promtail, that migration is now maintenance debt rather than an optional upgrade, and Grafana published a conversion tool for existing configurations. New pipelines should start on Alloy.

Where you run all this is a real decision. Self-hosting Loki on a VPS from a provider like Contabo or InterServer is cheap and entirely workable for a single integration’s log volume, and it keeps the data under your control. Grafana Cloud or Amazon Managed Grafana removes the operational surface if you would rather not babysit an ingester. Both are defensible. The honest trade-off is that self-hosted Loki is easy to stand up and less easy to keep healthy once retention and compaction start to matter.

Troubleshooting the monitoring itself

• EMF metrics never appear. The JSON must be a single line and structurally valid. A pretty-printed or truncated log line is ingested as an ordinary log event with no error, so check the raw event in the log stream rather than the metrics console.
• The alarm never fires despite obvious silence. Look at the alarm state. Sitting in INSUFFICIENT_DATA means the missing-data treatment was left at its default.
• absent_over_time returns nothing. That is the success case. It emits a value only when the stream is empty, which makes it feel broken while everything is working.
• Loki queries slow to a crawl. Check stream count before blaming resources. A label added “temporarily” during debugging is the usual cause.
• Webhook deliveries stop after a deployment. Check whether the subscription script reran and created a channel without an expiry. An hour later it will look identical to a network problem.
• Credit consumption does not match your call count. Expected. Operations deduct different credit amounts, and Deluge integration tasks draw from the same pool.

Common mistakes

• Alerting on error rate alone, which is blind to every failure described above.
• Creating notification channels without an explicit channel_expiry and inheriting the one-hour default.
• Assuming an absent X-API-CREDITS-REMAINING header means healthy usage rather than sub-threshold usage.
• Building a daily credit dashboard when the limit is a rolling window.
• Putting record IDs into EMF dimensions or Loki labels.
• Treating an HTTP 200 on a bulk write as confirmation that every record landed.
• Running alert rules only in the Grafana UI, so nothing evaluates if that instance is down.

Best practices for monitoring Zoho integrations

1. Start with a deadman alert per channel. If you build one thing, build this.
2. Set channel_expiry explicitly, renew on a schedule, and monitor the renewal job as carefully as the integration.
3. Count records, not requests, on both sides of every write.
4. Keep a client-side call counter so you are never dependent on a header that may not be sent.
5. Gauge in-flight requests to see concurrency pressure before it becomes TOO_MANY_REQUESTS.
6. Attach a correlation ID to every record as structured metadata so one identifier joins logs and metrics.
7. Evaluate critical rules server-side, in the Loki ruler or as CloudWatch alarms.
8. Test by disabling a channel in a sandbox org and confirming the alert fires. An untested deadman alert is decoration.

Frequently asked questions

Why did my Zoho webhook stop working on its own?

Almost always channel expiry. Zoho CRM notification channels last a maximum of one week, and if the expiry is omitted or set beyond that maximum, the channel defaults to one hour. Nothing is logged on your side because the deliveries simply stop arriving.

Can I list my active Zoho notification channels?

There is no endpoint for enumerating active channels, which is a long-standing complaint in Zoho’s developer community. Keep the inventory in your own configuration and treat the receipt of events as the only reliable liveness signal.

Do I need both CloudWatch and Loki?

No. Either can carry the whole job. The pairing makes sense when your integration already runs on AWS, because EMF gives you alarmable metrics from log output at no extra instrumentation cost, while Loki holds richer request context more cheaply. If you are starting fresh outside AWS, one backend is simpler and simpler is worth a lot.

How do I monitor Zoho API credits properly?

Record the X-API-CREDITS-REMAINING header when it appears, alert on the moment it first appears in a window, and maintain your own call counter as the fallback. Model the budget as a rolling twenty-four hour window rather than a daily reset.

What causes TOO_MANY_REQUESTS if I am under my credit limit?

Concurrency, not consumption. Credits and concurrency are independent ceilings. Expensive operations such as COQL queries and composite requests share a tighter sub-concurrency pool, so a burst of parallel queries can trip it while your credit balance is untouched.

Should I still use Promtail for shipping integration logs?

No. Promtail reached end of life on 2 March 2026 and its functionality moved into Grafana Alloy. Existing deployments will keep running, but they are unsupported, and Grafana provides a tool to convert Promtail configuration to Alloy.

How long should the deadman alert window be?

Base it on the quietest legitimate period for that module. A leads webhook that goes quiet overnight and at weekends needs a longer window than an order sync, or you will train the team to ignore it. Measure the real gap distribution for a fortnight before picking a number.

The one thing worth remembering

Monitoring Zoho integrations is mostly the discipline of detecting absence. Expired channels, exhausted credits and silently rejected records all present as nothing happening, and nothing happening is invisible to the instrumentation most teams reach for first.

So invert the default. Alarm when the count is too low rather than when the errors are too high. Set --treat-missing-data breaching. Use absent_over_time. Count records on both sides and watch the gap. If you build only one thing from this post, build the deadman alert on notification delivery, because that is the failure that will otherwise be reported to you by a salesperson wondering where the leads went.

Need help instrumenting a Zoho integration?

Most of the work here is judgement about what to measure rather than volume of code. If you are staring at a sync that keeps going quiet, this is the sort of thing I take on:

• Building deadman alerting for Zoho notification channels, including renewal jobs that monitor themselves.
• Instrumenting credit and concurrency headroom so throttling is visible before it becomes TOO_MANY_REQUESTS.
• Designing EMF dimensions and Loki label schemas that will not turn into a cardinality bill.
• Adding reconciliation jobs that compare record counts across systems and surface drift early.
• Standing up Loki and Grafana Alloy, or migrating an existing Promtail pipeline that is now unsupported.
• Reviewing an existing Grafana dashboard and cutting the panels that look reassuring but detect nothing.

Send me a sample webhook payload, a redacted response header set, or a screenshot of the dashboard you currently trust, and I will tell you what it would miss.

The post Monitoring Zoho Integrations: Catching the Failures That Never Throw an Error appeared first on John Nessime.

That is the failure worth writing about. Drift itself is normal. Somebody fixes production at 2am, a managed service rotates a value, a policy engine stamps a tag. The part that breaks is the alert, and it breaks quietly, because a muted channel looks identical to a quiet one.

Switching on Terraform drift detection takes one flag. Building Terraform drift detection that engineers still read six months later means deciding what not to send: which plan mode you run, what you classify as noise, how you route by blast radius, and whether the message names who made the change. A detector firing daily with the same content is worse than none, because no detector leaves you knowingly blind while a muted one leaves you blind with a green dashboard. Every section below asks the same question differently: does this message change what a human does in the next ten minutes?

Pick the right plan mode or you will alert on your own backlog

This is the biggest source of false drift, and it catches almost everyone building their first check. A normal terraform plan compares three things: your configuration, your state, and the real infrastructure. It returns a non-zero exit code when any of them disagree, so it fires for real drift and just as loudly for a pull request that merged this morning and has not been applied. Your drift channel becomes a notification feed for your own deployment backlog, and people learn that most messages are work in progress.

Refresh-only mode narrows the question. HashiCorp documents it as a plan whose goal is only to update state and root module outputs to match changes made to remote objects outside of Terraform. That is the actual definition of drift.

terraform init -input=false

terraform plan -refresh-only -detailed-exitcode -lock=false -input=false -no-color

# exit 0  state matches the real infrastructure
# exit 1  the check itself failed
# exit 2  something changed outside Terraform

Two flags there matter more than they look. -detailed-exitcode is what makes the result machine-readable at all: without it a successful plan returns zero whether or not anything changed, so your job passes forever. And -lock=false is safe here specifically because a plan never writes state. Only terraform apply -refresh-only updates the state file. A read-only detector that grabs the state lock will collide with real applies, and the loser of that race is the engineer trying to ship.

The trade-off, stated plainly: refresh-only will not tell you that merged code has not shipped, and it will not find resources created outside Terraform, because it can only refresh what is already in state. I run both modes and route them separately. Refresh-only feeds the alert. Normal plan feeds a weekly digest tied to the repository, where unapplied changes are a delivery problem rather than a security one.

Classify the noise before you route anything

Most recurring drift falls into a few families. Sorting yours takes an hour and decides most of your routing rules.

Provider normalization

A JSON policy document comes back with keys reordered. An identifier returns in different case. An empty block reads back as null instead of an empty map. Terraform sees a difference because it compares a stored representation against an API response, and that round trip is not always lossless. Fix these in the configuration where you can, because a phantom diff that follows you into every plan also hides real diffs on the same resource.

Attributes another system legitimately owns

The classic is autoscaling capacity. Your config says two instances, a scaling policy moved it to five during a spike, and the next plan proposes taking it back to two. Applying that kills three instances mid-spike. The drift is real; reverting it is the wrong move.

resource "aws_autoscaling_group" "web" {
  min_size         = 2
  max_size         = 10
  desired_capacity = 2

  lifecycle {
    # Owned by the scaling policy after creation.
    # Reverting this during a spike removes live capacity.
    ignore_changes = [desired_capacity]
  }
}

Understand what you bought. ignore_changes stops Terraform proposing an update for that attribute, and it applies only after creation, not on the first apply. It also delegates the attribute permanently. If someone sets desired capacity to zero at 3am, your check will not raise a hand. Fair trade for capacity. Terrible trade for an ingress rule.

Tags stamped by something else

Organization tag policies, cost allocation tooling, backup schedulers and compliance agents all add tags. The AWS provider exposes both the tags you set and the effective set including defaults, and ignoring the effective set silences all tag drift at once. Convenient, and slightly dangerous if you use tags for access control, because tag-based policy conditions stop being watched. The same logic covers values the platform rotates or defaults on its own, like a managed secret version: they drift on a schedule that has nothing to do with your team, and belong in a digest.

The goal here is not a clean plan. A clean plan is a side effect. The goal is a decision per family: see it immediately, see it weekly, or never see it again.

Route by blast radius, not by count

“Drift detected: 12 resources changed” makes a human open a terminal before they know whether to care, and that friction is where attention goes to die. Split the output by what each resource can do if it is wrong:

• Wake someone. Anything governing exposure or access: security groups and firewall rules, IAM policies and roles, key policies, public access blocks, bucket policies, DNS records pointing at production.
• Review today. Instance sizes, scaling bounds, database parameters, retention settings, anything that costs money or changes capacity.
• Weekly digest. Tags, descriptions, names, and every family you classified as platform noise.

To split them you need structure, not text. Terraform’s machine-readable output emits a message for each resource that changed outside Terraform, carrying the resource address and the action it would take.

terraform plan -refresh-only -no-color -json 
  | jq -c 'select(.type == "resource_drift") | .change'

Check the exact field names inside that object against the machine-readable output reference for your Terraform version before building matching logic on them. Keep the severity mapping in the repository next to the code it describes. A resource type moving from digest to page should be a pull request, not something changed in a monitoring UI at midnight.

Make the alert answer “who did this”

Terraform tells you what changed. It has no idea who changed it, and that gap is where the investigation time goes. Half the drift you find is a legitimate incident fix nobody wrote down. If the alert names the actor, the response is a two-word reply in the thread instead of twenty minutes of archaeology. On AWS the audit trail can be queried by resource:

aws cloudtrail lookup-events 
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=sg-0123456789abcdef0

CloudTrail’s console event history only reaches back a limited window, so a weekly check may be hunting an event that has aged out; a trail delivering to S3, or CloudTrail Lake, covers the gap. Azure exposes the equivalent through Activity Logs, Google Cloud through Cloud Audit Logs. Wiring the lookup into the alert body does more for adoption than any amount of formatting.

Decide the remediation path before you automate it

There are three answers to a drift finding, and picking the wrong one can be worse than the drift.

1. Revert. Apply the configuration and push reality back. Right when the change was unauthorized or wrong.
2. Adopt. Change the code to match what is running. Right when the change was correct and the code was stale, which happens more than people admit.
3. Tolerate. Suppress the attribute deliberately, with a comment and an owner. Right when another system owns it.

The trap is automating option one. Auto-approving a remediation apply sounds like the natural endpoint of a drift pipeline, and it is fine right up until the drifted attribute is one that forces replacement. Then “restore the desired state” means destroying and recreating a live resource, unattended, at whatever hour the schedule runs. If you automate anyway, gate it hard: an allowlist of resource types, a check that the planned action is an in-place update rather than a replace, and a notification either way.

Troubleshooting a drift job that quietly stopped working

The nastiest failure mode is not a noisy check. It is a check reporting “no drift” because it never ran the plan successfully.

Exit code 2 kills your script

With set -e, a non-zero exit terminates the script. Exit code 2 means drift, which is the case you care about, so a naive script dies exactly when it has something to report. Capture the code without letting the shell act on it:

set -euo pipefail

terraform plan -refresh-only -detailed-exitcode -lock=false 
  -input=false -no-color > plan.txt 2>&1 && code=0 || code=$?

case "$code" in
  0) echo "no drift" ;;
  2) notify_drift plan.txt ;;
  *) notify_broken plan.txt; exit 1 ;;
esac

Note that the failure branch notifies too. Exit code 1 deserves the same urgency as drift, because a broken detector and a clean estate produce identical silence.

Credentials that rot without complaining

Drift checks run unattended, so nobody notices when a role trust policy changes, an OIDC subject claim stops matching after a repository rename, or a long-lived key gets rotated out. Add a deadman check: if the job has not reported within its expected interval, that absence is the alert. Same reasoning as heartbeat monitoring for backups.

A provider upgrade that drifts everything

Bump a provider and a new attribute default can appear across every resource of a type. Overnight the check reports hundreds of changed resources and whoever is on call assumes the tool is broken, which it sort of is. Commit the dependency lock file, pin the provider version, run the drift job on the same Terraform version as your apply pipeline, and treat provider upgrades as a reviewed change rather than something a bot merges.

Refresh times that outgrow the schedule

Refresh queries the API for every resource in state. Large workspaces get slow, then overlap their own schedule, then time out. The durable fix is splitting state along boundaries that already exist in your team structure. Terraform does offer targeting to narrow a plan, but it is documented as an exceptional-use option and prints a warning for good reason, so treat it as a stopgap rather than the design.

When to stop building this and buy it

A scheduled job in GitHub Actions or GitLab CI, or a systemd timer on a small always-on box from a provider like Contabo or InterServer, covers a handful of workspaces perfectly well and costs almost nothing. I would not reach for a platform before that stops working. What eventually stops working is the surrounding machinery: credentials for a dozen accounts across providers, concurrency against real applies, and retaining history so you can see that the same resource has drifted repeatedly this quarter, which is a process problem no single alert reveals.

That is where HCP Terraform’s health assessments, or a platform like Spacelift, Scalr or env0, starts earning its fee, since drift detection becomes a workspace setting with history and notifications attached rather than code you maintain. Two honest caveats: HCP Terraform gates health assessments behind its paid editions and requires remote or agent execution, so it is not a drop-in for local-state workflows, and any platform will happily send you the same undifferentiated wall of drift the homegrown script did. Severity tiering is a decision about your infrastructure that no vendor can make for you. Buying the runner does not buy the judgement.

Common mistakes

• Using normal plan mode for the alert. You notify yourself about unapplied work and teach people the channel is noise.
• Alerting on a count instead of a resource. “Nine resources changed” makes someone open a terminal to find out if they should care.
• Treating exit code 1 as a non-event. A detector that errors is indistinguishable from an estate with no drift.
• Taking the state lock for a read-only check. Monitoring should not be able to block a deployment.
• Auto-approving remediation across all resource types. One force-replacement attribute and the fix becomes the outage.

Best practices worth the effort

• Run refresh-only for drift and normal plan for unapplied changes, routed to different destinations.
• Keep the severity mapping in the repository, version controlled and reviewed like code.
• Give every suppression a comment naming the system that owns the attribute and a date to revisit.
• Alert on the detector failing and on the detector going quiet, not only on findings.
• Review recurring drift monthly. The same resource drifting repeatedly is a process gap, not an infrastructure one.

Terraform drift detection FAQ

What is the difference between a normal plan and refresh-only?

A normal plan compares configuration, state and reality, so it flags genuine drift and any committed change that has not been applied. Refresh-only compares state against reality only, which matches what most people mean by drift. Use refresh-only for the alert and normal plan for a delivery-focused report.

Does a refresh-only plan modify my state file?

No. Plan operations do not write state. Only applying a refresh-only plan updates the state file to match the drifted remote objects, and even then it makes no changes to infrastructure. That is why a detection job can safely skip the state lock.

How often should Terraform drift detection run?

Tie the interval to how long you would tolerate not knowing. Daily for production and anything holding customer data, weekly for lower environments, less often for slow-moving workspaces where refresh is expensive. Frequency matters less than whether the message gets read.

How do I find resources created outside Terraform?

Not with a plan. Terraform refreshes only what is already in state, so a resource it has never seen is invisible. You need a cloud-side inventory such as AWS Config, Azure Resource Graph or Cloud Asset Inventory, compared against your state, and import blocks to bring the results under management. Worth knowing that driftctl, the open-source tool most often recommended here, has moved into maintenance mode.

Does ignore_changes hide drift I actually care about?

Yes, and that is the point of using it. It stops Terraform proposing updates for that attribute regardless of what changed it, so a deliberate suppression and a malicious change look identical afterwards. Scope it to specific attributes, never whole resources you still care about, and document the owner.

Why did drift suddenly appear across hundreds of resources?

Almost always a provider or Terraform version change rather than something happening in your cloud account. A new provider release can add or reshape an attribute and every resource of that type reports a difference at once. Compare the versions used by the drift runner and the apply pipeline before investigating the infrastructure.

The one thing worth keeping

Terraform drift detection is not a monitoring problem, it is an attention problem. The flag that finds drift took an afternoon. Everything determining whether anyone still reads the output next quarter is judgement: refresh-only rather than normal plan, noise families sorted and suppressed deliberately, severity by blast radius rather than resource count, an actor name in the message, and an alert when the detector itself goes dark.

Judge it by one test: when a message lands in that channel, does anyone open it? If the honest answer is no, your detection is working and your alert is not.

Need help making your drift alerts worth reading?

This is common work once a Terraform estate grows past one team. Things I can help with:

• Auditing an existing drift job and finding the false positives that trained everyone to ignore it
• Building scheduled refresh-only checks in GitHub Actions, GitLab CI or a systemd timer, with correct exit code handling and a deadman alert
• Sorting recurring drift into families and deciding, per attribute, what to fix, adopt or suppress
• Building severity routing from the plan’s structured output so exposure changes page and tag changes do not
• Enriching alerts with CloudTrail, Activity Log or Cloud Audit Log lookups so findings arrive with an actor attached
• Splitting oversized state so refresh finishes inside its schedule, and judging whether a managed platform is worth it at your size

If you want a second opinion, send me a week of your drift output or the workflow file that produces it, and I will tell you what I would cut first.

The post Terraform Drift Detection That Nobody Mutes appeared first on John Nessime.

That slope is the signature. CloudWatch rarely fails loudly, it accumulates: a log group created automatically by a service you enabled once, retention left at the default, a metric filter with a dimension nobody thought about, and a dashboard quietly calling the API every few seconds forever.

This post breaks down where CloudWatch costs actually come from, why the same log line gets charged more than once, how cardinality multiplies your bill without your traffic changing, and how to attribute spend to specific log groups before you delete anything. It is organised by cost lever rather than by feature, because that is the order you work in when the invoice lands.

The four meters behind your CloudWatch bill

CloudWatch is not one product with one price. It is a set of independent meters that happen to share a console. Working out which meter is running is most of the job, because the fix for each is completely different.

• Ingestion. Per GB of log data accepted into a log group. Usually the largest component on log-heavy accounts.
• Storage. Per GB-month for what is retained afterwards. Cheap per unit, but it compounds because it never stops unless you tell it to.
• Analysis. Logs Insights bills on data scanned by a query, not on rows returned and not per query.
• Metrics and API calls. Custom metrics bill per unique time series per month. Separately, calls like GetMetricData bill per metric requested, which is how third-party dashboards end up on an AWS invoice.

The thing worth internalising: a single log line can be billed three times. Once when it arrives, again every month it sits in storage, and again every time a query scans the block it lives in. None of that is hidden, but nothing in the console surfaces it either.

Ingestion is the meter that actually hurts

Uncompressed in, compressed out

Ingestion is metered on the uncompressed volume of events you send. Storage is billed on the compressed size CloudWatch keeps. Text logs compress well, so the stored figure can be a small fraction of what you were charged to get the data in.

That asymmetry is why people look at a log group holding a few gigabytes, decide it is harmless, and miss that it cost several times that to arrive. The storedBytes value shown on a log group is the compressed number. It tells you about your storage line, not your ingestion line.

The log groups you never created

Most runaway ingestion comes from things switched on with a checkbox. VPC Flow Logs pointed at CloudWatch. EKS control plane logging with every type enabled. Lambda functions emitting a full structured payload per invocation. CloudTrail delivering to a log group as well as S3. RDS general query logs left on after a debugging session.

None of those feel like a decision at the time, and all of them scale with traffic rather than headcount. That is why the slope keeps rising while the team stays the same size.

The most effective lever here is not a CloudWatch setting at all. It is cutting verbosity at the source, before the bytes leave your application. Dropping debug-level logging in production, sampling health check lines, trimming repeated stack frames. Filtering after ingestion does nothing for this meter, because you already paid.

Retention: the one-line fix almost nobody applies

Log groups default to Never Expire. Not thirty days, not a year. Forever. Every log group created implicitly by another AWS service inherits that default, so an account accumulates hundreds of them without a single deliberate retention decision.

Find the offenders first. This lists every log group with no retention policy, alongside its compressed size:

aws logs describe-log-groups 
  --query 'logGroups[?!retentionInDays].[logGroupName,storedBytes]' 
  --output table

The ?!retentionInDays filter selects entries where the field is absent, which is how the API represents indefinite retention. There is no explicit “never” value to match on.

aws logs put-retention-policy 
  --log-group-name /aws/lambda/order-processor 
  --retention-in-days 30

Three things are worth knowing before running that across an account.

1. The value is an enum, not a free integer. Accepted values run 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731 and upward to ten years. Asking for 45 days is rejected.
2. Deletion is not instant. Expired events are marked for deletion and typically removed within about three days. They stop contributing to your storage charge once marked, and are excluded from the reported storedBytes, so the billing effect lands before the data physically goes.
3. Reverting means removing the policy, not setting zero. To go back to indefinite retention you call delete-retention-policy rather than passing a zero value.

Dry run before you bulk-apply. This prints what would change without touching anything:

aws logs describe-log-groups 
  --query 'logGroups[?!retentionInDays].logGroupName' 
  --output text | tr 't' 'n' | while read -r lg; do
    echo "would set 30-day retention on: $lg"
  done

Retention is where compliance and cost pull against each other, and cost should lose. If an auditor needs seven years of logs, seven years is the answer. The move is to export them to S3 and apply lifecycle rules there, rather than paying CloudWatch rates for archival data you will query twice a decade.

Cardinality: the multiplier nobody sees coming

This is the part that surprises people, because it breaks the intuition that cost tracks traffic. Cardinality can multiply your metrics bill by an order of magnitude while request volume stays flat.

CloudWatch treats every unique combination of namespace, metric name and dimension values as a separate metric. Not a variant of one metric. A separate, separately billed metric. Publish OrderLatency with a Region dimension holding three values and you have three metrics. Add a CustomerId dimension and you have three times however many customers you have.

The dangerous dimension values are the ones that look like identifiers: request IDs, trace IDs, pod names, session tokens, customer IDs, IP addresses, URL paths with IDs embedded in them. Each new value is a new billable series that persists and accrues.

Metric filters have a guard rail, and it bites

Metrics extracted from logs by a metric filter bill as custom metrics. A filter can carry up to three dimensions, and AWS documents an explicit safety mechanism: CloudWatch Logs disables a metric filter that generates a thousand distinct dimension name/value pairs within a period.

That guard rail is useful, but read what it means operationally. Hitting it does not just cost money, it silently stops the filter, so the metric you built an alarm on stops receiving data. If that alarm treats missing data as not breaching, it goes quiet rather than firing. You lose the alert and keep the bill.

Audit what exists before adding more, and count what a namespace actually produces:

aws logs describe-metric-filters 
  --log-group-name /aws/lambda/order-processor

aws cloudwatch list-metrics 
  --namespace MyApp/Orders 
  --query 'length(Metrics)'

If that count is materially larger than the number of distinct things you consciously measure, you have found your multiplier.

EMF does exactly what you tell it

Embedded Metric Format is a good design: you emit structured events, CloudWatch extracts metrics from declared fields, and you get high-cardinality context in the logs with low-cardinality metrics for alarming. That split is the point.

The failure mode is promoting a high-cardinality field into the dimension set. AWS documents this plainly: declare something like a request ID as a dimension and EMF will, by design, create a custom metric for every unique combination. Keep identifiers in the log body as properties, and keep dimensions to values you would put on a dashboard axis.

Anomaly detection alarms cost more than they look

An anomaly detection alarm does not evaluate one metric. It evaluates the source metric plus the upper and lower bounds of the predicted band, and those bounds are metrics too. That is roughly a threefold multiple against a standard threshold alarm, per alarm.

It earns its keep on genuinely seasonal signals where a static threshold produces constant noise. Switching an entire alarm estate to it “to be safe” triples a line item for very little detection benefit.

Query cost spikes exactly when you need it most

Logs Insights bills on the volume a query scans. Not rows returned. Not query count. Scanned bytes. A query returning twelve lines that scans a terabyte costs the same as one returning a million lines that scans a terabyte.

During an incident, an engineer runs a dozen exploratory queries across a wildcard log group selection and a wide time window. The pricing model charges you hardest at the moment the tool is most valuable, which is an uncomfortable incentive. Two habits reduce it without reducing your ability to debug:

• Narrow the time window first, then the filter. Scanning is bounded by the selected range across the selected groups. Starting at fifteen minutes and widening is far cheaper than starting at a week and narrowing.
• Select specific log groups, not patterns. Querying across every Lambda log group multiplies scanned bytes by the number of functions when you only care about one.

Watch for scheduled queries too. A dashboard widget or automation running an Insights query every few minutes against a large log group is a fixed monthly cost nobody remembers creating.

Finding out where your CloudWatch costs come from

Attribute before you cut. Deleting log groups because they look big is how teams lose the one log they needed three weeks later.

1. Split the bill by meter. In Cost Explorer, filter service to CloudWatch and group by usage type. The region-prefixed usage types map to the meters directly: DataProcessing-Bytes is log ingestion, TimedStorage-ByteHrs is log storage, DataScanned-Bytes is Logs Insights, and CW:Requests and CW:GMD-Metrics cover API calls.
2. Rank log groups by ingestion. The IncomingBytes metric in the AWS/Logs namespace, dimensioned by log group, is the per-group ingestion figure. Graph it with the Sum statistic. One documented gotcha: use an absolute range covering exactly thirty days, because a relative range or a different length produces a number that does not correspond to a billing month.
3. Check stored bytes separately. A log group can be small in ingestion but large in storage because it has accumulated since the account was created. That is a retention problem, not a verbosity problem, and the fix differs.
4. Count custom metrics per namespace. Compare the count against your mental model of what you measure. The gap is cardinality.
5. Look inside the noisiest group. Once you know which log group dominates, find out which streams are responsible before changing any application code.

For that last step, a Logs Insights query over a short window is enough. Keep the range tight, because this query is itself billed on what it scans:

stats count(*) as events, sum(strlen(@message)) as approx_bytes by @logStream
| sort approx_bytes desc
| limit 20

That approximates byte volume per stream, which is usually enough to identify the chatty component. It is an approximation, not a billing figure, because it counts message characters rather than the metered event size.

If you manage several accounts and want attribution done for you, cost platforms such as Vantage and CloudZero break CloudWatch spend down by usage type and resource without you building dashboards. Worth the spend if client reporting is part of the job. For one account and one engineer, Cost Explorer plus the steps above gets the same answer.

Cheaper destinations, and what they cost you

Once you know your ingestion is dominated by logs you rarely read, there are three real options. None is free of trade-offs.

• The Infrequent Access log class. Set at the log group level, it ingests at a materially lower per-GB rate in exchange for a reduced feature set. Historically that meant no metric filters, no alarming from logs and no Live Tail, though the capability list has expanded since launch. Check the current log classes documentation rather than assuming, and note that a log group you extract metrics from is not a candidate.
• Subscription filter to S3 via Amazon Data Firehose. Sends log data to object storage where lifecycle rules and Athena queries apply. You trade Insights and the console experience for much cheaper long-term retention. Good for audit and forensic logs, poor for anything you troubleshoot weekly.
• A separate log stack. Grafana Cloud, or self-hosted Loki and Prometheus on a VPS from a provider like Contabo or InterServer, moves cost from per-GB metering to a fixed monthly box you administer. The economics can be dramatically better at volume. The price is that you now own retention, backups, upgrades and the availability of your own observability during an outage, which is exactly when you least want to be debugging it.

The honest version: CloudWatch is expensive partly because it is operated for you and wired into everything. At a few dozen gigabytes a month, migrating away costs more engineering time than it saves. At terabytes, the arithmetic flips hard.

Troubleshooting a bill that will not go down

You set retention everywhere and storage barely moved. Check whether storage was ever the dominant cost. If your usage type breakdown was mostly DataProcessing-Bytes, retention was never going to fix it.

Custom metric count grows with no code changes. Something is emitting a dimension whose value set is open-ended. Look for deployment identifiers, container IDs, or a version string appended to a dimension.

An alarm went quiet and the bill jumped the same week. Suspect a disabled metric filter. Check the filter still exists and is producing data, and verify the alarm’s missing data treatment. Alarms treating missing data as not breaching fail silently by design.

API request charges you cannot place. External dashboards are the usual answer. A Grafana instance polling CloudWatch, or any tool calling GetMetricData on a short interval, bills per metric requested. Widening the refresh interval on dashboards nobody watches live is a quick win.

Costs dropped, then came back. Look at your infrastructure as code. If Terraform or CloudFormation creates log groups without a retention argument, every apply restores the default. Fixing it in the console fixes it until the next deploy.

Common mistakes

• Treating storage as the problem because it is the number the console shows. Ingestion is usually larger and invisible in the log group list.
• Adding a dimension because it would be useful to filter by, without checking how many distinct values it can take.
• Filtering logs after they reach CloudWatch. The ingestion charge already applied.
• Deleting log groups to cut cost without checking whether anything alarms on them.
• Setting retention in the console while the IaC that owns the log group has none configured.
• Enabling every EKS control plane log type when only the audit log is consumed.

Best practices for controlling CloudWatch costs

• Make retention a required argument. If log groups come from Terraform or CloudFormation, treat a missing retention setting as a policy violation and fail the plan. Checkov and similar scanners have rules for exactly this.
• Decide verbosity per environment, not per service. Debug in staging, info in production, with a documented way to raise it temporarily during an incident.
• Keep dimensions bounded, identifiers in the message body. If you cannot enumerate a dimension’s possible values on a whiteboard, it does not belong in the dimension set.
• Alarm on the bill itself. A billing alarm on estimated charges, or an anomaly monitor scoped to CloudWatch, turns a quarterly surprise into a same-week signal.
• Review the top five log groups by ingestion monthly. Five minutes with the IncomingBytes graph catches new noise before it compounds.
• Route archival logs out deliberately. Anything kept for compliance rather than operations belongs in S3 with lifecycle rules from the start.

Frequently asked questions

Why is my CloudWatch bill so high when my traffic has not changed?

Two causes account for most cases. Storage accumulating because log groups default to indefinite retention, so the bill grows every month even at constant ingestion. Or metric cardinality, where a dimension with an open-ended value set keeps creating new billable series. Neither tracks request volume, which is why the bill and the traffic graph disagree.

Does deleting a log group reduce my bill immediately?

It stops future storage charges for that data and nothing else. Ingestion already billed is spent, and deleting the group does not stop whatever was writing to it, which will simply recreate it. Turn off the source first, then delete.

What counts as one custom metric for billing?

One unique combination of namespace, metric name and dimension values. The same metric name published with three different dimension value sets is three billable metrics. This is the single most important sentence in CloudWatch pricing and the easiest one to skim past.

Should I switch log groups to the Infrequent Access class?

For logs you keep but rarely read, it is usually a straightforward saving on ingestion. For any group you extract metrics from, alarm on, or stream live, it is not a candidate. The feature gap between the classes has narrowed since launch, so check current documentation rather than ruling it out on old information.

Is Logs Insights charged per query or per gigabyte?

Per gigabyte scanned. Rows returned are irrelevant. A tightly filtered query over a wide range across many log groups is expensive; a broad query over fifteen minutes on one log group is cheap. Narrow the range before you narrow the filter.

Do VPC Flow Logs have to go to CloudWatch?

No. Flow logs can be delivered to S3 instead, avoiding CloudWatch ingestion and storage entirely, at the cost of losing Insights queries over them. If you analyse flow logs occasionally with Athena rather than continuously in the console, S3 is the better destination.

The one thing to take away

CloudWatch costs are not one number with one cause. They are four meters running independently, and each has a different fix: cut verbosity at the source for ingestion, set retention for storage, narrow scope for queries, bound your dimensions for metrics. Applying the wrong fix to the wrong meter is why so many cost-reduction efforts produce a bill that barely moves.

So attribute first. Split by usage type, rank log groups by IncomingBytes, count custom metrics per namespace. Ten minutes of that tells you which lever you are actually pulling, and stops you deleting the log group that turns out to be the only place a production alarm was getting its data.

Need a hand with your CloudWatch spend?

This is well-defined work with a measurable outcome, which makes it a good fit for a short engagement. Things I can help with:

• Breaking your CloudWatch bill down by meter and by log group, so you know what you are paying for before anything changes
• Auditing custom metric cardinality and identifying which dimensions are multiplying your series count
• Building a retention policy that satisfies your compliance requirement without paying CloudWatch rates for archival data
• Enforcing retention and log class in Terraform or CloudFormation so the fix survives the next deployment
• Designing a subscription filter and S3 archive path for logs you need to keep but rarely query
• Reviewing alarms and dashboards for anomaly detection and API polling costs that are not earning their keep

If you want a second opinion, send me your Cost Explorer breakdown grouped by usage type, or the output of describe-log-groups for the account. That is usually enough to say where the money is going before we talk about anything else.

The post The Real Cost of CloudWatch: Logs, Metrics, Retention and Cardinality appeared first on John Nessime.

That gap is the entire problem. Run status tells you the code executed. It tells you nothing about whether data arrived. Data pipeline freshness monitoring is the signal that closes the gap, and most teams add it only after an analyst finds the stale table first.

This post covers the three signals worth instrumenting for a batch or streaming pipeline: how current the data is, how long it took to get there, and how often the thing breaks. For each one I’ll cover what to measure, where to measure it, the queries and config that produce it, and the specific ways the measurement lies to you.

The signal that lies to you first: run status

Job success is a statement about your code path, not about your data. A pipeline that reads an empty S3 prefix, transforms zero rows, writes zero rows and exits cleanly has succeeded. So has one whose upstream API silently started returning an empty page after a token rotation. So has one whose incremental watermark got stuck and now re-reads the same already-loaded slice on every run.

All three are green. All three are producing stale data. This is why the ordering matters: freshness is the signal that fires first in a real incident, and run status is usually the last one to notice anything at all.

So keep run status. It’s cheap and it catches crashes. Just stop treating it as your top-line health indicator.

Freshness: measure the table, not the job

Freshness is the age of the newest record in a dataset. It’s measured against the dataset itself, not against the process that filled it, and that distinction is what makes it useful. If your job disappears entirely, freshness keeps climbing and keeps alerting. If your job succeeds while doing nothing, freshness keeps climbing and keeps alerting. It’s the one signal that survives both failure modes.

Two clocks, and you need both

Every row usually carries two timestamps: when the event happened in the source system, and when your pipeline wrote it. Track the maximum of each.

• Max event time tells you whether the upstream system is still producing. If this stops moving, the problem is upstream of you.
• Max load time tells you whether your pipeline is still writing. If this stops moving while event time is fine, the problem is yours.

Watching only one of them means every stale-data page starts with twenty minutes of figuring out which side of the boundary the fault sits on. Watching both answers that in the alert body.

A freshness probe is a small scheduled query. This one is PostgreSQL syntax, and it returns both clocks plus a volume check in a single round trip:

select
  'orders' as dataset,
  extract(epoch from max(event_ts))  as max_event_ts,
  extract(epoch from max(loaded_at)) as max_loaded_at,
  count(*) filter (where loaded_at >= now() - interval '1 hour') as rows_last_hour
from analytics.orders;

The row count matters. A pipeline can advance its load timestamp while writing almost nothing, which is what a partially broken source looks like. Freshness alone will not catch that; freshness plus volume will.

Run that probe on a schedule that’s independent of the pipeline, push the results as gauges, and alert on age in PromQL:

# Newest row is older than 90 minutes
time() - max by (dataset) (dataset_max_loaded_timestamp_seconds) > 5400

# The probe itself has stopped reporting: a deadman check
absent(dataset_max_loaded_timestamp_seconds{dataset="orders"})

That second rule is the one people forget. A freshness metric that disappears looks identical to a healthy silence on a graph. absent() is what turns “no data” into a page.

Where dbt fits

If you’re already running dbt, source freshness is built in and worth using before you write anything custom. You declare thresholds per source, and dbt queries the maximum of your timestamp column and compares it to now.

sources:
  - name: raw_shop
    schema: raw
    config:
      loaded_at_field: _loaded_at
      freshness:
        warn_after: {count: 2, period: hour}
        error_after: {count: 6, period: hour}
    tables:
      - name: orders
      - name: refunds
        config:
          freshness:
            warn_after: {count: 12, period: hour}
            error_after: {count: 24, period: hour}

Three details that trip people up. First, dbt source freshness is a separate command; dbt build does not run it, so a green build says nothing about source staleness. Second, it exits non-zero when a source hits its error_after threshold, which makes it a natural gate at the top of a job: fail fast rather than building models on stale input. Third, results land in target/sources.json, which is the artifact you parse if you want to distinguish warn from error, or ship the numbers into Prometheus rather than just failing the run.

dbt source freshness --select source:raw_shop --output target/freshness.json

One caveat on the YAML above: dbt moved these keys under a config: block in recent releases, and loaded_at_field followed later. Older projects nest them directly under the source. Check what your project’s version expects before copying, because a misplaced key fails quietly by simply not calculating freshness at all.

Latency: name the clock before you name the number

“Our pipeline latency is twelve minutes” is meaningless until you say which two points you measured between. There are at least three plausible definitions, and teams routinely argue past each other because they’re each using a different one.

• Run duration. Start to end of the job. Easy, and mostly useless for anyone downstream.
• Ingestion latency. Source extract to target load. This is what you control.
• End-to-end latency. Event time to the moment the row is queryable. This is what the business actually feels, and it includes queue wait, scheduler delay, and every upstream hop you don’t own.

Publish end-to-end as the headline and keep the others as breakdown. If you only track run duration, a scheduler backlog that delays every run by forty minutes is completely invisible to you: each individual run still takes eleven minutes.

Instrument per stage with a histogram so you can ask percentile questions later without re-instrumenting:

histogram_quantile(
  0.95,
  sum by (le, stage) (rate(pipeline_stage_duration_seconds_bucket[6h]))
)

Use the median for capacity planning and p95 or p99 for the SLO. Averages hide the exact tail that generates the complaints, and on a pipeline that runs a few dozen times a day, a single pathological run is a real fraction of your day.

For streaming, the equivalent is consumer lag expressed in time rather than offsets. Offset lag of fifty thousand messages means nothing without a rate; two hundred seconds of lag means something to everyone. Kafka Lag Exporter popularised this by interpolating a time estimate from observed offset and timestamp samples, exposing kafka_consumergroup_group_max_lag_seconds alongside the offset-based kafka_consumergroup_group_lag. Worth knowing: that project’s repository has been archived and is read-only, so if you’re starting fresh, check whether your broker vendor or a maintained fork covers it before you deploy something unmaintained into the critical path.

Failure rate: decide what counts as a failure

Failure rate is trivially easy to compute and surprisingly easy to compute wrongly. The denominator and the definition both need a decision.

Count runs, not tasks. A DAG with sixty tasks where one flaps on a transient network error looks catastrophic at task level and fine at run level. The run is the unit the consumer cares about.

Count a run that succeeded on its third retry as a success for availability and a failure for a separate reliability metric. Both are true and they answer different questions. If your only metric folds retries into success, you will never see the slow degradation of an upstream API until it stops responding entirely.

sum by (pipeline) (rate(pipeline_runs_total{result="failure"}[6h]))
/
sum by (pipeline) (rate(pipeline_runs_total[6h]))

And add a category label for the failure reason at push time: source unavailable, schema mismatch, permission denied, timeout, validation failed. The rate tells you something is wrong. The category tells you who to wake up. Without it, every failure alert costs you a log dive before you can even route the incident.

Getting metrics out of a job that exits

Prometheus scrapes. Batch jobs finish and vanish. Pushgateway bridges that: the job pushes before exiting, and Pushgateway holds the values for Prometheus to scrape on its own schedule.

#!/usr/bin/env bash
set -euo pipefail

JOB="orders_load"
PGW="http://pushgateway.internal:9091"

start=$(date +%s)
if python /opt/pipelines/load_orders.py; then
  result=0
else
  result=1
fi
end=$(date +%s)

{
  cat <<EOF
# TYPE pipeline_run_duration_seconds gauge
pipeline_run_duration_seconds $((end - start))
# TYPE pipeline_last_run_timestamp_seconds gauge
pipeline_last_run_timestamp_seconds $end
# TYPE pipeline_last_run_success gauge
pipeline_last_run_success $((1 - result))
EOF
  if [ "$result" -eq 0 ]; then
    cat <<EOF
# TYPE pipeline_last_success_timestamp_seconds gauge
pipeline_last_success_timestamp_seconds $end
EOF
  fi
} | curl --fail --data-binary @- "$PGW/metrics/job/$JOB"

exit $result

The conditional block is doing real work. curl’s --data-binary issues a POST, and a POST to Pushgateway replaces only the metrics whose names appear in the payload, leaving the rest of the group intact. So a failing run updates the run timestamp and the success flag while leaving the previous pipeline_last_success_timestamp_seconds exactly where it was. That’s what lets time() - pipeline_last_success_timestamp_seconds keep climbing across consecutive failures. Send a PUT instead and you replace the whole group, wiping the value you needed.

On the Prometheus side, one setting is not optional:

scrape_configs:
  - job_name: pushgateway
    honor_labels: true
    static_configs:
      - targets: ['pushgateway.internal:9091']

Without honor_labels: true, Prometheus overwrites the job label your pipeline pushed with the scrape job’s own name, and every pipeline in your estate collapses into one indistinguishable series called pushgateway.

The trap worth internalising: Pushgateway never forgets. Metrics persist until something explicitly deletes them or the process restarts. A pipeline you decommissioned last quarter is still cheerfully reporting a success timestamp and a duration, and it looks alive on every dashboard. Treat Pushgateway as a cache of past executions, not a picture of current state, and use the push_time_seconds gauge that Pushgateway attaches to each group to tell the difference between a fresh push and a fossil.

If you’re on Airflow, the SLA feature is gone

This one will bite anyone upgrading. The sla and sla_miss_callback parameters were removed in Airflow 3.0, and the replacement, Deadline Alerts, arrived in 3.1. DAGs carrying the old configuration need manual migration; they don’t quietly keep working.

from datetime import timedelta

from airflow.sdk import AsyncCallback, DAG, DeadlineAlert, DeadlineReference
from airflow.providers.slack.notifications.slack_webhook import SlackWebhookNotifier

with DAG(
    dag_id="orders_load",
    deadline=DeadlineAlert(
        reference=DeadlineReference.DAGRUN_QUEUED_AT,
        interval=timedelta(minutes=45),
        callback=AsyncCallback(
            SlackWebhookNotifier,
            kwargs={"text": "orders_load has not finished 45 minutes after queuing."},
        ),
    ),
):
    ...

Note the reference point. Measuring from when the run was queued, rather than from its logical date, means scheduler backlog counts against the deadline. That’s usually what you want, because a run that sat in a queue for an hour is late to its consumers regardless of how fast it executed once it started.

There’s a structural weakness here worth naming: any alert that lives inside the orchestrator dies with the orchestrator. If the scheduler is down, nothing evaluates your deadline and nothing notifies anyone. The freshness probe from earlier is the answer, and it needs to run somewhere else. A small VPS from a provider like Contabo or InterServer running Prometheus and Alertmanager, or a hosted option like Grafana Cloud, gives you a watcher outside the blast radius of the thing being watched. This is the single highest-value piece of monitoring most data teams are missing.

Alerting without burning your on-call

The fastest way to make all of this worthless is to alert on every threshold crossing. Three rules keep it survivable.

1. Set thresholds from observed behaviour, not from wishes. Look at a month of actual freshness values and set warn_after a comfortable margin above the normal worst case. A source that habitually passes at eleven hours against a twelve-hour threshold is not healthy, it’s one upstream hiccup from paging you.
2. Page on consumer impact, not internal events. A task retry is not an incident. A dataset breaching the freshness commitment its consumers rely on is. Route everything else to a channel someone reads in the morning.
3. Use burn rate over multiple windows for SLOs. A short window catches fast breakage, a long window catches slow erosion, and requiring both to fire filters out the transient spikes that generate most false pages.

One more: alert on the absence of your own telemetry. A freshness gauge that stops updating is indistinguishable from one that’s fine, right up until someone asks about the numbers.

Troubleshooting

Freshness alert fires but the data looks current. Almost always a timezone problem. Your loaded_at column is in local time, now() is in UTC, and the offset shows up as a constant bias in the age. Store load timestamps in UTC and cast explicitly at read time.

Freshness flaps in and out of breach. Your check runs too close to the expected arrival. If loads land around six and your probe runs at five past, a fifteen-minute upstream delay produces an intermittent failure that trains everyone to ignore the alert. Move the probe later or widen the threshold.

Every pipeline reports as one series. honor_labels: true is missing from the Pushgateway scrape config.

A decommissioned pipeline still shows healthy. Stale group in Pushgateway. Delete the group and add a check on push_time_seconds so the next one surfaces on its own.

Latency looks fine but consumers say data is late. You’re measuring run duration and they’re feeling end-to-end. Add queue wait and event-time-to-load and the gap will be obvious.

Freshness passes, row count is near zero. The load timestamp advanced without meaningful data. This is a broken source or a stuck watermark, and it’s the reason the volume check belongs in the same probe.

Common mistakes

• Treating job success as the health signal and discovering staleness through a human.
• Running the freshness check inside the same pipeline it’s meant to police.
• Tracking freshness without volume, so a zero-row load reads as healthy.
• Reporting latency without saying which two timestamps it spans.
• Putting run IDs, batch IDs or timestamps into Prometheus labels, which multiplies your series count without bound. Those belong in traces or logs.
• Alerting on task-level failures instead of run-level outcomes, then muting the whole channel a week later.
• Assuming Pushgateway reflects current state rather than the last thing anyone pushed.

Best practices for data pipeline freshness monitoring

• Start with your three most-used tables. Full coverage is a project; three tables is an afternoon and catches most of the pain.
• Write down the freshness commitment per dataset in plain language, then encode it. “Yesterday’s orders are complete by 07:00” converts directly into a threshold.
• Name metrics consistently across pipelines. A shared prefix and a stable label set is what makes one dashboard work for all of them.
• Keep labels low-cardinality: pipeline, dataset, stage, environment, result. Nothing unbounded.
• Emit a failure category alongside every failure so alerts route themselves.
• Run the deadman check on infrastructure that doesn’t share a failure domain with the pipeline.
• Link every alert to a runbook that names the owner and the first three things to check.

Frequently asked questions

What is the difference between pipeline latency and data freshness?

Latency measures how long a specific batch of data took to travel from source to destination. Freshness measures how old the newest available record is right now, regardless of whether anything is currently running. A pipeline can have excellent latency and terrible freshness if it stopped being triggered.

How often should freshness checks run?

Frequently enough that you find out before your consumers do. A useful rule is roughly a quarter of your tolerance window: if data may be up to four hours old, check hourly. Freshness probes are cheap single-aggregate queries, so the limiting factor is usually warehouse billing rather than load.

Do I need a data observability platform for this?

Not to start. Freshness, latency and failure rate for a handful of critical datasets is a scheduled query, a push, and a few alert rules. Commercial platforms earn their cost when you need automatic column-level lineage, anomaly detection across hundreds of tables, or coverage of assets nobody has explicitly instrumented. That’s a real problem at scale, and a genuinely expensive one to build yourself. It’s just not the problem you have on day one.

How do I monitor freshness for a table that only updates weekly?

Set the threshold from the schedule plus a delivery margin, and add a deadman rule so a missing metric alerts on its own. For genuinely static reference tables, disable freshness explicitly rather than leaving a check that always warns; in dbt that means setting freshness to null for the table.

Should freshness checks fail the pipeline or just warn?

Both, in different places. Checking source freshness at the start of a job and failing hard prevents you building models on stale input, which is the cheapest bug to prevent and the most expensive to unwind. Checking output freshness after the fact should alert rather than fail, because the run is already over.

What percentile should I use for a latency SLO?

p95 for most internal analytics pipelines, p99 where downstream systems make automated decisions on the data. Track the median separately for capacity planning. Never use the average as the SLO number, because it hides exactly the tail that produces complaints.

Does this work for streaming pipelines too?

Yes, with different plumbing. Freshness becomes consumer lag measured in time, latency becomes event-time to availability-time, and failure rate becomes connector and task state plus dead-letter volume. The reasoning is identical; only the source of the numbers changes.

Conclusion

If you remember one thing, make it this: measure the data, not the job. Run status, task counts and duration all describe your code. Only freshness describes what your consumers actually receive, and it’s the one signal that stays honest when the pipeline succeeds at doing nothing.

Good data pipeline freshness monitoring is not a platform purchase. It’s a scheduled query that reports the age and volume of your most important tables, pushed somewhere durable, with a deadman rule so that silence is treated as a failure rather than as health. Add latency broken down by stage and failure rate categorised by reason, and you can answer “is the data good right now” without opening a single log file.

Need help instrumenting your pipelines?

Most of my consulting work in this area is retrofitting observability onto pipelines that already exist and can’t be paused. Specifically:

• Defining freshness and latency commitments per dataset, then translating them into thresholds and alert rules that hold up on-call.
• Adding freshness and volume probes to existing warehouses without touching the pipelines themselves.
• Wiring batch jobs into Prometheus through Pushgateway, including the grouping-key and stale-metric problems that bite six months later.
• Migrating Airflow DAGs off the removed SLA feature onto Deadline Alerts, or onto external checks that survive a scheduler outage.
• Building the Grafana dashboard that answers “is the data good right now” in one screen, with drill-down by stage and failure category.
• Cutting alert noise on pipelines where the channel has already been muted, by moving from task-level events to consumer-impact SLOs.

If you want a concrete starting point, send me a DAG file, a scrape config, or a screenshot of the dashboard you don’t trust, and I’ll tell you what I’d instrument first and why.

The post Green Pipeline, Stale Data: Tracking Latency, Freshness and Failure Rates appeared first on John Nessime.

That’s the signature failure of telemetry pipelines on AWS. Transport worked, authentication worked, the endpoint answered 200. It kept part of the payload and dropped the rest, because something in the batch broke a limit your collector has no idea exists.

Getting started with OpenTelemetry on AWS is not the hard part. AWS Distro for OpenTelemetry (ADOT) will have you shipping traces in an afternoon. The hard part is four decisions underneath it: where the collector sits, which door telemetry uses to get into AWS, what happens to your metrics on the way through, and whether a second backend later costs you a config line or a migration. This post covers those four decisions and how each one fails without announcing itself.

What ADOT actually buys you

ADOT is not a separate protocol or agent. It’s a downstream build of the upstream OpenTelemetry Collector plus AWS-flavoured SDKs, tested by AWS and covered by AWS Support. The config syntax is the upstream syntax, so a config written for vanilla OTel runs on ADOT and vice versa. The difference is which components are compiled in: ADOT ships the X-Ray exporter, the CloudWatch EMF exporter, the SigV4 extension, and ECS metric receivers already there.

So the choice is narrower than it looks. Take ADOT for a build someone else validates and supports. Take upstream Contrib if you need a component ADOT hasn’t bundled, or you run one collector image across AWS, another cloud, and bare metal. Neither choice touches your instrumentation, which is the entire point.

Decision one: where the collector runs

Three shapes, trading the same three things: blast radius, cost, and how much processing happens before data leaves your network.

Sidecar

One collector container per ECS task or pod. The application talks to localhost, so no service discovery and no network hop to get wrong, and failure stays contained to one workload. The cost: you pay for that container everywhere, batches stay small because batching is per-instance, and any config change is a redeploy of every task.

Agent plus gateway

A light collector per node (a DaemonSet on EKS) forwarding to a small pool of gateways. This is the shape I reach for first on anything past a handful of services. The agent does host-level enrichment, the gateway does the expensive work: large batches, tail sampling, redaction, fan-out. Config changes hit the gateway only.

The catch: a gateway pool is a thing you now operate. It needs autoscaling and its own alerting, and undersized it starts refusing data at exactly the moment you have an incident and volume spikes.

No collector at all

The ADOT SDKs can export straight to the CloudWatch OTLP endpoints, signing requests with credentials already available to the process. For a handful of Lambda functions this is the right answer: nothing to run, nothing to scale, no extra container in the cold start path. What you give up is control. Batching, retries, sampling policy, redaction, and backend routing all move into application processes and their environment variables, so changing any of them is a fleet redeploy rather than a config push. Good starting point, poor steady state.

Decision two: which door into AWS

Two ways to hand telemetry to AWS, and they behave very differently.

The older path uses AWS-specific exporters. awsxray converts OTLP spans into X-Ray segment documents and calls the X-Ray API. awsemf converts OTLP metrics into CloudWatch Embedded Metric Format and writes them as log events. Both work, both are well trodden, and both reshape your data into an AWS-native format on the way out.

The newer path is native OTLP. CloudWatch exposes OTLP endpoints per signal, reached with the plain otlphttp exporter and a SigV4 signer. Same exporter you’d point at Grafana Cloud or Honeycomb, different URL and authenticator. That’s what “export anywhere” means in practice, and it’s why I default to it on new work.

The endpoints follow a per-service pattern:

• Traces: https://xray.<region>.amazonaws.com/v1/traces
• Metrics: https://monitoring.<region>.amazonaws.com/v1/metrics
• Logs: https://logs.<region>.amazonaws.com/v1/logs

Three constraints there will each cost you an afternoon if you don’t know them going in.

HTTP only. No gRPC. If your collector exports over otlp on 4317 you can’t just change the URL, you have to switch exporters. Receiving gRPC from applications is fine, it’s the outbound leg that must be HTTP.

SigV4 required. That means the sigv4auth extension, configured per signal because the signing service name differs: xray, monitoring, and logs respectively. Bearer tokens are an alternative for metrics and logs, but not for traces.

Logs need headers, not just a URL. Target log group and stream travel in x-aws-log-group and x-aws-log-stream headers. Omit them and the request has nowhere to land.

A working shape for traces and logs looks like this:

extensions:
  sigv4auth/traces:
    region: "us-east-1"
    service: "xray"
  sigv4auth/logs:
    region: "us-east-1"
    service: "logs"

exporters:
  otlphttp/traces:
    compression: gzip
    traces_endpoint: https://xray.us-east-1.amazonaws.com/v1/traces
    auth:
      authenticator: sigv4auth/traces

  otlphttp/logs:
    compression: gzip
    logs_endpoint: https://logs.us-east-1.amazonaws.com/v1/logs
    headers:
      x-aws-log-group: MyApplicationLogs
      x-aws-log-stream: default
    auth:
      authenticator: sigv4auth/logs

service:
  extensions: [sigv4auth/traces, sigv4auth/logs]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp/traces]
    logs:
      receivers: [otlp]
      exporters: [otlphttp/logs]

Note traces_endpoint and logs_endpoint rather than endpoint. This one bites people. The endpoint setting is a base URL and the exporter appends the signal path itself, so endpoint: https://monitoring.us-east-1.amazonaws.com/v1/metrics actually requests /v1/metrics/v1/metrics. The per-signal settings take a full path. If you’re getting 404s from a URL you’re certain is right, this is why.

Also worth a comment in your config: upstream has renamed the component to otlp_http and otlphttp is now a deprecated alias scheduled for removal. Both work today.

Traces need Transaction Search turned on first

This is the single most common reason a correctly configured trace pipeline produces nothing. The X-Ray OTLP endpoint requires Transaction Search to be enabled on the account, which redirects span ingestion into CloudWatch Logs:

aws xray update-trace-segment-destination --destination CloudWatchLogs

Spans then land in a log group named aws/spans, with a percentage indexed in X-Ray as trace summaries for search. The default index rate is one percent, enough to find traces while all spans stay queryable as structured logs. The caller needs xray:UpdateTraceSegmentDestination and xray:UpdateIndexingRule plus log group creation rights.

Two consequences to plan for. Span ingestion is billed separately from log ingestion, so it’s a line item rather than a rounding error. And AWS recommends always_on sampling in the SDK on this path, because the indexing rule already handles reduction. Sample in both places and your service map goes patchy.

The X-Ray SDK clock is running

The dates matter here, so plainly: the X-Ray SDKs and daemon entered maintenance mode on 25 February 2026 (security fixes only, no new instrumentation support), with end of support on 25 February 2027. The X-Ray service is fine and still gaining features. It’s the client libraries and the UDP daemon winding down. If you run aws-xray-sdk and a daemon sidecar, that is technical debt with a published expiry, and either the collector or the CloudWatch agent replaces the daemon.

Decision three: metrics, temporality, and the cardinality bill

Metrics are where the silent drops live, because collector defaults and endpoint limits disagree.

The metrics endpoint caps a single request at 1 MB uncompressed and 1,000 datapoints, counted across resource, scope, and metric levels combined. The upstream batch processor’s default is far larger. Leave it alone and you build oversized requests, and the response to an oversized or partially invalid request is not always a clean failure: it can come back 200 with some metrics accepted and others rejected or throttled. So set the batch size deliberately. AWS’s own examples use a conservative value:

processors:
  batch:
    send_batch_size: 200
    timeout: 10s

The other limits shape attribute design more than config:

• 150 labels maximum across resource, scope, and datapoint attributes per datapoint
• 40 KB combined label and value size per series per datapoint
• One million new series creatable per ten-minute window, per account
• Timestamps no more than ten minutes in the future or fourteen days in the past

That new-series ceiling catches teams out. Attach a request ID, a session ID, or a raw URL path to a metric attribute and every request mints a fresh series. You’ll hit a million faster than you expect, and it presents as metrics randomly going missing rather than a quota error. Unbounded identifiers belong on spans and logs. Metric attributes should be values you could enumerate on a whiteboard.

Temporality is the other conscious decision. The OpenTelemetry SDK spec defaults to cumulative, while CloudWatch’s metric model is delta-shaped: it wants what happened this period, not the total since your process booted. Counters that look like ever-climbing staircases instead of rates are that mismatch. The cumulativetodelta processor converts in the pipeline, keeping the decision in the collector rather than scattered across SDK environment variables in every service.

Decide once and centrally either way. Mixed temporality across one account produces dashboards that are subtly wrong for months before anyone notices. And once volume climbs, pointing a cost tool such as Vantage or CloudZero at the CloudWatch line items earns its setup time, because telemetry spend grows in steps nobody approved.

Decision four: a second backend without a second instrumentation

Here’s the payoff. A collector pipeline holds multiple exporters, and adding one is a config change:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/traces, otlphttp/vendor]

The same spans now reach CloudWatch and Grafana Cloud, Honeycomb, Datadog, or a self-hosted Tempo, with no application redeploy and no second agent. That’s the cashable value of instrumenting once against an open protocol: the switching cost of a backend drops from a migration project to a pull request.

Be honest about the price. You pay ingest twice, and egress from the gateway is real money at volume. Collector memory scales with exporter queue count, so a fan-out gateway needs headroom. And if one backend slows, backpressure can reach the pipeline feeding the other, which argues for separate pipelines per destination when their reliability differs.

It works in reverse too. Because the metrics and logs endpoints accept bearer tokens, a machine with no AWS credentials at all can ship into CloudWatch: a CI runner, another cloud, or a VPS at a provider like Contabo or InterServer running part of your stack. One collector config, one destination, regardless of who owns the hardware. Traces are the exception and still require SigV4.

Why OpenTelemetry on AWS fails quietly

Most breakages here don’t throw. How to recognise the common ones:

Data is missing but nothing errors

Almost always a limit breach inside an accepted request. Check batch sizes against the per-signal caps first, then attribute counts. Turn on the collector’s internal telemetry and compare the exporter’s sent counters against its failed counters. If sent looks healthy and data is still missing, the loss is happening server-side after acceptance, which narrows it to limits.

403 with a signature mismatch

A SigV4 problem, not an IAM problem. The usual cause is the wrong signing service name in the extension, since xray, monitoring, and logs are not interchangeable. If you sign requests yourself rather than using the extension, note that the traces endpoint is stricter about which headers land in the signed set, so sign a minimal stable set rather than everything the HTTP client added.

Backfilled data vanishes

Every endpoint enforces a timestamp window, and fourteen days in the past is the outer edge for all three signals. Replaying an old queue past that boundary gets rejected. Not a bug, but it looks like one when the replay job appears to succeed and produces nothing.

Traces arrive but the service map is empty

Usually a missing or inconsistent service.name resource attribute, or double sampling between the SDK and the indexing rule. Confirm what’s actually leaving the collector before hunting in the console: add the debug exporter to a copy of the pipeline, set verbosity: detailed, and read the resource attributes on real spans.

Common mistakes

• Pointing an OTLP gRPC exporter at a CloudWatch endpoint. They’re HTTP only, and the error won’t say so clearly.
• Using endpoint with a full signal path instead of the per-signal traces_endpoint, metrics_endpoint, or logs_endpoint.
• Leaving the batch processor at its default size on the metrics pipeline.
• Sampling in the SDK and relying on Transaction Search indexing rules, halving visibility twice over.
• Putting request IDs, user IDs, or raw paths into metric attributes.
• Running a gateway pool with no alerting on the collector’s own health, so saturation is invisible until you need it.

Best practices

• Keep instrumentation vendor-neutral. Plain OpenTelemetry SDK APIs, with the collector owning every AWS-specific decision.
• Set service.name, service.version, and deployment.environment as resource attributes everywhere. Almost every correlation feature depends on them.
• Alert on exporter failure counters and queue depth, and route those alerts somewhere that doesn’t depend on the pipeline being healthy.
• Version the collector config in Git and deploy it like application code, with a staging pipeline you can break safely.
• Redact in a processor before export. Once telemetry reaches a backend, removing it is a support ticket.
• Test a second exporter early, even to a throwaway account. A vendor migration is the wrong time to learn the fan-out path doesn’t work.

Frequently asked questions

Do I still need a collector if CloudWatch accepts OTLP directly?

Not to get data in, no. You need one to control what happens before it leaves: batching to fit endpoint limits, tail sampling, redaction, and fan-out. Small serverless estates can reasonably skip it. Anything with a dozen services will want one.

Can I send OTLP to CloudWatch over gRPC?

No. The CloudWatch OTLP endpoints are HTTP 1.1 only, accept binary or JSON payloads, and support gzip or no compression. Your collector can still receive gRPC from applications, it just can’t forward over it.

What’s the difference between ADOT and the upstream collector?

Same codebase, different build. ADOT is AWS’s tested distribution with AWS components bundled and AWS Support behind it; upstream Contrib carries a wider component set and moves faster. Configuration is compatible either way, so it’s a support and packaging choice, not an architectural one.

Do I have to enable Transaction Search to send traces?

For the X-Ray OTLP endpoint, yes. It’s a prerequisite, and enabling it routes span ingestion through CloudWatch Logs. Using the awsxray exporter against the classic X-Ray API instead doesn’t require it, but you give up the span-level analytics Transaction Search provides.

Can I ship telemetry to CloudWatch from outside AWS?

For metrics and logs, yes, using bearer token authentication instead of SigV4, which removes the need for AWS credentials on the host. Traces still require SigV4. Never hardcode the token in the config; read it from a mounted secret file or an injected environment variable.

Is it urgent to migrate off the X-Ray SDK?

Not an emergency, but it’s on a clock: maintenance mode now, end of support 25 February 2027. Existing applications keep working, they just won’t get new library instrumentation. Plan the migration on your schedule rather than someone else’s.

The one thing worth remembering

A pipeline that returns 200 is not a pipeline that works. Judge every design decision in OpenTelemetry on AWS against one question: when this drops data, do I find out from a dashboard or from a colleague asking why a service vanished off the map?

Size batches against the published limits, keep unbounded identifiers off metric attributes, decide temporality in one place, and instrument with plain OpenTelemetry so the export target stays a config line. Do that and “instrument once, export anywhere” stops being a slogan and becomes a property you can test.

Need help with your OpenTelemetry pipeline on AWS?

Most of this work is unglamorous and specific, which is why it gets deferred. Things I can help with:

• Auditing an ADOT or upstream collector config for silent drops, oversized batches, and limit breaches
• Designing collector topology for ECS, EKS, or Lambda, including gateway sizing and autoscaling
• Migrating X-Ray SDK and daemon workloads to OpenTelemetry ahead of end of support
• Cutting telemetry spend through sampling policy, cardinality control, and attribute pruning
• Dual export to CloudWatch and a third-party backend, so a future switch is a config change
• Meta-monitoring that tells you the pipeline is broken before your users do

Send me a collector config, an exporter log, or a screenshot of the gap in your dashboard and I’ll tell you what I’d look at first.

The post Instrument Once, Export Anywhere: OpenTelemetry on AWS With ADOT appeared first on John Nessime.