Grafana Alert Notifications That Actually Arrive: Email, Slack and Telegram

The rule went red at 02:04 and nobody found out until 09:15. Rule evaluation and notification delivery are two different systems, and only one of them is visible in the UI. A practical setup for Grafana alert notifications to email, Slack and Telegram, plus how to detect the day delivery quietly stops.

Continue ReadingGrafana Alert Notifications That Actually Arrive: Email, Slack and Telegram

Cut AWS Costs Without Breaking Production: A Blast-Radius Playbook

Most AWS cost work goes wrong because it starts with the biggest line item, which is also the riskiest. Order the work by blast radius instead: free networking and storage fixes first, performance envelopes one workload at a time, and commitments last. Includes the commands to find the waste and the cuts that look harmless and are not.

Continue ReadingCut AWS Costs Without Breaking Production: A Blast-Radius Playbook

Monitoring n8n Workflow Failures in Grafana: Why Your Green Dashboard Is Lying

n8n's Prometheus endpoint tells you the process is alive and busy. It does not tell you a workflow stopped producing correct output. This post walks through the three failure families that slip past a default n8n dashboard, the PromQL that catches each one, and how to close the gap with a second source of truth from the executions API.

Continue ReadingMonitoring n8n Workflow Failures in Grafana: Why Your Green Dashboard Is Lying

ALB vs API Gateway: Pick the Limits You Can Live With

Choosing between an Application Load Balancer and API Gateway is not a feature comparison. It is a choice of hard limits, a billing curve and an authentication story. Here is how each one bills you, which constraints surface months later, what ALB's native JWT validation changes, and a decision procedure you can run in ten minutes.

Continue ReadingALB vs API Gateway: Pick the Limits You Can Live With

Backfilling Historical API Data into S3 Without Silent Gaps

A backfill that exits zero can still be missing a week of data, and nothing will tell you. This is a practical guide to the failure families behind silent gaps: pagination drift under a mutating source, retries that duplicate pages, prefix layouts designed for writes instead of reads, the seam where backfill meets live ingest, and the storage class rules that make mistakes expensive. Includes deterministic key derivation, S3 conditional writes, Athena partition projection, and a per-window manifest pattern that turns completeness into something you can query.

Continue ReadingBackfilling Historical API Data into S3 Without Silent Gaps

Troubleshooting High I/O Wait Without Blaming the Disk

High I/O wait is a symptom, not a diagnosis. It rises when nothing is wrong and stays flat when something is badly wrong, and it only sees one layer of the storage stack. This is how to read iowait and PSI correctly, then work through the five failure families that actually produce storage stalls: device saturation, memory pressure, synchronous writes, throttling, and latency that lives above the block layer entirely.

Continue ReadingTroubleshooting High I/O Wait Without Blaming the Disk