The rule was fine. The query was fine. The evaluation ran on schedule and the state transition happened exactly when it should have. What failed was everything after that, and none of it made a sound.

That is the failure mode worth designing around. Rule evaluation and notification delivery are two different systems, and a red rule only proves the first one works. Grafana alert notifications run against SMTP servers, Slack’s API and Telegram’s bot endpoint, all of which fail in their own quiet ways. A rejected login, a revoked webhook, a message over a length limit: each produces a log line and nothing else. The alert list still shows the rule firing, so everything looks healthy from the one place you’d think to check.

This post covers the practical setup for the three channels most small and mid-sized teams actually use: email, Slack and Telegram. It covers what breaks in each of them, how to route alerts so the right ones interrupt people and the rest don’t, how to provision the whole thing as files instead of clicking through the UI, and how to detect the case where delivery has stopped entirely.

Where a firing alert actually goes

Before touching any integration, it helps to know how many hops sit between “the rule fired” and “my phone buzzed”.

• The rule evaluates on its own schedule and produces alert instances, each carrying labels.
• Those instances go to an Alertmanager. Grafana ships its own and can also use an external Prometheus Alertmanager.
• The Alertmanager applies mute timings, groups instances, and walks the notification policy tree for a match.
• The matched policy names a contact point, which holds one or more integrations.
• Each integration makes its own outbound call to a third-party service.

Steps one and two are visible in the Grafana UI. Steps three through five are where things go wrong, and they are visible only in logs. That asymmetry is the whole problem.

One thing worth knowing early: a contact point can hold several integrations at once, so a single contact point named critical can fire email, Slack and Telegram together. Building one contact point per channel instead makes routing much harder than it needs to be.

Email: the channel everyone assumes works

Email is the one people configure first and verify least. The catch is that the email contact point holds only the recipient addresses. The actual SMTP credentials live in grafana.ini, so a perfectly valid contact point can sit there doing nothing because the server underneath was never configured.

[smtp]
enabled = true
host = smtp.example.com:587
user = alerts@example.com
# Wrap the password in triple quotes if it contains # or ;
password = """your-smtp-password"""
from_address = alerts@example.com
from_name = Grafana Alerts
skip_verify = false
# Default is OpportunisticStartTLS
startTLS_policy = MandatoryStartTLS

Several details in that block matter more than they look.

• The port is part of host. There is no separate port setting. Port 587 with STARTTLS is the normal choice; 465 is implicit TLS and behaves differently, which is usually why one fails cleanly and the other hangs.
• startTLS_policy decides how hard Grafana insists on encryption. The default is opportunistic: it upgrades the connection if the server offers it and quietly continues in the clear if not. On a relay you control, mandatory turns a silent downgrade into a visible failure.
• from_address has to be something the relay accepts. Managed relays reject mail from unverified domains after the connection succeeds, which is why the Grafana side looks fine.
• Leave skip_verify false. It exists for internal relays with self-signed certificates. Turning it on to clear an error removes the check that was telling you the certificate is wrong.

Changes to this file need a Grafana restart. Nothing in the UI reloads SMTP settings.

Deliverability is a separate problem from delivery

Sending directly from a VPS is the fastest way to get alerts filed as spam. A fresh IP range from a provider like Contabo or InterServer has no sending reputation, and a mail server that emits one message a week during incidents never builds one.

Point Grafana at a transactional relay instead. Amazon SES, Postmark, Mailgun and Brevo all do the same job at this volume. Whichever you pick, publish SPF and DKIM records for the sending domain and set a DMARC policy, then send a real alert to a Gmail address and a Microsoft 365 address and check the spam folder. That is the only test that tells you anything.

Slack: webhook or bot token

Grafana supports two ways of posting to Slack, and the choice has consequences beyond setup time.

Incoming webhooks are the quick path. Slack generates a URL under hooks.slack.com/services/ that is bound to a single channel, and Grafana posts to it. There’s no OAuth flow and no scopes to reason about. The trade-off is that the URL is the credential and it maps to exactly one channel, so a team with six alert channels ends up managing six secrets with no central view of them.

A Slack app with a bot token takes longer. You create the app, add the chat:write.public scope so it can post to public channels without being invited to each one, then copy the bot user OAuth token, which starts with xoxb-. In Grafana you supply that token plus the channel ID as the recipient. One credential covers every channel, permissions are visible in one place, and revoking it kills all posting at once.

Webhooks win when you have one or two channels and want to be done in five minutes. The bot token wins as soon as channels start multiplying or someone asks what the integration is permitted to do. I reach for the webhook on small setups and regret it around the fourth channel.

Two constraints either way. A Slack integration targets one channel, so several channels means several integrations. And notification templates control the message title and body text, not its visual structure, so rich block layouts aren’t on the table.

Telegram: cheap paging, one nasty edge

Telegram is genuinely useful for out-of-hours alerts because it pushes to a phone without anyone paying for a paging product. The setup is two values: a bot API token and a chat ID.

1. Message BotFather and send /newbot. The username has to end in bot or _bot.
2. Copy the HTTP API token it returns. That’s the bot API token Grafana wants.
3. Create a group for alerts and add the bot to it. A bot cannot message a chat it has never seen.
4. Send a slash command in the group, such as /start@your_bot_name. An explicit command is the reliable way to make the group visible to the bot.
5. Fetch the chat ID from the Telegram API.

curl -s "https://api.telegram.org/bot<YOUR_BOT_TOKEN>/getUpdates" | jq '.result[].message.chat'

The id field in that output is what goes in Grafana’s Chat ID field. Group chat IDs are negative. Copy the minus sign; leaving it off is the single most common reason a Telegram contact point tests as failing.

Now the edge case, and it’s the reason Telegram belongs in a post about invisible failures. Telegram caps messages at 4096 UTF-8 characters. If you’re using a parse mode other than none, truncating at that boundary can leave the markup unbalanced, and Telegram rejects the whole message as malformed. You don’t get a shortened alert. You get nothing.

This is exactly backwards, because the message most likely to blow the limit is a large grouped alert during a real outage. The channel works perfectly for months of single alerts, then goes quiet the first time forty instances fire together. Three ways to deal with it, in the order I’d try them:

• Write a deliberately short notification template for Telegram: alert name, severity, instance count and a dashboard link. Treat it as a pager, not a report.
• Narrow group_by on the policy that feeds Telegram so a single notification covers fewer instances.
• Set the parse mode to none, which makes truncation harmless at the cost of formatting.

Routing: split by severity, not by tool

The most common design mistake is a policy tree organised around channels: a Slack policy, an email policy, a Telegram policy. That gives you no way to express the thing you actually care about, which is that some alerts should wake someone up and most should not.

Put severity on the alert rule as a label and route on that instead. Contact points then become bundles of channels chosen to match urgency.

• severity=critical goes to a contact point holding Telegram and Slack. Something is broken and someone needs to look now.
• severity=warning goes to Slack only, in a channel people read during working hours.
• Everything else falls through to the default policy and lands in email, where it accumulates without interrupting anyone.

Four timing settings control how noisy this is, and each one does something specific:

• group_by decides what counts as the same notification. Grouping by alertname and a service label means one cascading failure produces one message instead of forty.
• group_wait (default 30 seconds) is how long to hold the first notification for a new group, so related instances that fire a few seconds apart arrive together.
• group_interval (default 5 minutes) is the minimum gap before a group that has already notified sends again because new instances joined it.
• repeat_interval (default 4 hours) is how often an unresolved alert re-notifies with nothing new to say.

The temptation is to drop repeat_interval to fifteen minutes so nothing gets forgotten. Resist it. Teams that do this stop reading the channel within a month, which converts a routing problem into a much worse cultural one. Longer intervals plus a real escalation path beats short intervals every time.

Grafana also offers a simpler path: pick a contact point directly on the alert rule form and skip label matching entirely. Reasonable when a rule has one obvious destination. The trade-off is that routing knowledge moves into individual rules, so changing where a class of alerts goes means editing every rule instead of one policy.

Provision it as files, not clicks

Contact points built through the UI are invisible to review, easy to delete by accident, and impossible to recreate after a bad restore. If Grafana alert notifications matter enough to build, they matter enough to keep in Git. Grafana reads alerting configuration from files in the provisioning/alerting directory under its provisioning path at startup.

Before writing any YAML by hand, build one contact point in the UI, test it, then use the export option on the contact points list to dump it as YAML. That gives you the exact setting keys for that integration type instead of the ones you half-remember from a blog post. Setting names are not consistent between integration types, and a guessed key produces a config that loads without error and sends nothing.

apiVersion: 1

contactPoints:
  - orgId: 1
    name: critical-oncall
    receivers:
      # Telegram receiver goes here. Export it from the UI first:
      # the setting key names differ from what you would guess.
      - uid: critical-slack
        type: slack
        settings:
          url: https://hooks.slack.com/services/REPLACE/WITH/YOURS
        disableResolveMessage: false

  - orgId: 1
    name: warnings-slack
    receivers:
      - uid: warnings-slack
        type: slack
        settings:
          url: https://hooks.slack.com/services/REPLACE/WITH/YOURS

  - orgId: 1
    name: email-digest
    receivers:
      - uid: email-digest
        type: email
        settings:
          addresses: ops@example.com
          singleEmail: false

And the policy tree that uses them:

apiVersion: 1

policies:
  - orgId: 1
    receiver: email-digest
    group_by: [grafana_folder, alertname]
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - receiver: critical-oncall
        matchers:
          - severity = critical
        group_wait: 10s
        repeat_interval: 1h
      - receiver: warnings-slack
        matchers:
          - severity = warning
        repeat_interval: 12h

Two things to know about provisioned resources. By default they become read-only in the UI, which is the point but surprises people the first time. And a contact point that a policy references cannot be deleted until the policy stops referencing it, so removals have to happen in the right order.

Keep tokens out of the files themselves. Grafana supports environment variable substitution in provisioning files, which means the YAML can live in Git and the secrets can come from your existing secret store.

Detecting the silence

Everything above still leaves the original problem unsolved. If the Slack app gets revoked or the SMTP password rotates, notifications stop and Grafana does not tell you.

The fix is a heartbeat alert, and it’s worth the twenty minutes. Create a rule whose condition is always true, give it a low severity label, route it to each contact point with a short repeat_interval, and point the resulting notification at an external dead-man’s-switch service. Healthchecks.io, Cronitor and Better Stack all offer this, and Healthchecks.io can be self-hosted if you’d rather not add a dependency. The service expects a signal on a schedule and alerts you when it stops arriving.

The logic is inverted on purpose. You aren’t monitoring infrastructure here, you’re monitoring the path that tells you about your infrastructure, and it has to be watched by something that isn’t itself.

As a second layer, Grafana exposes its own internal metrics on a Prometheus-compatible endpoint, including counters for notification attempts and failures. If you’re already scraping Grafana, alert on that failure counter increasing. It won’t catch a credential that works but delivers to a dead mailbox, which is why the heartbeat comes first.

Troubleshooting: the rule fires, nothing arrives

Work down this list in order. Each step rules out one hop, and stopping early is how people spend an afternoon on the wrong layer.

1. Test the contact point in isolation. The Test button sends a notification without touching rules or policies. If it fails, the problem is credentials or network and nothing else matters yet.
2. Watch the Grafana logs during the test. On a systemd install, run journalctl -u grafana-server -f while you click Test. SMTP, Slack and Telegram delivery errors all surface here, usually specific enough to fix directly.
3. Compare the instance’s labels against your matchers. A policy matching severity = critical never sees severity = Critical. Matchers are exact and case-sensitive. Read the firing instance’s real labels, not the ones you think the rule sets.
4. Look for a policy earlier in the tree that swallowed it. Evaluation stops at the first match unless the policy continues. A broad route near the top silently absorbs everything below it.
5. Check active silences and mute timings. A silence created during last month’s maintenance with a generous matcher is the classic. So is a mute timing in the wrong timezone.
6. Check whether you’re inside repeat_interval. A red alert now does not mean a notification is due now. This looks identical to a broken integration and isn’t.
7. For email, read the relay’s own logs. Grafana handing the message off is not delivery. Bounces and spam filing happen after Grafana’s involvement ends.

Common mistakes

• Configuring the email contact point and never touching grafana.ini. The contact point is only a recipient list. Without the SMTP block, it has nothing to send with.
• Routing by channel instead of severity. It feels tidy and it makes urgency impossible to express.
• Dropping the minus sign from a Telegram group chat ID. Group IDs are negative and the field takes the value literally.
• Sending verbose templates to Telegram. The 4096-character limit turns your biggest incident into your quietest one.
• Testing with the Test button and calling it done. Test bypasses the policy tree entirely. It proves the credential works, not that a real alert would ever reach it.
• Leaving resolved notifications on for every channel. Resolution messages in a pager channel double the volume and train people to ignore it.
• Storing webhook URLs and bot tokens in the UI only. They exist in one place, are invisible to review, and disappear with the database.

Best practices for Grafana alert notifications

• Set a consistent severity label on every rule from day one. Retrofitting labels across dozens of rules is miserable work.
• Use one contact point per urgency level, holding as many integrations as that level needs.
• Put a dashboard or runbook link in an annotation your templates render. An alert that says what broke but not where to look costs the responder five minutes every time.
• Write channel-specific templates: short for Telegram, richer for email, in between for Slack.
• Provision contact points and policies from files, with secrets injected from the environment, and run a heartbeat through every channel.
• Re-test every integration after credential rotation, Slack workspace changes or a Grafana upgrade. Those are the three events that quietly break delivery.

FAQ

Can one contact point send to email, Slack and Telegram at the same time?

Yes. A contact point holds multiple integrations and every one of them fires when that contact point is selected. This is the right way to build an escalation tier: one contact point per severity, with the channel mix chosen to match how urgently that severity needs attention.

Why does the Test button work but real alerts never arrive?

Test sends straight to the contact point and skips the policy tree. If it works and real alerts don’t, the break is in routing: matchers that don’t match, a broader policy earlier in the tree absorbing the alert, an active silence, or a mute timing. Open the firing instance, read its real labels, and compare them character by character against your matchers.

Do I need Grafana Cloud for any of this?

No. Contact points, notification policies, templates and all three integrations ship in the open source build. Grafana Cloud handles hosting and adds on-call tooling on top, which matters if you need rotations and escalation chains. For a team routing alerts to three channels, a self-hosted instance on a modest VPS does everything described here.

Should I use a Slack webhook or a bot token?

Webhook for one or two channels where speed of setup matters. Bot token once you have several channels, want one revocable credential, or need to see in one place what the integration is allowed to do. Both deliver identically; the difference is entirely in credential management.

Why did my Telegram alerts stop during a big outage?

Almost certainly the 4096-character limit. A large grouped notification exceeded it, truncation broke the message markup, and Telegram rejected the send outright. Shorten the template for that channel, group more narrowly, or drop the parse mode. It’s the failure mode most worth guarding against, because it only shows up when you need the channel most.

Can I keep using the Prometheus Alertmanager I already run?

Yes. Grafana can send to an external Alertmanager instead of its built-in one, and the contact point selector at the top of the alerting pages lets you choose which one you’re configuring. If you already have a mature Alertmanager routing config, keeping it and pointing Grafana at it is usually less work than migrating the tree.

The one thing to remember

A firing rule proves evaluation works. It proves nothing about whether anyone was told. Grafana alert notifications cross three or four systems after the rule turns red, and every one of them fails quietly by default.

So build the channels carefully, route by severity rather than by tool, then run a heartbeat through every path and let something outside Grafana watch it. The alternative is finding out during the incident, which is the one time you have no attention to spare.

Need alerting that reaches people?

Most of the Grafana setups I get asked to look at have good dashboards and alerting nobody trusts. Work I take on here:

• Fixing email delivery end to end: SMTP config, relay selection, SPF, DKIM and DMARC so alerts stop landing in spam.
• Setting up Slack and Telegram contact points properly, including bot apps, scoped tokens and channel-appropriate templates.
• Designing a policy tree around severity labels, with grouping and timing tuned so people keep reading the channel.
• Converting click-built alerting into provisioned YAML under version control, with secrets kept out of Git.
• Adding heartbeat monitoring so a broken notification path announces itself instead of waiting for an outage.
• Auditing alert rules for missing labels, missing runbook links, and rules that fire constantly and are therefore ignored.

Send me your notification policy export and the output of journalctl -u grafana-server from a period when an alert should have gone out, and I’ll tell you which hop is dropping it.

The post Grafana Alert Notifications That Actually Arrive: Email, Slack and Telegram appeared first on John Nessime.

That is where most cost work goes wrong. The pressure is on the total, so people attack the biggest line item first. The biggest line item is almost always compute, and compute is the one place where a wrong guess shows up as latency at peak. Three weeks later the bill is lower and the p99 is worse, and nobody connects the two, because the change that caused it was a Tuesday afternoon instance swap that “went fine.”

This post is about how to cut AWS costs without breaking production by ordering the work differently: not by how much each change saves, but by how far the damage travels if the change is wrong. I will go through the levers in that order, with the traps in each one, the commands I use to find the waste, and the handful of cuts that look harmless and are not.

The cuts that break production are rarely the ones you fear

The line items that look most alarming on a bill are usually networking and storage overhead: NAT Gateway processing, public IPv4 hours, log ingestion, snapshots nobody owns. They feel dangerous to touch because they sit in the middle of the VPC. In practice, almost all of them can be removed or rerouted with zero behavioural change to your application. A gateway endpoint for S3 does not alter a single byte your code sees. Deleting an Elastic IP associated with nothing cannot affect anything, by definition.

The changes that feel safe are the ones with real blast radius. Dropping an instance one size. Switching a volume type. Moving objects to a colder storage class. Each is a single API call, reversible on paper, and each quietly narrows a performance envelope you were relying on without knowing it. The failure does not show up at deploy time. It shows up the next time you hit the load level that used to fit inside the old headroom.

So the rule I work by: sequence cost work by blast radius, not by savings size. Take everything production cannot possibly notice first. Then the things that move a performance envelope, one workload at a time, with a metric to watch. Then, last, the financial commitments, which cannot break your uptime at all but can lock you into paying for architecture you are about to throw away.

First, find out what you are actually paying for

The default Cost Explorer view groups by service, which is close to useless here. “EC2 – Other” is not a thing you can act on. It is a bucket holding EBS volumes, snapshots, NAT Gateway hours, NAT Gateway data processing, data transfer and public IPv4 charges, all mixed together.

Group by usage type instead

Usage type names the actual mechanism being billed. Strings like NatGateway-Bytes, NatGateway-Hours, PublicIPv4:InUseAddress and EBS:VolumeUsage.gp2 tell you what to go fix. This pulls last month’s usage types sorted by cost:

aws ce get-cost-and-usage 
  --time-period Start=$(date -d '1 month ago' +%Y-%m-01),End=$(date +%Y-%m-01) 
  --granularity MONTHLY 
  --metrics UnblendedCost 
  --group-by Type=DIMENSION,Key=USAGE_TYPE 
  --output json 
  | jq -r '.ResultsByTime[0].Groups[] | [.Keys[0], .Metrics.UnblendedCost.Amount] | @tsv' 
  | sort -k2 -gr | head -30

That uses GNU date, so run it on Linux unless you have coreutils on macOS. Cost Explorer API calls are themselves billed per request, which surprises people the first time they wrap this in a cron job.

Billing data cannot tell you where your bytes went

This is the most useful thing to internalise about networking spend. Cost data knows how many gigabytes crossed the NAT Gateway. It has no idea whether they were headed to S3, to ECR, to CloudWatch or to a third-party API. There is no destination breakdown in billing data.

VPC Flow Logs are the missing half. Turn them on for the private subnets, then query them grouped by destination address with Athena or CloudWatch Logs Insights and map the ranges back to AWS services. Now you have the split: this much to S3, this much to ECR, this much genuinely to the internet. That is what tells you which endpoint is worth adding. Flow Logs cost money to ingest, so scope them tightly and turn them off once you have the answer. A week on the right subnets is enough.

Tier zero: savings production cannot notice

Everything in this tier changes billing without changing behaviour. Do all of it before you touch a single instance type.

The orphan sweep

AWS charges for allocated resources, not used ones. A volume attached to nothing bills at full rate. An Elastic IP attached to nothing bills by the hour, and has done ever since AWS began charging for public IPv4 addresses whether or not they are in use.

# Elastic IPs associated with nothing
aws ec2 describe-addresses 
  --query 'Addresses[?AssociationId==`null`].[PublicIp,AllocationId]' 
  --output text

# EBS volumes in the "available" state, meaning attached to no instance
aws ec2 describe-volumes --filters Name=status,Values=available 
  --query 'Volumes[].[VolumeId,Size,VolumeType,CreateTime]' 
  --output text

Run the same pass over load balancers with no healthy targets, snapshots whose source volume is gone, and old AMIs still holding snapshots behind them. Snapshot a volume before deleting it. The snapshot costs a fraction of the volume and buys you a rollback if something did care after all.

NAT Gateway is usually the largest risk-free win

A NAT Gateway bills three separate ways: an hourly charge for existing, a per-gigabyte charge for every byte it processes, and then whatever data transfer applies to where that traffic actually went. Teams look at the hourly rate, decide it is a rounding error, and never look at the processing line. The processing line is the one that grows with traffic.

The part that stings: traffic from a private subnet to S3 or DynamoDB in the same region is charged NAT processing even though it never leaves the AWS network. Gateway endpoints for those two services are free, with no hourly and no per-gigabyte charge. If your private subnets talk to S3 through a NAT Gateway, this is the best ratio of saving to risk available anywhere on the platform.

aws ec2 create-vpc-endpoint 
  --vpc-id vpc-0123456789abcdef0 
  --service-name com.amazonaws.eu-west-1.s3 
  --vpc-endpoint-type Gateway 
  --route-table-ids rtb-0123456789abcdef0

A gateway endpoint works by adding a prefix-list route to the route tables you name, so the main failure mode is naming the wrong tables and leaving traffic on the old path. Nothing breaks; you just do not save anything. The real caveat is policy: if you have bucket policies or endpoint policies that restrict access by source, check them first, because requests arriving through an endpoint present differently.

Interface endpoints for other services (ECR, CloudWatch Logs, Secrets Manager, SSM) are a different calculation. They charge per hour per availability zone plus per gigabyte, so they only pay off above a certain volume for that one service. Use the Flow Logs breakdown to decide. Pulling container images from ECR through NAT on every scale-out is the case that usually justifies one.

Log groups that never expire

CloudWatch Logs bills mostly on ingestion and secondarily on stored volume. A log group created without an explicit retention setting keeps data forever, so every debug group someone created during an incident years ago is still accruing storage.

# Log groups with no retention policy set
aws logs describe-log-groups 
  --query 'logGroups[?!retentionInDays].[logGroupName,storedBytes]' 
  --output text

aws logs put-retention-policy 
  --log-group-name /aws/lambda/my-function 
  --retention-in-days 30

Retention is the safe half. Reducing ingestion is a tier-one change, because you are removing evidence you might need during an incident. If you need a long retention window cheaply, exporting to S3 with a lifecycle rule beats paying CloudWatch storage rates indefinitely, and a log platform like Grafana Loki or a hosted equivalent gives you a cheaper long tail while CloudWatch keeps the recent window.

Incomplete multipart uploads

Failed large uploads leave parts behind in S3. Those parts are billed as storage and do not appear in a normal object listing, which is why some buckets bill for far more than they seem to contain. One lifecycle rule ends it:

{
  "Rules": [
    {
      "ID": "abort-incomplete-multipart",
      "Status": "Enabled",
      "Filter": {},
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    }
  ]
}

Apply it with aws s3api put-bucket-lifecycle-configuration. That call replaces the entire lifecycle configuration on the bucket, so fetch the existing one and merge rather than overwrite.

Tier one: savings that move a performance envelope

Now a wrong call has consequences. Everything here goes one workload at a time, with a named metric you watch for at least a full traffic cycle afterwards.

gp2 to gp3, and the throughput trap

This is presented everywhere as a free win, and for most volumes it is. gp3 costs less per gibibyte and gives every volume a baseline of 3,000 IOPS regardless of size, where gp2 scales at 3 IOPS per GiB and only reaches 3,000 sustained at 1 TiB. For a small database volume quietly burning through burst credits, gp3 is a straight upgrade in both cost and stability.

The trap is throughput, not IOPS. gp3’s baseline throughput is 125 MiB/s. A large gp2 volume can deliver up to 250 MiB/s. Migrate a big sequential-read volume to gp3 with the defaults and you halve its throughput ceiling while the cost report still shows a saving. Log processing, backup jobs and analytics scratch space are exactly the workloads that hit this, and exactly the ones where the symptom is “the nightly job got slower” rather than an alert.

Check VolumeReadBytes and VolumeWriteBytes in CloudWatch first, and provision throughput explicitly where the volume needs it:

aws ec2 modify-volume --volume-id vol-0123456789abcdef0 
  --volume-type gp3 --iops 3000 --throughput 250

aws ec2 describe-volumes-modifications --volume-id vol-0123456789abcdef0 
  --query 'VolumesModifications[].[ModificationState,Progress]' 
  --output text

Elastic Volumes changes the type in place with the instance running, so there is no downtime. Two things to watch anyway. If the root volume is declared inline in a CloudFormation block device mapping, changing its type there can replace the instance rather than modify the volume; modify through the EC2 API instead. And there is a cooldown before a volume can be modified again, so plan on getting it right rather than iterating. On RDS the same change also runs without downtime, but the instance then enters a storage optimization state during which further storage changes are blocked.

Rightsizing without guessing

AWS Compute Optimizer reads CloudWatch metrics and returns recommendations with a confidence rating. Better than eyeballing CPU graphs, with one important gap: without the CloudWatch agent installed it cannot see memory. A JVM service sitting at 12% CPU and 88% heap looks like an obvious downsize candidate and is not.

aws compute-optimizer get-ec2-instance-recommendations 
  --query 'instanceRecommendations[?finding==`OVER_PROVISIONED`].[instanceArn,currentInstanceType]' 
  --output text

My working rules:

• Look at a window that includes your worst week, not a rolling average. Averages hide the peak that justifies the size.
• Change one dimension at a time, family or size, so you know what caused any regression.
• Treat burstable families as a separate decision. A T-family instance that exhausts CPU credits under sustained load either throttles or silently bills you for unlimited mode. Neither is what you wanted.
• Moving to Graviton usually saves more than downsizing x86, but it is an architecture change. Images need to be multi-arch and every compiled dependency needs to build. That belongs in a sprint, not in a cost-cutting afternoon.
• Write down the metric you expect to move and the threshold at which you revert, before you make the change.

S3 storage classes and the small-object trap

Intelligent-Tiering is the low-thought option and it works well for what it was designed for: objects of meaningful size with access patterns you cannot predict. It charges a small per-object monitoring fee, moves objects between tiers automatically, and applies no retrieval fee on the automatic access tiers.

Where it fails is object count. Objects below the eligibility threshold of 128 KB are not auto-tiered at all. They sit in the frequent access tier permanently while still paying monitoring. A bucket holding hundreds of millions of thumbnails, metadata records or small JSON documents can pay for monitoring that produces no saving whatsoever. Check average object size first, from S3 Storage Lens or an inventory report: total bytes divided by object count.

Lifecycle transitions have their own trap, since each transition is a billed request. Moving a billion small objects to a colder class can cost more in transition requests than the storage saving returns in a year. Transitions pay off on size, not on count. Expiry is the exception: deleting costs nothing and removes the storage entirely. While you are in there, check non-current versions in versioned buckets, which accumulate invisibly and need their own rule. And if your workload is genuinely egress-heavy, an object store with no egress charge such as Cloudflare R2 changes the arithmetic enough to model, in exchange for cross-provider latency and an operational split.

Tier two: commitments, which break budgets rather than uptime

Savings Plans and Reserved Instances cannot cause an outage. They are billing constructs. What they can do is hold you to a spend rate for one or three years after the architecture that justified it has changed. What I would want anyone to know before signing:

• Savings Plans cannot be cancelled during the term; AWS documentation is explicit. There is a limited return path with eligibility conditions and quotas per billing family, but plan as though there is no exit.
• Standard Reserved Instances can be listed on the Reserved Instance Marketplace, which is a real exit but not a guaranteed one, since somebody has to buy. Convertible RIs cannot be sold, only exchanged for a different configuration.
• Commit after rightsizing, never before. A plan applied to an oversized fleet locks in the waste and removes the incentive to fix it.
• Savings Plans do not reserve capacity. If you need a guarantee that instances will be available in a specific zone, that is an On-Demand Capacity Reservation, a separate mechanism the discount can then apply to.
• Savings Plans do not apply to Spot usage or to usage already covered by an RI. If much of your fleet is Spot, your committable baseline is smaller than the bill suggests.
• Compute Savings Plans survive architectural drift: family changes, size changes and moves between EC2, Fargate and Lambda stay covered. EC2 Instance Savings Plans and Standard RIs discount more deeply and tolerate drift poorly. Choose based on how stable your architecture actually is, not how stable you would like it to be.

Most teams land on a layered shape: commit at the baseline you are confident will still exist in a year, leave the variable top of the curve on demand or on Spot. Flat usage and a frozen architecture justify deeper, longer commitments. Mid-migration, mid-container-adoption or mid-anything, take the smaller discount for the shorter term.

One honest question worth asking here: does the workload need to be on AWS at all? Steady, non-elastic things like build runners, staging environments and internal tooling are often cheaper on a plain VPS from a provider such as Contabo or InterServer, while genuinely elastic workloads are often cheaper on AWS. The trade is operational, because you take back patching, backups and capacity planning that AWS was doing for you.

Troubleshooting: the bill went down and something broke

The hard part of diagnosing a cost regression is the delay. The change lands on Tuesday, the symptom appears at month-end batch time, and by then three other things have shipped. Work backwards from the symptom:

• Latency got worse but CPU is fine. Look at storage. VolumeQueueLength climbing with throughput flat at a round number means you are sitting on a provisioned ceiling, which is the gp3 default-throughput signature. On gp2, a falling BurstBalance is the tell, and it usually starts hours before anything is user-visible.
• Intermittent timeouts after rightsizing. Check whether the new instance type has lower network or EBS bandwidth, not just fewer vCPUs. Baseline bandwidth scales with size, and smaller sizes burst their network the way T instances burst CPU.
• Something stopped reaching a service. If you added VPC endpoints, check whether an endpoint policy is denying it. Interface endpoints also need the endpoint’s own security group to allow inbound from your workload, which is the most common reason a new endpoint appears to break connectivity.
• Objects are slow or missing from an application path. Check whether a lifecycle rule moved them into a class with retrieval latency. The Glacier tiers inside Intelligent-Tiering are opt-in for exactly this reason, and enabling them turns instant reads into restore requests.

Keep a change log for cost work specifically: a dated line per change, what you expected to save, what metric you were watching. Five minutes each, and it is the difference between diagnosing this in an hour and in a week.

Common mistakes

• Starting with the biggest number. Compute is the biggest line and the riskiest to touch. It is the last tier, not the first.
• Committing before rightsizing. This locks in the exact waste you were trying to remove, for years.
• Accepting gp3 defaults on high-throughput volumes. Cheaper storage, halved throughput ceiling, no alert.
• Enabling Intelligent-Tiering on buckets full of tiny objects. Monitoring fees on objects that will never move a tier.
• Turning off monitoring to save on monitoring. Understandable at 2am on a budget deadline, indefensible during the next incident. Cut cardinality and retention, not coverage.
• Doing it once. Cost drifts back within two quarters unless something structural stops it.

Best practices that make the savings stick

• Make cost visible before it is a crisis. AWS Budgets with alerts at a percentage of forecast, plus Cost Anomaly Detection, catch the runaway job on day two rather than on the invoice. Platforms like Vantage, CloudZero or Kubecost for per-namespace Kubernetes attribution earn their place once several teams share an account structure.
• Tag at creation, in the IaC. Tags applied afterwards do not backfill historical cost data. If it was not tagged when it was created, that spend is unattributable forever.
• Put non-production on a schedule. Dev and staging running nights and weekends is the most common waste in any account, and stopping them has no production blast radius at all.
• Make the cheap thing the default in the module. gp3, sensible log retention, gateway endpoints and lifecycle rules belong in your Terraform or OpenTofu modules, so the next environment is born correct rather than remediated later.
• Review commitment coverage on a schedule. Coverage and utilisation are different numbers and both matter. High coverage with low utilisation means you over-committed.

Frequently asked questions

What is the fastest way to cut AWS costs without breaking production?

Add gateway endpoints for S3 and DynamoDB, delete unassociated Elastic IPs and unattached EBS volumes, and set retention on log groups that have none. All four are billing-only changes with no behavioural effect on your application, and together they often account for a large share of the surprise on a bill.

Why is my “EC2 – Other” line item so large?

It is a mixed bucket: EBS volumes and snapshots, NAT Gateway hours and data processing, data transfer, and public IPv4 address charges. Regroup Cost Explorer by usage type rather than by service to see which of those is driving it.

Is migrating from gp2 to gp3 always cheaper?

No. gp3 has a lower per-gibibyte price, but IOPS above the included baseline and throughput above the included baseline are billed separately. A volume needing high provisioned throughput can cost more on gp3 than it did on gp2. Model the volumes that genuinely need extra performance instead of assuming a flat saving across the fleet.

Can I cancel a Savings Plan if my usage drops?

Assume no. AWS documents Savings Plans as non-cancellable for the term. There is a limited return mechanism with eligibility rules and quotas, and no secondary marketplace of the kind that exists for Standard Reserved Instances. Size the commitment to the baseline you are confident about, not the peak you hope for.

Should I use Spot Instances to cut costs?

For fault-tolerant, interruptible work such as CI runners, batch jobs and stateless workers behind a queue, the discount is hard to ignore. For anything holding state in memory or serving a request that cannot be retried, the interruption notice is short and the failure is abrupt. Spot is an architecture decision, not a billing toggle.

How do I find out where my NAT Gateway traffic is going?

Billing data cannot tell you. Enable VPC Flow Logs on the relevant subnets, query them with Athena or CloudWatch Logs Insights grouped by destination address, and map the results back to AWS service ranges. That gives you the split between S3, ECR, other AWS services and the genuine internet, which is what determines whether an endpoint pays for itself.

The one thing worth remembering

If you take one idea from this, take the ordering. You cut AWS costs without breaking production by sorting the work by blast radius rather than by savings size, and by accepting that the frightening-looking networking and storage line items are usually the safe ones, while the innocuous single-API-call changes are the ones that bite.

Do the tier-zero sweep first and completely. It is free, it is fast, and it buys you the standing to say “not yet” when someone asks you to shrink a production database on a Thursday. Then move one performance envelope at a time, with a metric and a revert threshold. Commit last, at the baseline you actually believe in.

Need a second pair of eyes on your AWS bill?

This is work I do regularly, and it goes faster with someone who has no attachment to how the account got this way. Things I can help with:

• A usage-type level breakdown of your bill, with savings ranked by blast radius, so you know what is safe this week and what needs a change window.
• NAT Gateway and data transfer analysis from VPC Flow Logs, ending in a specific list of which endpoints to add and what each is worth.
• EBS and S3 review: gp3 candidates with throughput checked properly, lifecycle rules, version cleanup, and storage class decisions based on your real object size distribution.
• CloudWatch and logging spend reduction that keeps the signal you need during incidents, including moving long-tail retention off CloudWatch storage rates.
• Savings Plan and Reserved Instance modelling against your actual baseline, including what happens to coverage if the migration you are planning goes ahead.
• Guardrails so it stays fixed: Budgets, anomaly alerting, tagging enforced in Terraform or OpenTofu, and cheap defaults baked into your modules.

If you want a concrete starting point, send me a Cost Explorer export grouped by usage type, or the output of the orphan sweep commands above, and I will tell you what I would touch first and what I would leave alone.

The post Cut AWS Costs Without Breaking Production: A Blast-Radius Playbook appeared first on John Nessime.

That is the failure mode nobody catches during a review, and it is why most AWS accounts have dozens of alarms and about three anyone trusts. The rest either fire constantly and get muted, or never fire at all and get mistaken for coverage.

This post is about building useful CloudWatch alarms: which signals deserve a page, which settings decide whether an alarm fires correctly, how to collapse alarm storms without going blind, and how to prove the notification path works before you need it.

The test an alarm has to pass before it exists

Before the threshold, before the metric, one question: if this fires at 3am, what does the person on call do?

If the honest answer is “look at it and go back to bed”, it is not an alarm. It is a dashboard panel. That single filter removes most of the CPU, memory and disk alarms people inherit. High CPU is not an incident. High CPU while latency climbs and the queue backs up is an incident. Every alarm that fires without requiring action trains the on-call to ignore the channel, and that training is permanent.

• Page on symptoms a customer can feel: error rate, latency at a high percentile, queue age, failed jobs, expiring credentials.
• Ticket on causes that bite next week: disk trending toward full, approaching a service quota, a renewal that is not happening.
• Graph only on everything else. Resource utilisation is context during an incident, not the incident.

Encode that split in the alarm’s action, not in your head. A page goes to an SNS topic wired to PagerDuty or Opsgenie. A ticket goes to a different topic that opens a Jira issue or a Systems Manager OpsItem. Same engine, different consequences.

The alarm that goes quiet: missing data is the invisible failure

Every metric alarm has a TreatMissingData setting, and if you never touched it, it is missing. That default means the alarm ignores gaps entirely when deciding state. It keeps whatever state it had, or sits in INSUFFICIENT_DATA if there is nothing at all. For a metric that always reports, that is a silent hole: the instance dies, the metric stops, and the alarm does not go red. It just stops having an opinion.

• breaching for metrics that report continuously. If the data stops, something is wrong. Heartbeats, health checks, CPU on a box that should always be up.
• notBreaching for metrics that only exist when something bad happens. Error counts, throttled requests, dead letter queue depth. No data means no errors.
• ignore keeps the current state through gaps, so an alarm stays red until genuinely resolved rather than flapping to OK on a hole in the data.
• missing is the default, and defensible only when you decided it was right.

There is a second-order effect that surprises people. When an alarm evaluates, CloudWatch pulls a wider range of data points than the evaluation periods you configured. AWS calls that the evaluation range. If some points in that range are missing but enough real ones exist, CloudWatch evaluates against the real points and your missing-data setting never applies to the gaps.

The consequence: an alarm set to eight breaching points out of ten can fire on one breaching point followed by nine gaps, because the older breach is still inside the evaluation range and everything newer is absent. Teams hit this with load balancer 5XX alarms constantly, and it looks exactly like an unexplained false positive.

Setting notBreaching on sparse error metrics is usually the fix. Filling gaps with metric math is the other, and it is more explicit:

# Treat absent error data points as zero rather than as gaps.
# m1 is the raw error metric; e1 is what the alarm evaluates.
e1 = FILL(m1, 0)

One exception: alarms on metrics in the AWS/DynamoDB namespace always ignore missing data regardless of what you set. If you need a DynamoDB alarm that reacts to silence, build the fill into metric math.

Then find the ones already stuck. This is the fastest audit you can run on an inherited account:

aws cloudwatch describe-alarms 
  --state-value INSUFFICIENT_DATA 
  --query 'MetricAlarms[].[AlarmName,MetricName,StateReason]' 
  --output table

Anything on that list older than a deploy cycle is either watching something that no longer exists, or watching something that stopped reporting. Both deserve thirty seconds.

The alarm that cries wolf: periods, data points and windows

Three settings decide whether an alarm is jumpy: Period (how long each data point covers), EvaluationPeriods (how many recent points to look at), and DatapointsToAlarm (how many of those must breach).

M out of N beats consecutive breaches

Most people set evaluation periods to 3 and leave data points to alarm unset, which makes it 3 as well. That demands three consecutive breaching points. Real degradation is rarely that tidy; it flickers, one clean point resets your patience, and the alarm never fires.

Set them separately. Three out of five catches intermittent problems while still filtering single-point spikes. The breaching points do not need to be consecutive, only inside the last N.

aws cloudwatch put-metric-alarm 
  --alarm-name "api-5xx-elevated" 
  --namespace "AWS/ApplicationELB" 
  --metric-name HTTPCode_ELB_5XX_Count 
  --dimensions Name=LoadBalancer,Value=app/my-alb/0123456789abcdef 
  --statistic Sum 
  --period 60 
  --evaluation-periods 5 
  --datapoints-to-alarm 3 
  --threshold 25 
  --comparison-operator GreaterThanThreshold 
  --treat-missing-data notBreaching 
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:oncall-page

Read that as: over the last five minutes, if three or more individual minutes each saw more than 25 backend 5XX responses, page. Minutes with no data count as fine, because a load balancer with no errors publishes nothing.

Percentiles need a sample-count decision

Average latency hides everything interesting, so alarm on p95 or p99 using the extended statistic rather than --statistic Average.

The catch is low traffic. A p99 computed from four requests is noise, and by default CloudWatch still evaluates it. The EvaluateLowSampleCountPercentile setting controls that: set it to ignore and the alarm holds its current state when the sample count is too small to mean anything. On an overnight window or a quiet endpoint, that one setting removes a whole class of 4am pages.

Sliding versus wall clock windows

By default the evaluation window slides: it advances a minute at a time and is not aligned to the clock. That is what you want for latency and errors, because you want to know now.

CloudWatch also supports a wall clock window aligned to real boundaries in a time zone you pick: top of the hour, midnight, start of the calendar week. That is the right choice for anything measured per calendar period, like a daily batch job or a monthly threshold, because the alarm considers a completed period rather than a rolling one. Note that the wider evaluation range described earlier applies only to sliding windows, so wall clock alarms are easier to reason about but have no safety net for late-arriving data.

Alarming on the wrong number

A large share of useless alarms are correctly configured alarms watching a meaningless quantity.

Rates, not counts

An alarm on “more than 50 errors” is wrong twice. During a traffic peak, 50 errors out of half a million requests is nothing. At 3am, 50 errors out of 60 requests is a total outage that never crosses the threshold.

# m1 = Errors (Sum), m2 = Invocations (Sum)
# Guard the zero case so the alarm gets a data point at idle
# instead of a gap it has to interpret.
e1 = IF(m2 > 0, m1 / m2 * 100, 0)

Only the final expression should return data to the alarm; the underlying metrics are inputs. A rate alarm alone is still incomplete, though, because at very low volume one error is 100 percent. Pair it with a minimum-volume condition in a composite alarm, or fold the request count into the expression so tiny denominators cannot trip it.

Quotas and saturation, before they bite

Service quotas are the outage nobody sees coming, because everything is healthy right up until it is not. For services that publish usage metrics, the SERVICE_QUOTA metric math function pulls your current limit from Service Quotas, so you can alarm on percentage consumed instead of a number you hardcoded and forgot:

# m1 is a usage metric under the Usage namespace.
# Alarm on consumption as a share of the live quota.
e1 = m1 / SERVICE_QUOTA(m1) * 100

This is the definitive ticket-not-page alarm. Nobody needs waking for concurrency at 80 percent, but somebody needs to file a limit increase before the deploy that takes it to 100. Same category: queue age rather than queue depth. A deep queue draining fast is fine; a shallow queue whose oldest message is twenty minutes old means your consumers are stuck.

Cutting the storm: composite alarms, suppression and mute rules

One bad deploy in an Auto Scaling group with per-instance alarms produces forty notifications describing one event. A composite alarm evaluates a boolean rule over other alarms. The metric alarms stay exactly as they are and keep their own state, but stop being wired to your paging topic. The composite becomes the only thing that pages.

aws cloudwatch put-composite-alarm 
  --alarm-name "checkout-degraded" 
  --alarm-rule 'ALARM("checkout-latency-p99") AND ALARM("checkout-error-rate")' 
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:oncall-page

Latency alone might be a slow dependency. Errors alone might be a bad client. Both together is a real problem, and that is what earns a page.

The second half is suppression. A composite alarm accepts a suppressor alarm: while that alarm is in ALARM, the composite still evaluates but takes no actions. Point it at a deployment-in-progress alarm and you stop paging for known-bad windows.

aws cloudwatch put-composite-alarm 
  --alarm-name "checkout-degraded" 
  --alarm-rule 'ALARM("checkout-latency-p99") AND ALARM("checkout-error-rate")' 
  --actions-suppressor "deployment-in-progress" 
  --actions-suppressor-wait-period 60 
  --actions-suppressor-extension-period 60 
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:oncall-page

Those two periods are the part people skip and then get burned by. The wait period gives the suppressor time to reach ALARM before the composite decides to fire; the extension period gives the composite time to settle back to OK after the suppressor clears. AWS suggests 60 seconds for each, on the reasoning that metric alarms evaluate once a minute. Leave them at zero and you get paged in the gap between the deploy starting and the deploy alarm noticing.

CloudWatch has since added alarm mute rules, which handle the scheduled case directly. A mute rule targets specific alarms by name and mutes their actions during a one-time or recurring window. The alarms keep evaluating and keep changing state; only the actions are muted. That removes the EventBridge and Lambda contraption teams used to build for maintenance windows. Two things to keep straight: mute rules take precedence over composite action suppression when both apply, and a mute rule targets alarms by name, so a rename in your Terraform or OpenTofu module silently orphans the rule.

Suppression is for known-bad windows you chose. If you are muting an alarm because it is annoying rather than because you scheduled the disruption, the alarm is wrong and muting it just hides that.

Test the path, not the threshold

An alarm you have never seen fire is a hypothesis. The threshold is the easy part. The fragile part is everything downstream: SNS subscriptions that were never confirmed, a Lambda whose role lost a permission, a chat webhook rotated three months ago, a topic policy that quietly rejects CloudWatch.

You can force a state transition without touching the metric:

aws cloudwatch set-alarm-state 
  --alarm-name "api-5xx-elevated" 
  --state-value ALARM 
  --state-reason "Routing test, ticket OPS-1234"

This drives the real actions, so the page really goes out. Tell the on-call first. The forced state is temporary; the next real evaluation overwrites it. Do this for every new alarm before you call it done, and again after any change to the notification stack. On a mixed estate where some workloads sit on a VPS at a provider like Contabo or InterServer alongside your AWS footprint, test each path separately, because a shared SNS topic creates the illusion that one confirmed subscription covers everything.

Then confirm the transition and the action both happened:

aws cloudwatch describe-alarm-history 
  --alarm-name "api-5xx-elevated" 
  --history-item-type Action 
  --max-records 10

What useful CloudWatch alarms cost you

Alarms bill per alarm metric per month, so the mental model that matters is this: a standard metric alarm counts as one, and a metric math alarm counts as one per metric referenced in the expression. An error-rate alarm built from errors and invocations bills as two, not one. Anomaly detection alarms carry their own multiplier because the band is evaluated alongside the metric. High-resolution alarms, with periods of 10, 20 or 30 seconds, are charged at a higher rate than standard ones, which makes them worth it for a handful of latency-critical paths and wasteful everywhere else.

Check current rates on the CloudWatch pricing page rather than trusting a number in a blog post, including this one. And note the bigger cost is usually not the alarm at all. It is the custom metrics feeding it, especially high-cardinality ones published per request or per container.

Troubleshooting an alarm that misbehaves

1. Stuck in INSUFFICIENT_DATA. Either the metric is not being published, or the alarm’s dimensions do not match the metric’s dimensions exactly. Dimension mismatch is the common one and it is silent: an alarm on a dimension set that never existed looks identical to an alarm on a dead resource.
2. Fired but nobody heard. Check describe-alarm-history filtered to Action items. Transition present but action missing points at the SNS topic policy or the subscription. Both present points downstream of SNS.
3. Fires with no visible breach on the graph. Almost always missing data plus the wider evaluation range. The state reason names how many points were breaching and how many were unknown.
4. Flapping between OK and ALARM. The period is too short for the metric’s natural variance, or data points to alarm equals evaluation periods. Widen the window before raising the threshold; raising the threshold loses sensitivity you may want later.
5. Composite alarm never fires. Confirm every child alarm named in the rule exists and is spelled correctly, then confirm no suppressor or mute rule is active. The console shows a mute indicator on affected alarms.

Common mistakes

• Leaving TreatMissingData at the default because you did not know it existed.
• Picking 80 percent CPU because it sounds like a threshold, on a service that normally runs at 75.
• Alarming per instance in an Auto Scaling group instead of on the aggregate a customer experiences.
• Wiring OK actions to the same paging channel, so every recovery is also an interruption.
• Alarm descriptions that restate the alarm name. The description is the only context the responder gets at 3am, so put the runbook link there.
• Anomaly detection on a metric with no stable pattern. The band is only as good as the history it learned from.

Best practices for useful CloudWatch alarms

• Set TreatMissingData explicitly on every alarm, and write down why in the description.
• Separate DatapointsToAlarm from EvaluationPeriods. M out of N is almost always better than N consecutive.
• Observe a metric across a full weekly cycle before fixing its threshold. Weekend traffic is a different distribution.
• Define alarms in code, in whatever you already use: Terraform, OpenTofu, CDK, CloudFormation. Console-created alarms drift and disappear.
• Use anomaly detection for the signal and composite alarms for the logic. An anomalous shape plus real user impact is a page; an anomalous shape alone is a graph.
• Review what fired last quarter. Any alarm that fired and was closed without action is a candidate for deletion. Send it to a Grafana panel instead.

Frequently asked questions

How many CloudWatch alarms should a service have?

Fewer than you think. A handful of symptom alarms that page, plus a set of cause alarms that open tickets, covers most services. The number that matters is not how many exist but how many fired last quarter and led to somebody doing something.

Should I use anomaly detection or static thresholds?

Static thresholds where a real limit exists: a disk that must not fill, a latency budget you promised. Anomaly detection where the metric has a genuine daily or weekly shape and no fixed correct value, like request volume. Anomaly detection needs history before it is useful and struggles when a deploy changes the pattern, so it is a poor fit for anything new.

Why did my alarm fire when the graph looks fine?

Usually missing data. CloudWatch pulls a wider evaluation range than the periods you configured, so an older breaching point can still count while newer points are simply absent. Read the state reason, which spells out how many points were breaching and how many were unknown, then reconsider the missing-data setting.

Can an alarm watch a metric in another AWS account?

Cross-account observability lets a monitoring account view and alarm on metrics from linked source accounts, which is the usual pattern for an AWS Organizations estate. The alternative is running alarms in each account and aggregating at the SNS or paging layer. That is more alarms to manage but keeps the blast radius per account smaller.

How do I stop alarms firing during a deployment?

Two options. An alarm mute rule on a schedule, if your deploys are predictable. Or a composite alarm with a deployment-in-progress alarm as its actions suppressor, if they are not. The suppressor approach suits continuous delivery better because it reacts to the actual deploy rather than a calendar guess.

Do CloudWatch alarms work on logs?

Indirectly. A metric filter on a log group turns matching log lines into a numeric metric, and you alarm on that metric like any other. Metric filters only emit data points when they match, so notBreaching is almost always the right missing-data setting for them.

Is CloudWatch enough, or do I need Datadog or Grafana?

For a single-account AWS workload, CloudWatch alarms plus SNS handle alerting fine, and staying inside AWS avoids another integration to maintain. Third-party platforms earn their keep when you need correlation across non-AWS systems, richer on-call routing, or long metric retention with fast queries. The honest test is whether you are already spending engineering time rebuilding those features on top of CloudWatch.

The one thing to keep

Useful CloudWatch alarms are not the ones with the cleverest thresholds. They are the ones where somebody decided, on purpose, what silence means.

Every alarm in your account is making a claim about the absence of data, whether you configured that claim or inherited it. Go through them, set TreatMissingData deliberately, split data points from evaluation periods, delete the ones nobody acts on, and force a state transition on the ones you keep so you know the page lands. Six alarms that fire correctly beat sixty that nobody reads.

Need a second pair of eyes on your alerting?

I work with teams on AWS monitoring that people actually trust. Typically that looks like:

• Auditing existing CloudWatch alarms and separating the ones that page, the ones that should open tickets, and the ones to delete.
• Fixing missing-data and evaluation settings so alarms stop firing on gaps and stop going silent on dead metrics.
• Building composite alarms, suppression and mute rules so one incident produces one page instead of forty.
• Writing metric math for error rates, queue age and quota headroom instead of raw counts.
• Codifying the whole alarm set in Terraform or OpenTofu so it survives the next person.
• Wiring CloudWatch into Grafana dashboards and testing every notification path end to end.

If you want a concrete starting point, send me the output of describe-alarms for one account and I will tell you which alarms are load-bearing and which are decoration.

The post CloudWatch Alarms That Are Actually Useful (and the Ones to Delete) appeared first on John Nessime.

That gap is the entire problem. Monitoring n8n workflow failures in Grafana is not hard because the tooling is missing. It’s hard because the built-in metrics endpoint answers a question you didn’t ask. It tells you the process is alive and busy. It does not tell you that a workflow stopped producing correct output.

This post covers what n8n’s Prometheus endpoint actually exposes, the three families of failure it handles badly, and the specific PromQL and alert rules that close each gap. There’s also a troubleshooting section for the case where /metrics returns nothing at all, which is the first wall most people hit.

What the n8n metrics endpoint actually gives you

n8n uses the prom-client library and serves metrics on the same port as the editor, at /metrics. The endpoint is off by default, and almost everything interesting inside it is off by default too, behind its own environment variable.

N8N_METRICS=true
N8N_METRICS_INCLUDE_MESSAGE_EVENT_BUS_METRICS=true
N8N_METRICS_INCLUDE_WORKFLOW_ID_LABEL=true
N8N_METRICS_INCLUDE_NODE_TYPE_LABEL=true
N8N_METRICS_INCLUDE_QUEUE_METRICS=true

The first line opens the endpoint. The second turns on the workflow and node event counters, which is the part people assume is included and is not. The third attaches a workflow_id label to workflow metrics, without which you get a single instance-wide number and no way to tell which automation broke.

Those labels are opt-in for a reason worth understanding rather than working around. Every distinct label value creates a separate time series. Turn on workflow_id and you get one series per workflow per counter; add node_type and node counters fan out across every integration you use. On a few dozen workflows that’s fine. On several hundred with per-node labels, you’re signing up for a cardinality problem that surfaces later as slow queries and a Prometheus process that keeps getting OOM-killed.

With the event bus metrics enabled, the counters you get include n8n_workflow_started_total, n8n_workflow_success_total, n8n_node_started_total and n8n_node_finished_total. Queue mode adds n8n_scaling_mode_queue_jobs_active, _completed, _failed and _waiting, gathered from Bull and exposed on the main instance only.

Prometheus side, the scrape job is unremarkable:

scrape_configs:
  - job_name: n8n
    static_configs:
      - targets:
          - n8n-main:5678
          - n8n-worker-1:5678
    metrics_path: /metrics

Note both main and workers. Each process exposes its own endpoint and its own counters. Scrape only the main and you lose everything the workers did, which in queue mode is nearly all the real execution work.

One thing the n8n docs are emphatic about, and they’re right: don’t put /metrics on the public internet. It leaks workflow IDs, execution volumes and instance topology to anyone who asks. Keep the scrape path on a private network segment, a WireGuard tunnel, or your provider’s internal networking. If you’re running the whole stack on a single VPS from somewhere like Contabo or InterServer, binding n8n to the Docker network and scraping it by container name is enough. Publishing port 5678 to the host and firewalling it afterwards is not, because you will forget.

Failure family one: the execution that never happened

This is the one that costs the most and shows up the least. A schedule trigger gets deactivated during a deploy. A credential expires and the workflow is switched off rather than fixed. An upstream service stops calling a webhook. Nothing errors. Nothing increments. The workflow simply stops existing as far as your metrics are concerned.

Every error-rate alert you write is blind to this, because an error rate needs a denominator. Zero executions divided by zero executions is not an alert, it’s a NaN.

The instinct is to write this:

sum by (workflow_id) (increase(n8n_workflow_started_total[6h])) == 0

It looks correct and it does not work. When a workflow stops running, its series stops being exported. Prometheus marks it stale after a few missed scrapes and instant queries stop returning it. There is no series left for == 0 to match against. The alert never fires, because the thing you wanted to alert on is exactly the thing that removed the data.

What you want is a function that returns something when a series is absent:

absent_over_time(
  n8n_workflow_started_total{workflow_id="AbC123XyZ"}[6h]
)

absent_over_time returns 1 if the matched series produced no samples in the window and nothing at all if it did. That inversion is the point. The alert fires on silence.

The cost is that absent_over_time needs an exact matcher, so you can’t write it once for everything. That’s more acceptable than it sounds. You don’t need a heartbeat on all two hundred workflows, you need one on the six a human would notice within an hour. Write those six by hand and treat the list as part of the runbook.

One Grafana-specific trap. When a query returns nothing, Grafana’s alert rule enters a No Data state, which is configurable separately from the alert condition and defaults to firing in most versions. Set it deliberately. For an absent_over_time rule, No Data means the whole scrape target is gone, which is a different incident from one workflow going quiet, and you probably want it routed differently.

Failure family two: the failure you can see but can’t attribute

You have the counters. Something is failing. Which workflow?

Without N8N_METRICS_INCLUDE_WORKFLOW_ID_LABEL, the answer is: no idea. The counters are instance-wide totals. You know the number moved and that’s it. That’s a fine signal for capacity planning and useless during an incident.

Turn the label on and you get IDs, which are opaque strings like AbC123XyZ. A dashboard full of those is technically attributable and practically unreadable at 3am. Newer n8n versions solve this with a gauge that maps IDs to names, enabled with N8N_METRICS_INCLUDE_WORKFLOW_INFO:

n8n_workflow_info{workflow_id="VaQPuPmx9tPpo6BX",workflow_name="Invoice sync"} 1

It’s a constant-1 gauge that exists only to be joined against. The pattern is a standard Prometheus vector match:

sum by (workflow_name) (
  rate(n8n_workflow_started_total[5m])
  * on(workflow_id) group_left(workflow_name)
  n8n_workflow_info
)

The multiplication is arithmetically a no-op because the gauge is always 1. Its only job is to carry the workflow_name label across with group_left. Do this once in a Grafana variable or a recording rule rather than in every panel, or you’ll be copy-pasting the join for the rest of your life.

On an older build without the info gauge, the fallback is a hand-maintained relabel config or a Grafana value mapping. Both drift, and the relabel version drifts silently.

Failure family three: the failure that counts as a success

This is the invisible one, and it’s why a green n8n dashboard is not evidence of anything.

n8n gives you several deliberate ways to make a node’s failure not be an execution failure. They’re all reasonable features. Together they mean the success counter and “the automation worked” are different statements.

• Continue on error. A node set to continue passes its error downstream instead of stopping the run. The execution finishes. It finishes as a success.
• Error output branches. A node with a separate error output routes failures to a different path. That path might log and stop. The execution still completes normally.
• Never Error on HTTP responses. The HTTP Request node can be configured to treat any status code as a valid response. A run of 500s from an upstream API becomes a workflow that processed zero useful records, successfully.
• Empty result sets. A query that returns nothing is not an error. A sync that syncs nothing, every night, looks exactly like a sync that has nothing to do.

There are also user reports on the n8n community forum of started and success counters matching even when a node failed outright, in queue mode. I have not reproduced that and I’d treat it as unconfirmed rather than a known behavior. But it’s the right instinct: don’t assume the success counter is a strict complement of failure. Verify it on your own instance before you build alerting on the assumption.

For webhook-triggered workflows, newer n8n versions give you a real answer here. The n8n_webhook_request_duration_seconds histogram carries a status_code label, so you can alert on what the caller actually received rather than on what n8n thought about the run:

sum by (workflow_id) (
  rate(n8n_webhook_request_duration_seconds_count{status_code!~"2.."}[5m])
)

Worth flagging: the n8n documentation shows this example with status_code!="2..", using the equality operator. In PromQL, != compares against a literal string, so that matcher excludes only a status code whose literal value is the three characters 2.., which never occurs. It matches everything, including your 200s. Use !~ for the regex form. Test any label matcher you copy from documentation by running it in Prometheus and checking the series count against what you expect.

Getting a second source of truth

For everything that isn’t a webhook, the metrics endpoint alone won’t close this gap. You need a second signal that reads execution outcomes from n8n’s own store.

The public REST API exposes exactly that. Executions can be filtered by status, and the failed ones are one request away:

curl -s -H "X-N8N-API-KEY: $N8N_API_KEY" 
  "https://n8n.example.com/api/v1/executions?status=error&limit=50"

Two ways to get that into Grafana. Write a small exporter that polls the API on a timer and serves its own /metrics, or run a scheduled job that pushes counts to a Prometheus Pushgateway. The exporter is more work and behaves correctly. Pushgateway is faster to build and carries the usual caveat: pushed metrics persist until deleted, so a job that stops running leaves a stale value sitting there looking healthy. That reintroduces failure family one through the back door.

Two caveats on the API. There’s an open report that the status=crashed list filter returns executions whose stored status is something else entirely when re-fetched by ID, which would make crash alerting built on the list endpoint unreliable. Check the behavior on your version before depending on it. And n8n’s log streaming feature, which forwards execution events to syslog or a webhook in real time, is an Enterprise capability. If you’re on the community edition, the API poll is your path.

The third option, and often the best one for a small setup, is an Error Trigger workflow. n8n lets you designate a workflow that runs whenever another workflow fails. Point it at Pushgateway, at Loki, or at a Grafana webhook contact point. It’s native, it needs no API key, and it gives you the error message rather than just a count. The trade-off is that it only fires on real execution failures, so it inherits every blind spot in this section.

The Grafana panels that earn their space

Four panels cover most of what you need. Everything past that is usually decoration.

1. Executions per workflow, by name. A time series of rate(n8n_workflow_started_total[5m]) joined to n8n_workflow_info. Its real job is showing you the lines that flatten to zero.
2. Success ratio. rate(n8n_workflow_success_total[15m]) / rate(n8n_workflow_started_total[15m]). Anything below 1 is worth a look. Anything above 1 means your window is catching a restart and the counters reset.
3. Queue depth and failed jobs. n8n_scaling_mode_queue_jobs_waiting next to increase(n8n_scaling_mode_queue_jobs_failed[1h]). A waiting count that climbs and never drains means workers are dead or starved, which looks identical to “no failures” on every other panel.
4. Webhook non-2xx rate. The query from the previous section. This is the only panel that reports what the outside world experienced.

Skip the node-level breakdown unless you have a specific reason. It is the most expensive thing you can enable in cardinality terms and it answers a question you can answer faster by opening the execution in the n8n UI.

Alert rules worth paging on

1. Named workflow silent. absent_over_time on each business-critical workflow, window set to roughly twice its expected interval. This is the rule that catches the invoice sync.
2. Success ratio below threshold. Sustained for 15 minutes, not instantaneous. A single failed run is not a page.
3. Queue waiting count rising. Compare the current value to fifteen minutes ago rather than to a fixed number, so the threshold survives traffic growth.
4. Webhook error rate. Per workflow, so a single noisy endpoint doesn’t mask the rest.
5. Scrape target down. The up metric. Obvious, and routinely forgotten, and it invalidates every other rule on this list.

Troubleshooting: the endpoint returns nothing useful

• “Cannot GET /metrics”. N8N_METRICS isn’t set on the process you’re hitting, or it’s set as a string that n8n doesn’t read as true. In Docker Compose, quote it as "true" and confirm with docker compose exec n8n env | grep N8N_METRICS.
• Endpoint responds but has no n8n_ counters. You have the default Node.js process metrics and nothing else. Set N8N_METRICS_INCLUDE_MESSAGE_EVENT_BUS_METRICS=true and restart.
• Counters exist but no workflow_id label. N8N_METRICS_INCLUDE_WORKFLOW_ID_LABEL is off. Restart is required; it’s read at startup.
• Queue metrics are zero or missing. They’re gathered from Bull and exposed on the main instance, not the workers. Querying the worker endpoint gives you nothing. Confirm you’re looking at the right target.
• Counters look wrong after a deploy. They reset to zero when the process restarts. Always wrap them in rate() or increase(), which handle counter resets. Raw counter values on a dashboard will lie to you every time you deploy.
• Prometheus can’t reach the target. Usually a Docker network mismatch. Scrape by container name on the shared network rather than by published host port, and check the target page in Prometheus for the actual error rather than guessing.

Common mistakes

• Treating the success counter as proof the automation worked. It proves the run finished.
• Building only error-rate alerts, so the workflows that stop entirely go unmonitored.
• Enabling every N8N_METRICS_INCLUDE_* variable on day one and discovering the cardinality bill a month later.
• Scraping only the main instance in queue mode and losing all worker-side execution data.
• Copying label matchers out of documentation without running them and checking the returned series count.
• Exposing /metrics publicly because it’s “just numbers”.

Best practices

• Decide which workflows are business-critical and write a silence alert for each one by name. Six good heartbeats beat two hundred generic panels.
• Run a second source of truth. Metrics tell you about volume, the executions API tells you about outcomes. You need both.
• Push the ID-to-name join into a recording rule so panels stay readable and cheap.
• Break a staging workflow on purpose and confirm the alert fires. An untested alert is a decoration.
• If you’d rather not run Prometheus and Grafana yourself, Grafana Cloud will scrape a remote endpoint and gives you alerting without a control plane to maintain. Weigh that against your metrics leaving your network.

FAQ

Does n8n expose a failed-executions counter?

Not a reliable per-workflow one. You get started and success counters from the event bus, and a failed-jobs counter for queue mode. Deriving failures by subtracting success from started works only if you’ve verified on your own instance that a failing run genuinely doesn’t increment success, which is exactly the assumption that breaks when nodes are set to continue on error.

Can I monitor n8n Cloud this way?

No. The /metrics endpoint isn’t available on n8n Cloud. On Cloud you’re limited to the public API and the built-in insights views, so the executions-API approach in this post is the only one that transfers.

Why does my workflow error rate look like zero when I know things are failing?

Almost always one of three things: the event bus metrics aren’t enabled so you’re only seeing Node.js process metrics, the failures are being absorbed by continue-on-error settings and recorded as successes, or you’re scraping the main instance while the work happens on workers.

How do I alert when a scheduled workflow stops running entirely?

Use absent_over_time against that workflow’s started counter with an exact workflow_id matcher, and set the window to about twice the expected interval. A threshold comparison won’t work, because a workflow that stops running stops producing the series you’d compare against.

Is enabling the workflow_id label safe on a large instance?

It multiplies your n8n series count by the number of workflows. On a few dozen it’s negligible. Past a few hundred, combined with node-type labels, watch your Prometheus memory and head series count before and after. Enable the workflow label, leave the node label off, and only turn the node one on when you have a specific question it answers.

Should I use OpenTelemetry instead of Prometheus for n8n?

They solve different problems. Metrics answer “how many, how often, how fast” cheaply and forever. Traces answer “where did this run spend its time and what did it call”, which is better for debugging a slow multi-service workflow. Tracing n8n usually means a custom image with the OpenTelemetry SDK loaded before startup, so it’s a project rather than an environment variable. Start with metrics; add tracing when you have a latency question metrics can’t answer.

Where do the n8n counters go when the container restarts?

They reset to zero. They’re in-process counters with no persistence. Prometheus handles this correctly as long as your queries use rate() or increase(). Any panel showing a raw counter value will appear to lose all its history on every deploy.

The one thing to take away

Monitoring n8n workflow failures in Grafana works, but only once you stop asking the metrics endpoint a question it can’t answer. It reports activity. Activity and correctness are different things, and the space between them is where the expensive outages live.

So build for absence first. A silence alert on the handful of workflows a human would miss will catch more real incidents than any error-rate panel, because errors are loud and stopped schedules are not. Then add attribution with the workflow label, then close the false-success gap with the executions API or an Error Trigger workflow. In that order, because each one is cheap and the value drops off fast after the third.

Need help wiring this up?

I build and maintain observability stacks for self-hosted automation platforms, and n8n is a common one. If any of the above sounds like a week you’d rather not spend, here’s what I can take off your plate:

• Enabling and securing the n8n metrics endpoint on a private network path, including queue-mode setups with multiple workers
• Building Prometheus scrape configs, recording rules and the ID-to-name join so your Grafana panels are readable rather than a wall of hashes
• Writing silence and error-rate alert rules that survive counter resets, Grafana No Data states and traffic growth
• Adding a second source of truth from the n8n executions API, either as a small exporter or an Error Trigger workflow feeding Loki or Pushgateway
• Auditing an existing setup for cardinality problems, stale Pushgateway series and alerts that quietly stopped being able to fire
• Running a failure drill against staging so you know which alerts actually work before you need them

Send me your Compose file, your prometheus.yml, or a screenshot of the panel that looks fine while things are on fire, and I’ll tell you what’s missing.

The post Monitoring n8n Workflow Failures in Grafana: Why Your Green Dashboard Is Lying appeared first on John Nessime.

Sometimes the disk really is dying. More often it isn’t, and the number you’re staring at is not measuring what you think it’s measuring.

This post covers troubleshooting high I/O wait the way I’d actually do it on a production box: what the metric means, why it lies in both directions, and how to work through the five failure families that produce it.

What high I/O wait actually measures

Here’s the definition that matters: iowait is a subcategory of idle time. For a given CPU, it’s the time that CPU spent doing nothing while at least one task scheduled on it had an outstanding block I/O request.

Two consequences fall out of that, and both will burn you. High iowait can mean nothing is wrong. If the box has spare CPU and one process is pulling a large file off disk, the CPU genuinely has nothing else to do. Copying a backup will push iowait toward 100 percent on an idle host, and that’s a working system, not a fault.

Low iowait can hide a severe problem. This is the one that costs people days. If the CPU has other runnable work, the kernel context switches to it and that time gets counted as user or system, not iowait. A busy application server can be stalling badly on storage while wa reads two percent, because the CPU is never actually idle.

Add the averaging problem on top. The wa figure in top is averaged across every core. One thread pinned to one core and blocked hard on disk shows up as roughly three percent on a 32-core machine. Press 1 in top to break out per-CPU lines before you conclude anything from the aggregate.

So iowait is a hint, not a verdict. Treat it as the thing that starts the investigation, never the thing that ends it.

Start with pressure, not percentages

Pressure Stall Information is the metric I reach for first, and it exists precisely because iowait is a poor proxy for pain. It answers the question you actually care about: how much time did tasks lose because they couldn’t get I/O?

cat /proc/pressure/io

You get two lines. some is the share of time at least one task was stalled waiting on I/O. full is the share of time every non-idle task was stalled at once, which is CPU capacity being burned for nothing. The avg10, avg60 and avg300 fields are percentages over the last 10, 60 and 300 seconds. total is a cumulative microsecond counter since boot, which is what you want for graphing because averages smooth away short spikes.

The practical rule I use: a rising full line on io is a real user-visible problem almost every time. A high some line with a flat full line usually means one noisy job is waiting while everything else gets on with its life.

If /proc/pressure doesn’t exist, the kernel was built without CONFIG_PSI, or it was built with PSI compiled in but disabled by default, in which case you need psi=1 on the kernel command line. On cgroup v2 systems you also get per-cgroup pressure, which is how you narrow from “the host is stalling” to “this container is stalling” in one command:

# Pressure for one systemd slice, useful on any modern distro
cat /sys/fs/cgroup/system.slice/io.pressure

# Pressure for every leaf cgroup that has it, sorted by nothing,
# just eyeball the full lines
find /sys/fs/cgroup -name io.pressure -exec sh -c 
  'printf "%s: " "$1"; sed -n 2p "$1"' _ {} ;

If you run Prometheus, node_exporter has exposed these since it gained a pressure collector. The counters are node_pressure_io_waiting_seconds_total for some and node_pressure_io_stalled_seconds_total for full. Rate those and you have a far better storage alert than any threshold on %iowait will ever be. Grafana Cloud and Netdata both chart PSI out of the box if you’d rather not build the panels yourself.

Failure family one: the device really is saturated

This is the case everyone assumes, so get it confirmed or eliminated quickly.

iostat -xy 1

The -x gives extended per-device stats. The -y matters more than people realise: without it, the first report is a since-boot average, and reading that first block is the single most common iostat mistake there is. It will tell you the disk has been fine all year, which is true and useless.

Now the columns worth your attention:

• r_await and w_await: average milliseconds a read or write took, queue time included. This is the closest thing to “how slow does storage feel”. Split reads from writes, because they fail very differently.
• aqu-sz: average queue depth. Requests either queued or in service. A depth that climbs while throughput stays flat is the signature of a genuinely saturated device.
• r/s, w/s, rkB/s, wkB/s: the actual workload. Compare against what the device or volume class is rated for.
• %util: treat with suspicion, see below.

The %util trap. That column counts the fraction of time at least one request was in flight. On a spinning disk that serviced one request at a time, it was a fair saturation signal. On NVMe and on cloud volumes that handle many requests in parallel, a device can sit at 100 percent while running at a fraction of its real capacity. Provisioning a bigger volume off the back of a 99 percent %util reading is a common and expensive way to change nothing. Use aqu-sz and the await columns instead. Similarly, svctm was deprecated and dropped from newer sysstat output because it was unreliable on parallel devices; ignore it if an old build still prints it.

Once you know the device is busy, find the culprit. pidstat -d 1 gives per-process read and write rates from the same sysstat package and needs nothing special enabled. iotop -o is friendlier, but on kernels from 5.14 onward per-task delay accounting is off by default, so the SWAPIN and IO> columns come back empty until you turn it on:

sudo sysctl kernel.task_delayacct=1
sudo iotop -oPa
# turn it back off when you're done, it costs a little scheduler overhead
sudo sysctl kernel.task_delayacct=0

And check the boring things before the clever ones. dmesg -T | grep -iE 'i/o error|reset|medium' catches a failing drive or a flapping controller. smartctl -a /dev/sda catches reallocated sectors on hardware you own. If either of those turns something up, stop reading and replace the disk.

Failure family two: memory pressure in an I/O costume

This is the failure family I’d check second, and it’s invisible if you only look at storage.

When RAM gets tight, the kernel starts evicting page cache and swapping. Both generate disk traffic. The disk gets busy, iowait climbs, and every graph points at storage while the actual constraint is memory. Give the box more RAM and the “disk problem” disappears.

# si and so are swap-in and swap-out in KB/s.
# Sustained non-zero values are your answer.
vmstat 1

# Are we thrashing writeback? Dirty pages waiting to be written.
grep -E 'Dirty|Writeback' /proc/meminfo

# The direct question
cat /proc/pressure/memory

If memory pressure and I/O pressure rise together, fix memory first. Everything else is downstream.

There’s a related trap on the write side. The kernel buffers writes in page cache and flushes them in the background, governed by vm.dirty_background_ratio and vm.dirty_ratio. Cross the background threshold and flusher threads start writing. Cross the hard vm.dirty_ratio threshold and writing processes are forced to do writeback themselves, synchronously. That transition is brutal, and on a box with plenty of RAM and a modest disk the defaults let you accumulate gigabytes of dirty pages and then pay for all of them at once. Lowering the ratios makes writeback more frequent and less spiky, trading away a little write batching. Measure before you commit to it.

Failure family three: small synchronous writes

Throughput can be trivial and the system still crawls. A workload doing thousands of tiny fsync calls a second is bound by round-trip latency, not bandwidth, and no amount of extra megabytes per second will help.

Databases are the usual source, because durability has a cost and that cost is a flush. MySQL and MariaDB expose it through innodb_flush_log_at_trx_commit; PostgreSQL through synchronous_commit. Loosening either buys real throughput and trades away durability on power loss or crash. That can be a perfectly reasonable trade for a cache or an analytics replica, and a terrible one for anything holding orders or payments. Decide it explicitly, write down what you decided, and don’t let it be a thing somebody changed once at 2am.

The other common source is logging. Application logs, access logs, container JSON logs and a database sharing one volume will starve each other. Splitting hot write paths onto separate devices is unglamorous and works.

Failure family four: throttled, not saturated

A throttled device and a saturated device look almost identical from inside the guest: high await, growing queue, miserable latency. The difference is that no amount of tuning inside the box will fix a throttle. Three flavours are worth knowing.

• Cloud volume limits. Managed block storage gives you a baseline of IOPS and throughput, sometimes with a burst allowance on top. Burn the burst and you drop to baseline, and everything that was fine in testing falls over in production. The tell is a latency cliff that arrives at a consistent point into a workload rather than tracking load smoothly. On AWS, watch the volume-level CloudWatch metrics such as BurstBalance on burstable volume types and VolumeQueueLength; other providers publish equivalents. The fix is a volume class change, not a sysctl.
• cgroup limits you set yourself. On cgroup v2, io.max caps a cgroup at a given bytes-per-second or IOPS figure, and io.weight shares bandwidth proportionally under contention. A limit that made sense when it was written can become the bottleneck a year later, and nothing in the container’s own metrics announces it.
• Neighbours on shared hardware. On oversubscribed VPS plans your storage is shared, and someone else’s backup job is your latency spike. This is where cheap and expensive hosting genuinely differ. Budget providers like Contabo and InterServer are fine value for a lot of workloads, but if storage latency is load-bearing for you, look for plans with dedicated NVMe rather than shared pools, and test before you migrate anything that matters.

# Per-device I/O accounting for a cgroup, including any throttling
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/io.stat

# Is a cap set? An empty file means no limit on that cgroup.
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/io.max

One more signal while you’re here. In top, the st column is steal time, meaning the hypervisor gave your vCPU’s slice to someone else. Steal and iowait rising together on a shared host is a strong hint that the problem is the neighbourhood, not your configuration.

Failure family five: the latency isn’t in the block layer

Here’s the failure mode that wastes the most time, because every storage metric you check will look healthy.

iostat measures the block layer. It cannot see anything above it. So if your latency is in NFS, in a network filesystem, in an overlay or FUSE mount, in filesystem lock contention, or in a journal that’s serialising writes, the device will report sub-millisecond service times while your application waits tens of milliseconds per operation.

The diagnosis is the gap. Measure latency at the application’s layer and at the block layer, and if the two numbers disagree wildly, the problem lives in between.

# Block layer latency as a histogram. Look for a bimodal shape
# or a long tail rather than the average.
sudo biolatency-bpfcc -m 10 1      # Debian/Ubuntu naming
sudo /usr/share/bcc/tools/biolatency -m 10 1   # RHEL family

# Filesystem operations slower than 10ms, with the process and file
sudo ext4slower-bpfcc 10
sudo xfsslower-bpfcc 10

# Everything at the VFS layer, higher overhead, use briefly
sudo fileslower-bpfcc 10

If ext4slower is full of hundred-millisecond operations and biolatency shows everything completing in microseconds, you have your answer and it is not the disk. For NFS specifically, nfsiostat splits out RTT from execution time, which separates network cost from server cost. Install the tooling with bpfcc-tools on Debian and Ubuntu, or bcc-tools on RHEL, AlmaLinux and Rocky.

A working order for troubleshooting high I/O wait

On an unfamiliar box, this is the sequence I work through. Each step either finds the problem or eliminates a family.

1. Read /proc/pressure/io. If full is near zero, the stall is not hurting the whole system and you should widen the search before you tune storage.
2. Check /proc/pressure/memory at the same time. If it’s rising too, fix memory first.
3. Run iostat -xy 1 for thirty seconds. Note r_await, w_await and aqu-sz per device. Ignore %util.
4. List processes stuck in uninterruptible sleep: ps -eo state,pid,comm | awk '$1 ~ /^D/'. That’s your blocked set, and it names the workload.
5. Attribute the traffic with pidstat -d 1. Confirm the process you suspect is the process actually doing the I/O.
6. If await is high, decide saturated versus throttled. Check cgroup io.max, check provider volume metrics, check steal time.
7. If await is low but the application is slow, jump to the layer gap. Run biolatency and ext4slower together and compare.
8. Only now consider tuning: dirty ratios, I/O scheduler, flush settings, splitting volumes.

Common mistakes

• Reading iostat’s first report. It’s a since-boot average. Use -y, or discard the first block.
• Trusting %util on NVMe or cloud volumes. Parallel devices break the assumption it was built on.
• Alerting on a fixed %iowait threshold. It fires on healthy backups and stays silent during real stalls on busy hosts.
• Changing the I/O scheduler first. Going from mq-deadline to none on NVMe is sometimes right, but it’s a tuning step, not a diagnosis, and it fixes nothing if you’re throttled.
• Adding swap to fix swapping. More swap on a memory-starved box means more disk traffic, not less.
• Stopping at the block layer. If every storage metric looks clean and the app is still slow, you’re measuring at the wrong layer, not measuring a healthy system.
• Testing with the wrong benchmark. A sequential dd tells you nothing about a workload doing random 4K synchronous writes. Use fio and match the real access pattern.

Best practices worth the effort

• Record PSI continuously. Storage incidents are short and bursty. Without history you’re guessing about something that already ended.
• Capture a baseline while things are healthy. Save an iostat -xy sample and a PSI reading from a normal day. “Is 8ms bad?” is unanswerable without one.
• Keep sar running. The sysstat package can archive metrics on a schedule, which means sar -d can show you what happened at 3am. Enable it before you need it.
• Separate hot write paths. Database data, database logs and application logs on separate volumes stops one from starving the others.
• Cap container logs. An uncapped JSON log driver quietly filling the root volume is a recurring cause of host-wide stalls.
• Alert on latency and pressure, not utilisation. Users feel w_await and PSI full. Nobody has ever felt a percentage of device busy time.

Frequently asked questions

What is a normal or acceptable I/O wait percentage?

There isn’t one, and any article that gives you a single number is guessing. Iowait scales with how idle your CPUs are, so the same workload produces very different figures on a busy server and a quiet one. Judge against your own baseline, and prefer PSI, which measures lost time directly rather than as a share of idle.

Can high I/O wait be caused by something other than the disk?

Regularly. Memory pressure driving swap and page cache eviction is the most common non-disk cause. Network filesystems, hypervisor contention on shared hosts, and cgroup or cloud volume throttling all produce the same symptom. The block device being genuinely slow is one possibility among several.

Why does iostat show 100% utilisation when the disk isn’t busy?

Because %util measures the fraction of time at least one request was in flight, which made sense for devices that handled one request at a time. NVMe drives and cloud volumes process many requests in parallel, so they can be continuously occupied while nowhere near capacity. Read aqu-sz and the await columns instead.

How do I find which process is causing I/O wait?

Start with pidstat -d 1, which needs no special kernel settings. List processes in uninterruptible sleep with ps -eo state,pid,comm and filter for state D to see who is actually blocked. iotop -o gives a live view but needs kernel.task_delayacct=1 on recent kernels for its percentage columns to populate.

Does adding more RAM reduce I/O wait?

Often, yes, and it’s underrated. More RAM means a larger page cache, fewer reads reaching the device, and less swapping. If vmstat shows sustained swap activity or memory pressure tracks I/O pressure, RAM is likely the cheapest fix available.

Should I change the I/O scheduler to fix high I/O wait?

Only after you know the device is the bottleneck. Check the current setting with cat /sys/block/sda/queue/scheduler. On NVMe, none is a common and sensible default because the hardware already does the queuing. On spinning disks, mq-deadline or bfq give better latency behaviour. It’s a tuning knob, not a diagnostic, and it will not help if you’re being throttled.

Why is my application slow when iostat says storage is fine?

Because iostat only sees the block layer. Latency in NFS, FUSE, overlay filesystems, journal serialisation or filesystem locking is entirely invisible to it. Run ext4slower or xfsslower alongside biolatency: if the filesystem is slow and the device is fast, the gap between them is your problem.

The one thing worth remembering

High I/O wait is a symptom that points in a general direction, not a diagnosis. It’s a slice of idle time, averaged across cores, measured at one specific layer of the stack. It goes up when nothing is wrong and stays flat when something is very wrong.

Read PSI to find out whether anyone is actually suffering. Read r_await, w_await and aqu-sz to find out whether the device is the constraint. Compare filesystem latency against block latency to find out whether you’re even looking at the right layer. Do those three things and the answer usually falls out in a few minutes, instead of after a week of arguing with your hosting provider about a volume that was fine all along.

Need someone to look at your storage stalls?

I work with teams on exactly this kind of problem, usually the ones that have already survived a round of “the host says the disk is fine”. Things I can help with:

• Diagnosing a live or recurring I/O stall and identifying which failure family it belongs to
• Setting up PSI, sar and per-cgroup I/O metrics in Prometheus and Grafana, with alerts that fire on latency rather than utilisation
• Separating cloud volume throttling from genuine device saturation, and sizing storage correctly instead of guessing upward
• Tuning writeback, dirty ratios and I/O schedulers for a specific workload, with before-and-after measurements
• Tracing container and Kubernetes I/O contention through cgroup v2 limits, overlay filesystems and log drivers
• Benchmarking with fio against your real access pattern so capacity decisions rest on something other than a dd run

If you’ve got an iostat capture, a PSI reading or just a graph that doesn’t make sense, send it over and I’ll tell you what I see in it.

The post Troubleshooting High I/O Wait Without Blaming the Disk appeared first on John Nessime.

That framing is where most of these arguments go sideways. On one box serving a normal website, all three of these servers will handle your traffic without breaking a sweat. The web server is almost never the slow part. The difference shows up months later, at 2am, in who gets paged and why.

So this post compares Nginx vs Apache vs Caddy on the three things that genuinely change your on-call life: how each one behaves when it runs out of capacity, who owns the certificate lifecycle, and where configuration lives when someone other than you needs to change it. Then a decision procedure, the mistakes I keep running into, and an FAQ.

Why the benchmark comparison picks the wrong winner

Almost every throughput comparison you’ll find measures static file serving on an idle machine with no application behind it. That is a real measurement of a thing that is not your bottleneck. If you’re proxying to PHP-FPM, Gunicorn, or a Node process, the request spends milliseconds in the web server and everything else in your code and your database.

The honest exception: if you are running a CDN origin, terminating TLS at a high connection rate, or serving genuinely large volumes of static assets, the connection handling model does show up in your bill. That is a real workload and Nginx has earned its reputation there. It just isn’t most projects.

For everyone else, the useful question is not “which is fastest” but “which chores am I choosing to keep.” That’s what the rest of this is about.

Failure family one: the process model decides how you fall over

This is the invisible one. Nobody notices it during a migration and everybody notices it during a traffic spike.

Apache: a hard worker ceiling, and a queue behind it

Apache’s behaviour depends entirely on which Multi-Processing Module is loaded. With prefork, you get one process per connection, capped by MaxRequestWorkers. When you hit that cap, new connections don’t get rejected. They sit in the kernel’s listen backlog waiting for a worker to free up.

The symptom is the worst kind: the site “hangs.” Load average is unremarkable. The application log is quiet because the requests never reached the application. Your uptime check times out but a manual curl from the box works fine. People spend hours looking at the database.

Check which MPM you’re actually running before you theorise about anything:

# RHEL, AlmaLinux, Rocky
httpd -V | grep -i "Server MPM"

# Debian and Ubuntu
apache2ctl -V | grep -i "Server MPM"

The reason prefork survives is mod_php. It isn’t safe with Apache’s threaded MPMs, so PHP stacks historically pinned themselves to prefork and inherited its concurrency profile. The Apache project itself now points people at PHP-FPM over mod_proxy_fcgi instead, which frees you to run mpm_event and gets you HTTP/2 as a side effect, since mod_http2 disables itself under prefork.

If you inherit an Apache box, that migration is usually the single highest-value change available to you, and it does not require leaving Apache.

Nginx: cheap connections, different limits

Nginx runs a small number of worker processes, each with an event loop. An idle keepalive connection costs a file descriptor and a little memory, not a process. Ten thousand of them is not a crisis.

Its ceiling is worker_processes multiplied by worker_connections, plus whatever your file descriptor limit allows. When you hit it, Nginx says so plainly in the error log rather than silently queueing. That’s a genuine operational advantage: the failure announces itself.

The trade is that when Nginx is proxying, saturation moves downstream. You stop seeing worker exhaustion and start seeing 502s and 504s from an overloaded PHP-FPM pool or app server. The web server is fine; you now have to go find the real limit.

Caddy: goroutines, and a softer edge

Caddy is a Go binary and handles connections with goroutines that the runtime multiplexes onto OS threads. Practically, that sits between the other two: connections are cheap like Nginx, but you don’t tune a worker count at all.

The failure mode is correspondingly softer and, honestly, harder to reason about. Instead of a clean “workers exhausted” line, you get memory growth and garbage collection pressure. There’s no knob you turned wrong, which is pleasant right up until you need to explain what happened.

Failure family two: who owns the certificate lifecycle

This is the one that actually causes outages, and it is the sharpest genuine difference between the three.

With Nginx or Apache, certificate management is a separate moving part. Certbot or another ACME client writes new files to disk on a timer, and the running server keeps serving whatever it loaded at startup. If nothing tells it to reload, the certificate on disk is valid and the certificate on the wire is expired. Users see a browser interstitial while your monitoring, if it checks the filesystem, reports everything fine.

The fix is to attach the reload to the renewal itself, not to a separate cron entry that might drift:

# Run the renewal end to end without touching real certificates.
# This exercises the challenge and the hook, which is the point.
certbot renew --dry-run

# Attach the reload to renewal so it only fires when a cert actually changed
certbot renew --deploy-hook "systemctl reload nginx"

The distinction that matters: --deploy-hook runs only when a certificate was actually renewed, so you’re not reloading the server twice a day for nothing. A blanket post-hook reloads regardless.

Caddy removes this category of failure. Certificate issuance and renewal are built into the server itself, it swaps the certificate in memory when it renews, and there is no external hook to forget. Point several instances at the same storage backend and they coordinate rather than each hammering the CA. If you have ever been paged for an expired certificate, this alone is a serious argument.

It is not free of foot-guns, though. On-demand TLS, where Caddy obtains a certificate during the handshake for a hostname it has never seen, is genuinely useful for multi-tenant setups where customers point their own domains at you. Enabled without restriction, it is an open door: anyone who points a DNS record at your IP can make you request a certificate. Caddy’s own documentation is blunt that it must be both enabled and restricted, and the mechanism is an ask endpoint you host:

{
    on_demand_tls {
        ask http://localhost:9123/ask
    }
}

https:// {
    tls {
        on_demand
    }
    reverse_proxy 127.0.0.1:3000
}

Caddy asks that endpoint whether a hostname is allowed before it will issue. It should be a fast indexed lookup against your own tenant table, returning in milliseconds, because a TLS handshake is being held open while it answers. The older interval and burst rate-limiting options are no longer recommended and should be removed if you still have them in a config.

One more Caddy default worth knowing: the admin API listens on localhost port 2019 and accepts a full config replacement without authentication. That is fine bound to loopback and catastrophic if you helpfully expose it while debugging a container.

Failure family three: where the configuration lives

This is the axis people underweight, and it’s usually the one that actually decides the answer.

Apache supports .htaccess: per-directory configuration, read at request time, changeable by anything that can write to the directory. No reload, no root access. That is why the entire shared hosting world is built on Apache, and why control panels like cPanel and DirectAdmin, and hosts like InterServer, assume it. A WordPress plugin can rewrite its own rules and it just works.

The costs are real and they’re the mirror image of the benefit. With overrides enabled, Apache walks the directory tree looking for .htaccess files on every single request. And your security rules can be silently overridden by whatever gets deployed into a directory, by anyone who can deploy.

Nginx has no equivalent and this is deliberate. Everything is central, and a change requires a reload. Rewrites are handled with try_files rather than per-directory rule files. If you own the box, that’s a feature: nothing changes routing behind your back and the running configuration is exactly what’s in version control. If you host other people’s applications, it’s a support burden, because every customer rewrite rule becomes a ticket for you.

Caddy is central too, either a Caddyfile or JSON. The difference is that config changes can go through the admin API as an atomic swap rather than a file edit plus a signal, which suits automated pipelines nicely.

Nginx: where it wins and where it doesn’t

Where it wins: high connection counts on modest hardware, static and cached content, TLS termination in front of an application fleet, and the sheer weight of the ecosystem. Every stack you’ll meet has a known-good Nginx config, and everyone you hire has already read one.

Where it doesn’t: the DNS behaviour catches people constantly. A hostname written directly into proxy_pass, or into an upstream block, is resolved once at startup and cached for the lifetime of the process. When your upstream is a cloud load balancer or a container that moves, Nginx keeps sending traffic to an IP that stopped being yours. Nothing in the log says “stale DNS”; you just get connection timeouts to an address that looks plausible.

Forcing re-resolution means putting the name in a variable and giving Nginx a resolver, because that’s what moves the lookup from config parse time to request time:

location / {
    # resolver moves lookups to runtime; valid= caps how long a result is reused
    resolver 127.0.0.53 valid=30s ipv6=off;

    # the variable is the trick: it defers evaluation to per-request
    set $upstream_host app.internal.example.com;
    proxy_pass http://$upstream_host:8080$request_uri;
}

Note the $request_uri on the end. Once a variable appears in proxy_pass, Nginx stops doing its usual URI rewriting for you, so you have to pass the path explicitly. Miss that and every request lands on the upstream’s root. Also worth knowing: an upstream block defined elsewhere with the same name will take precedence over the runtime lookup, which produces a very confusing afternoon.

The other honest downside is governance. Nginx is owned by F5, and a longtime core developer forked it to freenginx after a dispute over how security issues were being handled, with an earlier fork, Angie, already in existence. Mainline Nginx remains overwhelmingly dominant and I would not choose a fork for a client project today. But “who decides what ships” is a fair question to ask about any dependency, and this one now has a public answer.

Apache: where it wins and where it doesn’t

Where it wins: anywhere applications need to configure the web server without touching the web server. Shared hosting, multi-tenant WordPress, legacy applications that ship their own rewrite rules. The module ecosystem is enormous and modules load at runtime rather than requiring a rebuild. And the documentation is, quietly, the best of the three: nearly every directive has a page explaining its interactions.

Where it doesn’t: memory per connection, if you’re still on prefork. Slow clients holding processes open is the specific scenario that hurts, because a process that’s waiting on a phone on bad mobile data is a process nobody else can use. Putting a CDN or a proxy in front absorbs a lot of that, which is a large part of why Apache origins behind Cloudflare behave so much better than bare ones.

I would not migrate a working Apache installation to Nginx purely for performance. I would move it from prefork and mod_php to mpm_event and PHP-FPM first, then measure, and quite often stop there.

Caddy: where it wins and where it doesn’t

Where it wins: anything where certificate management is the operational risk. Small teams without a dedicated ops person, internal tooling with a long tail of hostnames nobody tracks, multi-tenant products serving customer-owned domains, container stacks where a single static binary with no runtime dependencies is a real simplification. A working HTTPS reverse proxy is genuinely a few lines.

app.example.com {
    encode zstd gzip
    reverse_proxy app:3000
}

That obtains a certificate, redirects HTTP to HTTPS, negotiates compression and proxies, with no ACME client, no cron, no reload hook.

Where it doesn’t: the ecosystem gap is real. When something unusual breaks, there are fewer Stack Overflow answers and fewer engineers who have seen it before. Adding a plugin means rebuilding the binary with xcaddy rather than dropping in a module, which is fine in CI and annoying on a box you SSH into. And there’s a subtler cost: automation you don’t understand is still automation you’re responsible for. Being paged about ACME behaviour you’ve never had to learn is a worse afternoon than being paged about Certbot.

On upstream DNS, Caddy dials static upstreams by name so resolution happens at dial time rather than being frozen at config load. When the set of addresses changes, the dynamic a upstream module with a refresh interval is the built-in answer, which is a cleaner story than Nginx’s variable trick.

A decision procedure for Nginx vs Apache vs Caddy

Work down this list and stop at the first one that matches. It’ll be right more often than a benchmark.

1. Do applications you don’t control need to change server behaviour? If yes, Apache. Nothing else has .htaccess, and reimplementing it in tickets is not a plan.
2. Is TLS renewal your top operational risk? Small team, no dedicated ops, lots of hostnames, or customer-owned domains: Caddy. It deletes a whole class of incident.
3. Are you terminating a very high connection rate, or serving heavy static content? Nginx. This is the workload it was designed around.
4. Does the team already run one of these well? Keep it. Operational familiarity beats a 20% throughput difference you were never going to measure.
5. Is there a working Apache install and no specific complaint? Fix the MPM and PHP handler, then reassess. Most “we need Nginx” tickets are actually “we need to stop using prefork.”
6. None of the above? Nginx for anything that will be handed to another engineer. Caddy for anything you’ll run yourself.

Common mistakes

• Renewing certificates without reloading. The single most common self-inflicted TLS outage on Nginx and Apache. Use a deploy hook and verify it with a dry run.
• Migrating for benchmark numbers. You inherit a new failure mode and a team that has never debugged it, in exchange for latency nobody could perceive.
• Leaving Apache on prefork after moving to PHP-FPM. The migration is only half done, and you’re still paying prefork’s memory cost while also giving up HTTP/2.
• Hostnames baked into proxy_pass with dynamic upstreams. Works perfectly until the upstream IP changes, then fails in a way that looks like a network problem.
• Enabling Caddy’s on-demand TLS without an ask endpoint. You’ve handed strangers the ability to make you request certificates.
• Exposing Caddy’s admin API beyond loopback. Unauthenticated full config replacement is exactly as bad as it sounds.
• Monitoring the certificate file instead of the handshake. The file is almost never the thing that’s wrong. Check what the server actually presents, from outside the box.

How I’d decide, in practice

• Pick based on who has to change the config six months from now, not on requests per second today.
• Whichever you choose, dump the effective configuration rather than reading files: nginx -T, apachectl -S, caddy adapt --config /etc/caddy/Caddyfile. Include files hide surprises.
• Monitor TLS expiry as an external handshake check from a service like Better Stack or a Grafana Cloud synthetic, not as a filesystem check on the box.
• Put something in front. A CDN like Cloudflare or Fastly absorbs slow clients and connection floods, which flattens most of the architectural differences here.
• Run one server type per environment. Mixing Nginx in staging and Apache in production means your rewrite rules are untested by definition.
• Validate before reloading, every time. nginx -t, apachectl configtest and caddy validate all exist so you don’t find out from users.

Frequently asked questions

Is Caddy production ready?

Yes. It’s been running real workloads for years and its certificate automation removes more operational risk than its smaller ecosystem adds, for most teams. The caveat is support surface, not stability: when you hit something unusual, there are fewer people who have hit it before you.

Can I run Nginx and Apache together?

Yes, and it’s a common pattern: Nginx on 80 and 443 handling TLS and static files, proxying dynamic requests to Apache on a local port so applications keep their .htaccess behaviour. You get Nginx’s connection handling and Apache’s per-directory config. The cost is two servers to patch, two logs to correlate, and real client IP to forward correctly.

Which is best for WordPress?

Apache if plugins need to write rewrite rules, which on a typical multi-plugin site they do. Nginx if you control the site and are willing to translate those rules into try_files and static location blocks once. Plenty of high-traffic WordPress runs on Nginx; it just means plugin-generated rules are your job now.

Does Nginx really re-resolve DNS for upstreams?

Not by default. A literal hostname in proxy_pass or an upstream block is resolved at startup and reused indefinitely, regardless of TTL. You get runtime resolution by using a variable together with a resolver directive, and remembering that this changes how the request URI is passed.

Should I switch from mod_php to PHP-FPM?

In almost all cases yes. It’s the change that unlocks Apache’s threaded MPMs and HTTP/2, and it separates PHP’s lifecycle from the web server’s so you can restart one without the other. Check first for anything relying on PHP settings in Apache config or .htaccess, since those move to a .user.ini file with different syntax.

Does the freenginx fork mean I should stop using Nginx?

No. Mainline Nginx under F5 remains the overwhelmingly dominant option with the broadest packaging and support. The fork is worth knowing about as context on project governance rather than as a migration target, and I wouldn’t put a client on a fork without a specific reason.

Which uses the least memory?

Nginx, generally, with Caddy close behind and Apache furthest away when running prefork. But the gap narrows sharply once Apache is on mpm_event, and on a VPS from a provider like Hetzner or DigitalOcean the memory your application uses will dwarf all three. Measure your own stack before treating this as a deciding factor.

The one thing worth remembering

The Nginx vs Apache vs Caddy question is not a performance question. All three are fast enough for what you’re doing. What differs is which operational chores each one hands back to you: Apache gives applications control at the cost of memory and predictability, Nginx gives you central control at the cost of owning every rewrite rule and the certificate reload, and Caddy takes the certificate lifecycle off your plate at the cost of a smaller ecosystem when things get strange.

Pick the set of chores your team can actually carry. Then go fix the thing that’s genuinely slow, which is almost certainly not the web server.

Need a second opinion on your web server setup?

Most of the work I do here is unglamorous and pays for itself quickly. Things I can help with:

• Auditing an existing Nginx, Apache or Caddy config and telling you what will break under load before it does
• Migrating Apache from prefork and mod_php to mpm_event with PHP-FPM, without breaking existing .htaccess behaviour
• Translating a working Apache rewrite set into an Nginx configuration, with a test plan so you know it matches
• Fixing certificate renewal that runs but never reloads, including deploy hooks and external expiry monitoring
• Setting up Caddy for multi-tenant or customer-domain hosting with on-demand TLS properly restricted
• Diagnosing intermittent 502s, 504s and stale-upstream timeouts that look like network faults and aren’t

Send me your config, an error log excerpt, or the output of nginx -T and tell me what you’re seeing. I’d rather look at the real thing than guess from a description.

The post Nginx vs Apache vs Caddy: What Actually Decides It in Real Projects appeared first on John Nessime.

That gap is the one that costs you, because it is invisible from the control plane. Kubernetes did exactly what you asked. The problem is that a readiness probe only answers one of the four questions that Kubernetes zero-downtime rollouts depend on, and it is not the question that usually bites.

This post walks through the four failure families in the order they usually show up: traffic arriving too early, traffic arriving too late, capacity disappearing faster than it is replaced, and the load balancer that was never watching your endpoints at all. Each one has a different fix, and stacking all four is what actually gets you a clean deploy.

What a zero-downtime rollout actually requires

Three independent things have to be true at the same time, and each is owned by a different part of the system.

• Capacity never dips below what traffic needs. Owned by the Deployment’s rolling update strategy.
• No pod receives traffic before it can serve it. Owned by the readiness probe.
• No pod receives traffic after it stops serving it. Owned by graceful termination, and this is the one nobody configures.

Miss any one and you get errors. Miss the third and you get errors that look like an application bug, because the pod that failed the request no longer exists by the time you go looking for it.

Failure family one: traffic arrives before the pod can serve it

This is the well-documented one. With no readiness probe, a pod is considered Ready as soon as its containers are running, which for most runtimes means “the process was forked”, not “the process can answer HTTP”. The pod lands in the Service’s EndpointSlice, kube-proxy programs it, and requests start arriving while the app is still loading config, warming a connection pool, or compiling templates.

The readiness probe that lies

Adding a probe is easy. Adding one that means something is the part people skip. Two patterns cause trouble:

A readiness endpoint that returns 200 from a static handler. If /healthz is wired up before the rest of the app is, the probe passes while the service is still useless. The readiness endpoint should be the last thing your app enables, after migrations have run, caches are primed and the pool has connected.

A readiness endpoint that checks downstream dependencies. This one feels correct and is actively dangerous. If your readiness check pings the database, then a five-second database blip marks every replica unready simultaneously. The Service loses all its endpoints, traffic has nowhere to go, and a brief degradation becomes a full outage. Readiness answers “can this pod serve traffic”, not “is the whole system healthy”. Check the dependency in your app’s request path and return a sensible error, or expose it on a separate diagnostic endpoint that nothing routes on.

Let the startup probe own the boot budget

For anything slow to boot, do not stretch initialDelaySeconds on the readiness probe until it covers the worst case. That delay applies on every restart forever, and it makes recovery slower than it needs to be. Use a startup probe instead: while it is running, liveness and readiness probes are suppressed entirely, and once it succeeds the fast probes take over.

# The startup probe owns the boot window. failureThreshold x periodSeconds
# is the total budget: 30 x 5 = 150 seconds before the kubelet gives up.
startupProbe:
  httpGet:
    path: /healthz/started
    port: http
  periodSeconds: 5
  failureThreshold: 30

# Readiness is fast and cheap. Two consecutive failures pull the pod out
# of rotation within about 6 seconds.
readinessProbe:
  httpGet:
    path: /healthz/ready
    port: http
  periodSeconds: 3
  timeoutSeconds: 2
  failureThreshold: 2

# Liveness is deliberately slower and points at a DIFFERENT endpoint.
# This one restarts the container, so it should only fire on real deadlock.
livenessProbe:
  httpGet:
    path: /healthz/live
    port: http
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3

Pointing liveness and readiness at the same URL is a common and expensive mistake. Under load, a slow response fails both: readiness pulls the pod out of rotation, which is right, and liveness kills the container, which is wrong. You lose the pod entirely instead of letting it recover, and the remaining replicas absorb its traffic, get slower, and fail their own liveness checks. That is how a latency spike turns into a rolling restart of the whole Deployment.

Failure family two: traffic arrives after the pod has stopped

This is the invisible one, and in my experience it accounts for most of the leftover errors after someone has “already added readiness probes”.

When a pod is deleted, two things happen in parallel, not in sequence:

1. The kubelet begins the shutdown sequence: run the preStop hook if one exists, then send SIGTERM to the container’s main process.
2. The endpoints controller marks the pod as terminating in its EndpointSlice, and every consumer of that data has to notice and reprogram: kube-proxy on every node, your ingress controller, your service mesh sidecars, and any cloud load balancer with its own target registry.

Nothing coordinates those two tracks. Kubernetes offers no guarantee that routing has converged before your process gets SIGTERM. On a small cluster the gap might be under a second. On a busy cluster with a few hundred nodes and an ingress controller reconciling on its own schedule, it can be several seconds. Every request that lands in that window hits a socket that is closing or already closed, and your users see a 502 or a connection reset.

The preStop sleep, and what it does not do

The fix is to delay SIGTERM so the routing fabric gets a head start. A preStop hook must finish before the TERM signal is sent, so sleeping in it does exactly that.

spec:
  # Total budget for preStop + graceful shutdown. Default is 30.
  terminationGracePeriodSeconds: 60
  containers:
    - name: api
      lifecycle:
        preStop:
          # Native handler, run by the kubelet. No shell needed in the image,
          # which matters for distroless and scratch-based builds.
          sleep:
            seconds: 15

If you are on an older cluster without the native sleep handler, the equivalent is an exec hook, which does require a shell and a sleep binary in the image:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 15"]

Be clear about what this buys you. The sleep does not drain anything. Your application keeps serving normally during it, unaware anything is happening. All it does is push SIGTERM later, so that by the time your process starts shutting down, nothing is sending it new work. Draining in-flight requests is still your application’s job, on receipt of SIGTERM: stop accepting new connections, finish what is open, then exit.

There is no universally correct sleep value, and anyone who gives you one is guessing. Measure it in your own cluster, on a normal weekday, and add margin.

The grace period is a shared budget

This trips people up. The terminationGracePeriodSeconds countdown starts when the pod is marked Terminating, which is before the preStop hook runs, not after. The hook and your application’s shutdown both spend from the same clock. If the grace period is 30, the hook sleeps 25, and your app needs 10 seconds to drain, the app gets SIGKILLed mid-drain and you have made things worse.

Size it as: preStop sleep + your longest realistic request + a few seconds of slack. And if your app holds long-lived connections such as WebSockets, gRPC streams or server-sent events, remember that removing an endpoint does nothing to connections that are already established. Either the grace period has to be long enough to see them out, or the client needs to reconnect on its own.

One more detail worth knowing before you go looking for it: probes accept their own terminationGracePeriodSeconds override, but only liveness and startup probes. You cannot set it on a readiness probe, because a failing readiness probe never kills anything.

Failure family three: the rollout deletes capacity faster than it adds it

The default rolling update strategy allows 25% of your pods to be unavailable during the update. On a Deployment with four replicas, that is one pod gone before its replacement is ready. If you are running near capacity, losing 25% of your fleet mid-deploy means queueing, timeouts and retries, none of which show up as a failed rollout.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 4
  # Rollout has 10 minutes to finish before it is marked failed.
  progressDeadlineSeconds: 600
  # A new pod must stay Ready this long before it counts. Catches pods
  # that pass the probe and then immediately crash.
  minReadySeconds: 15
  revisionHistoryLimit: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      # Never drop below the declared replica count.
      maxUnavailable: 0
      # Allow one extra pod on top while rolling.
      maxSurge: 1

Setting maxUnavailable: 0 is the right default for anything user-facing. The trade-off is real and worth stating: you now need headroom for one extra pod, and the rollout is slower because it is strictly sequential, waiting for each new pod to become Ready before removing an old one. On a Deployment with thirty replicas and a slow start-up, that can turn a two-minute deploy into fifteen. Raise maxSurge to a percentage if you have the node capacity and want the speed back.

minReadySeconds is the underrated one. Without it, a pod that passes its readiness probe and then falls over two seconds later still counts as a successful step, and the rollout marches on, replacing healthy pods with broken ones until the whole Deployment is bad. A short hold turns that into a stalled rollout you can catch, which is exactly what you want.

Rollouts are not the only thing that replaces pods, either. Node drains during a cluster upgrade evict pods too, and they respect PodDisruptionBudgets rather than your rolling update strategy. If you tuned one and not the other, an upgrade on a self-managed cluster, whether on cloud instances or your own VPS nodes from a provider like InterServer or Hetzner, will happily take out more replicas at once than any deploy ever did.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  # Keep at least three replicas up during voluntary disruptions.
  minAvailable: 3
  selector:
    matchLabels:
      app: api

Failure family four: the load balancer was never watching EndpointSlices

Everything above assumes traffic reaches your pods through a Service, where endpoint removal is the whole story. Plenty of production setups do not work that way.

Cloud load balancers that target pod IPs directly keep their own target registry. When a pod terminates, a controller has to call the cloud API to deregister it, that call has to be accepted, and the load balancer has to propagate the change to its own data plane. Then a configurable deregistration delay runs before the target is fully removed. That chain is slower and more variable than kube-proxy, often by an order of magnitude, and it explains why teams on managed clusters sometimes report that a preStop sleep of five seconds fixed nothing.

Two things help here:

• Pod readiness gates. These let an external controller add a condition to the pod that must be true before the pod counts as Ready. With a load balancer controller that supports them, a new pod is not considered Ready until the load balancer has actually registered it and passed its own health check. That closes the mirror image of the termination race, on the way in.
• Match your preStop sleep to the deregistration delay. If your target group is configured to drain for 30 seconds, a 5-second sleep is not enough. The sleep needs to cover the time until the load balancer genuinely stops sending new connections.

The same reasoning applies to an ingress controller running in the cluster. Ingress-NGINX, Traefik and friends watch EndpointSlices and reload their own config, which is fast but not instant, and each has its own graceful shutdown settings that need to be consistent with the pod’s grace period. If your edge sits behind Cloudflare or another CDN, be aware that it may retry an idempotent request against another origin, which quietly hides some of these errors from your users while leaving them in your logs. That is a good safety net, not a substitute for fixing the race.

Verifying a zero-downtime rollout instead of hoping

A rollout that has never been tested under load is a rollout you have not tested. The whole point of these failure modes is that they only appear when requests are in flight.

1. Generate steady traffic against the real ingress path. Not against a pod IP, and not through a port-forward. Those bypass exactly the layers you are trying to test.
2. Trigger a rollout with no image change, so you are testing the mechanism rather than your new code.
3. Watch EndpointSlices in a second terminal and note how long a terminating pod stays in the list.
4. Count non-200 responses. Anything above zero is a bug, not noise.
5. Repeat under realistic concurrency. A single-threaded curl loop will miss a 300ms window that a real traffic level would hit hundreds of times.

# Terminal 1: steady probe traffic, printing status and latency
while true; do
  curl -s -o /dev/null -w '%{http_code} %{time_total}n' https://api.example.com/
  sleep 0.2
done

# Terminal 2: watch endpoints appear and disappear in real time
kubectl get endpointslices -l kubernetes.io/service-name=api -w

# Terminal 3: roll the deployment without changing the image,
# then follow progress and stop on a stall
kubectl rollout restart deployment/api
kubectl rollout status deployment/api --timeout=10m

Do this once per service and record the numbers. If you already run Prometheus and Grafana, or a hosted stack like Grafana Cloud or Datadog, put a deploy annotation on your error-rate panel so the correlation is obvious next time instead of something someone has to notice by eye.

Troubleshooting

Rollout hangs, new pods never become Ready. The readiness probe is failing. kubectl describe pod shows the probe failure and the response it got. Check that the port name in the probe matches a declared containerPort name, that the path exists, and that the app is listening on all interfaces rather than only 127.0.0.1, which is a classic one when moving from a local docker-compose setup.

Probes fail only under load. Look at CPU limits before you look at anything else. A container being CPU-throttled cannot answer a probe within timeoutSeconds, and the default timeout is one second. This produces restart storms that look like an application bug and are actually a resource limit.

Errors persist after adding a preStop sleep. Either the sleep is shorter than your routing convergence time, or the load balancer is not driven by EndpointSlices at all. Time it: mark the moment of deletion, then watch how long the endpoint stays listed. Increase the sleep past that number and re-test.

Pods stuck Terminating for the full grace period. Your app is ignoring SIGTERM. This is extremely common when the container’s entrypoint is a shell script, because the shell runs as PID 1 and does not forward signals to the child. Use the exec form of ENTRYPOINT, or an init like tini, so your process actually receives the signal.

Rollout succeeded but the new version is broken. Roll back first, investigate second.

# Freeze a rollout mid-flight without reverting what has already landed
kubectl rollout pause deployment/api
kubectl rollout resume deployment/api

# Inspect revisions, then go back
kubectl rollout history deployment/api
kubectl rollout undo deployment/api --to-revision=3

Common mistakes

• Pointing liveness and readiness at the same endpoint, so a slow response restarts the container instead of just removing it from rotation.
• Checking databases or downstream APIs in the readiness probe, turning a dependency blip into a total outage.
• Leaving maxUnavailable at the default on a service running near capacity.
• Adding a preStop sleep longer than the grace period, so the app gets SIGKILLed before it can drain.
• Running a single replica and expecting a rolling update to be seamless. With one pod there is nothing to roll onto.
• Using RollingUpdate for workloads that cannot tolerate two versions running at once, such as a single-writer process or a schema change that is not backward compatible. Recreate exists for these, and it does mean downtime.
• Testing the deploy with no traffic flowing, which makes every one of these failure modes invisible.

Best practices

• Three separate endpoints: /healthz/started, /healthz/ready, /healthz/live. They answer different questions and should be allowed to disagree.
• maxUnavailable: 0 plus a maxSurge you have node capacity for, as the default for anything user-facing.
• A preStop sleep on every pod behind a Service, sized from a measurement rather than a blog post.
• A grace period that covers preStop plus your slowest realistic request, with slack.
• Handle SIGTERM properly in the application, and make sure it reaches PID 1.
• A PodDisruptionBudget on anything that matters, so node drains are as safe as deploys.
• minReadySeconds long enough to catch a pod that passes its probe and then dies.
• Annotate deploys on your dashboards, and treat any non-200 during a rollout as a defect rather than background noise.

FAQ

Do readiness probes alone give me zero-downtime deployments?

No. A readiness probe stops traffic reaching a pod that is not ready yet. It does nothing about the window between a pod being marked for deletion and the routing layer noticing, and nothing about capacity dipping during the rollout. You need the rolling update strategy and graceful termination as well.

How long should the preStop sleep be?

Long enough for everything that routes traffic to your pods to stop doing so. Measure it: delete a pod, watch the EndpointSlice, and time how long it stays listed. Behind a cloud load balancer, add its deregistration delay on top. Then confirm by running the load test with the value you picked.

Should the readiness probe check the database?

Almost never. It couples the availability of every replica to a single shared dependency, so one slow query can empty your Service of endpoints all at once. Handle dependency failures in the request path and return a meaningful error instead.

What is the difference between maxSurge and maxUnavailable?

maxSurge is how many pods you may run above the replica count during a rollout. maxUnavailable is how many you may drop below it. Surge costs resources; unavailability costs capacity. For zero downtime you want unavailability at zero and surge at whatever your nodes can absorb.

Why do I still get 502s on EKS, GKE or another managed cluster?

Cloud load balancers that target pod IPs maintain their own target registry, and deregistration goes through a cloud API rather than kube-proxy. That path is slower and more variable. Use pod readiness gates if your load balancer controller supports them, and size the preStop sleep against the target group’s deregistration delay rather than against kube-proxy.

Can I get zero-downtime rollouts with a StatefulSet?

The same probe and termination mechanics apply, but StatefulSets update pods one at a time in reverse ordinal order and cannot surge, so there is no extra pod covering the gap. Whether that is zero downtime depends entirely on whether your application tolerates losing one member at a time, which for most quorum-based systems it does and for a single-writer database it does not.

Does kubectl rollout status prove the deploy was clean?

It proves the desired number of pods reached Ready within the progress deadline. It says nothing about requests that failed on the way there. The only thing that proves a clean deploy is traffic flowing through the real ingress path with a zero error count.

The one thing worth remembering

Kubernetes zero-downtime rollouts are not a feature you turn on. They are the result of four separate things being configured correctly at once, and the readiness probe is only the most obvious of them. The one that actually drops requests in most clusters is the race at the other end: your process receives SIGTERM while the routing layer still believes the pod is a valid destination.

Delay the signal, size the grace period to cover the delay plus a real request, and then prove it with traffic flowing. Everything else is bookkeeping.

Need a second pair of eyes on your rollouts?

Most of this work is unglamorous: reading manifests, timing endpoint propagation, and finding the one Deployment that never got a preStop hook. Things I can help with:

• Auditing your Deployments and Helm charts for probe, surge and termination settings, with a prioritised list of what to change
• Measuring real endpoint propagation time in your cluster and sizing preStop and grace periods from that number
• Building a load-test harness that runs against a rollout in CI, so a regression fails the pipeline instead of the pager
• Tracking down 502s and connection resets that only appear during deploys, including cloud load balancer and ingress controller paths
• Splitting a single overloaded health endpoint into proper startup, readiness and liveness checks
• Adding PodDisruptionBudgets and drain-safety so cluster upgrades stop being an event

If you have a Deployment manifest, a kubectl describe pod output, or a graph showing your error rate during a deploy, send it over and I will tell you what I see.

The post Kubernetes Zero-Downtime Rollouts: Why Your Readiness Probe Isn’t Saving You appeared first on John Nessime.

Nothing was broken. That is the part people get stuck on. The container did exactly what it was told, the daemon did exactly what it was told, and the disk filled anyway. Docker’s default logging driver has no size cap and no rotation, and it has been that way for a long time on purpose.

This post covers Docker log rotation properly: what the defaults actually do, why the daemon.json change you already made did nothing, why deleting the log file did not give you your disk back, how the available drivers differ in the ways that matter under load, and what to do when you need space in the next sixty seconds.

The default is unbounded, and that is documented behaviour

Docker’s default logging driver is json-file. It captures stdout and stderr from the container’s main process and appends one JSON object per line to a file on the host, one file per container. Each line carries the message, whether it came from stdout or stderr, and a timestamp.

The important part is the option defaults. For json-file, max-size defaults to unlimited, max-file defaults to 1, and compress defaults to false. Put together, that means: one file, no rotation, grows until something else breaks.

Docker’s own documentation is upfront about why. Keeping json-file unrotated by default preserves backwards compatibility with older engines and suits situations where Docker is acting as a runtime under Kubernetes, where the kubelet handles rotation itself. It is a deliberate choice that happens to be the wrong one for almost every standalone host.

Before you change anything, find out what you actually have. These three commands answer different questions and you want all three:

# What driver is the daemon handing to newly created containers?
docker info --format '{{.LoggingDriver}}'

# What is this specific container actually using right now?
docker inspect --format '{{.HostConfig.LogConfig}}' my-api

# Where does its log live, and how big has it got?
docker inspect --format '{{.LogPath}}' my-api
sudo du -h "$(docker inspect --format '{{.LogPath}}' my-api)"

The gap between the first command and the second is where most of the pain lives, and we will come back to it.

To find the offenders across the whole host, look at the per-container directories under the Docker data root. Each one holds that container’s log file plus a small amount of metadata, so the sizes are close enough to be useful:

sudo du -sh /var/lib/docker/containers/* | sort -rh | head -10

Container IDs are not memorable, so map the winner back to a name with docker ps --no-trunc or by grepping the ID against docker ps -aq. In my experience the top entry is almost never the application you were worried about. It is usually a reverse proxy logging every request, a health-check loop firing every few seconds, or a debug flag somebody set during an incident and never turned off.

Setting up Docker log rotation on the daemon

The daemon-wide fix goes in /etc/docker/daemon.json. If the file does not exist, create it. If it does, merge these keys in rather than overwriting it, because that file often already carries storage driver, address pool or registry mirror settings you do not want to lose.

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "compress": "true"
  }
}

What each key buys you:

• max-size caps a single file. Accepts an integer plus k, m or g. This is the setting that stops the runaway.
• max-file caps how many files are kept. When rotation would create one too many, the oldest is deleted. It is only effective if max-size is also set, which is the single most common misconfiguration I see.
• compress gzips rotated files. The active file stays uncompressed so it can still be appended to and read. Log text compresses extremely well, so this is close to free retention.

Two things trip people up here. First, every value in log-opts must be a JSON string, including the numeric ones. Writing "max-file": 3 without quotes is invalid and the daemon will refuse to start cleanly. Second, validate before you restart, because a malformed daemon.json on a production host is a much worse afternoon than a full disk:

# Confirm the file is valid JSON before touching the daemon
sudo python3 -c 'import json,sys; json.load(open("/etc/docker/daemon.json"))' 
  && echo "daemon.json parses OK"

sudo systemctl restart docker

Now do the arithmetic, because this is a budget and not a magic switch. Worst case per container is max-size multiplied by max-file. At 10m and 3, that is roughly 30 MB uncompressed per container, less once compression kicks in on the rotated files. Multiply by your container count and compare against the disk. On a small VPS plan from a provider like InterServer or Hetzner, where the whole root volume might be 20 to 40 GB shared with images and volumes, forty containers at 30 MB each is a real slice of your budget rather than a rounding error.

Why your daemon.json change did nothing

This is the failure mode that costs the most time, because it looks exactly like the fix not working.

Logging configuration is baked into a container’s host config when the container is created. Restarting the daemon does not rewrite it. Restarting the container does not rewrite it either, because docker restart stops and starts the same container object rather than making a new one. Your new limits apply to containers created after the daemon restart and to nothing else.

So the container that filled your disk keeps filling your disk, its log file still has no cap, and docker info cheerfully reports the new defaults. That mismatch is exactly what the second command in the earlier block is for.

To actually apply it, recreate the container and then verify rather than assuming:

# Compose: recreate the service so it picks up the new daemon defaults
docker compose up -d --force-recreate web

# Verify the container is really carrying the limits
docker inspect --format '{{.HostConfig.LogConfig}}' web

You want to see the driver name followed by the map of options. An empty map means the container is still unbounded no matter what the daemon says.

Note that recreating a container discards its existing log history along with the old container. If those logs matter for an ongoing investigation, copy them off first.

Pinning limits per service in Compose

Daemon defaults are a floor, not a policy. A chatty ingress proxy and a quiet cron worker do not deserve the same budget, and per-service configuration in Compose overrides daemon.json entirely. A YAML anchor keeps it from turning into copy-paste sprawl:

x-logging: &default-logging
  driver: json-file
  options:
    max-size: "10m"
    max-file: "3"
    compress: "true"

services:
  web:
    image: nginx:alpine
    logging: *default-logging

  worker:
    image: myorg/worker:latest
    logging:
      driver: json-file
      options:
        max-size: "50m"
        max-file: "3"
        compress: "true"

Be aware that the logging block replaces the daemon config rather than merging with it. If you specify a driver and forget the options, you get that driver with its defaults, which for json-file means straight back to unbounded. Half-specifying is worse than not specifying at all.

You deleted the log file and df did not move

Classic 3am mistake, and it is worth understanding rather than memorising.

When you rm the JSON log of a running container, you remove the directory entry. You do not remove the inode, because the Docker daemon still holds an open file descriptor pointing at it. The kernel keeps the data allocated until the last descriptor closes. So the file disappears from ls, the daemon keeps writing into a file you can no longer see, and df reports exactly the same usage as before. Space comes back only when you restart the daemon or recreate the container.

truncate -s 0 is the right tool. It zeroes the file in place without unlinking it, so the descriptor stays valid, the daemon keeps appending to the same inode, and the blocks are released immediately:

sudo truncate -s 0 "$(docker inspect --format '{{.LogPath}}' my-api)"

There is a cost. Truncating out from under the daemon can leave an in-flight docker logs -f stream reading from an offset that no longer exists, so the follow appears to hang with no new output until you reattach. It is a known behaviour rather than corruption, and reattaching clears it. Live with it during an incident, but do not build it into a cron job.

The same reasoning applies to logrotate rules pointed at /var/lib/docker/containers/*/*.log. Rotating with copytruncate works mechanically, but you are reaching into files the daemon considers its own, and you inherit the follow-stream issue on every rotation cycle. Docker’s own docs warn against external tools touching those files. If the driver can rotate for you, let it.

Choosing a driver, and what each one costs

json-file

The default. Widest tooling compatibility, since every log shipper and every scraper knows this format. docker logs works natively. Its genuine downside is the format itself: JSON per line is verbose, so you pay in disk and in parse cost on read. Reach for it when something downstream reads those files directly.

local

Docker’s recommendation for standalone hosts, and the one I reach for first. It uses a more efficient on-disk format, and critically it rotates and compresses by default, retaining five files of 20 MB each per container out of the box. docker logs works exactly as normal. The trade-off is that the format is Docker’s own, so third-party agents that expect to tail JSON files cannot read it directly. If your entire log path is docker logs plus a shipper that talks to the daemon, this is the safest default you can pick.

journald

Hands container output to systemd’s journal, which means you get journalctl filtering, structured fields, and the journal’s own size management via SystemMaxUse in journald.conf. This is a good fit on hosts where you already read system logs that way and want one retention policy instead of two.

The catch nobody mentions: the journal has rate limiting on by default. A container in a crash loop can trip it, and messages get dropped with a note saying how many were suppressed. That is fine for noise and terrible for the incident you are actually debugging. If you go this route, look at RateLimitIntervalSec and RateLimitBurst before you need them, not after.

syslog, fluentd and the remote drivers

These push logs off the box to rsyslog, Fluentd, Loki, or a hosted platform. Disk pressure genuinely goes away because the host stops being the system of record. That is the right destination for anything beyond a handful of hosts.

What you buy with it is a dependency. Your logging path now has a network hop, and the failure modes of that hop become your failure modes. Whether it is self-hosted Grafana Loki, or a managed service like Grafana Cloud, Better Stack or Datadog, the operational question is the same: what happens to the container when the collector is unreachable? Answer that during setup rather than during an outage.

none

Discards output entirely. Legitimate for a container that already writes its own logs to a mounted volume, and a trap everywhere else, because docker logs returns nothing and you will waste twenty minutes concluding the container is broken.

The blocking behaviour worth knowing about

By default the logging path is blocking. If the driver cannot keep up, the container’s write to stdout blocks, which means your application stalls. On a local file that is rarely an issue. Pointed at a remote collector that has gone slow, it means a logging problem becomes an application latency problem.

docker run -it 
  --log-opt mode=non-blocking 
  --log-opt max-buffer-size=4m 
  alpine ping 127.0.0.1

Non-blocking mode buffers in memory and drops messages once the buffer is full rather than stalling the writer. The default buffer is 1 MB. You are choosing which failure you prefer: a slow application, or missing log lines exactly when things are going wrong. There is no answer that is right everywhere, but for a user-facing service I would rather lose lines than add latency.

Dual logging, and why docker logs still works

If you switch to a remote driver and find docker logs still returns output, that is dual logging. When the configured driver cannot serve reads, the engine keeps a local cache using the local driver so the command still works. That cache rotates by default, limited to five files of 20 MB each per container before compression.

Useful to know for two reasons. It explains why disk usage does not drop to zero after you move logging off the box. And it means the local cache has its own retention that you can tune or disable separately from the driver you configured.

Reclaiming space when the disk is already full

Order matters here. Free space first, fix the cause second, verify third.

1. Confirm logs are actually the problem. Compare du -sh /var/lib/docker/containers against docker system df, which reports images, volumes and build cache. Images are often the real culprit and truncating logs will not help you.
2. Identify the largest log files with the du and sort command from earlier.
3. Copy anything you need out first, if the logs are part of an active investigation.
4. Truncate the offenders with truncate -s 0. Never rm.
5. Set the limits in daemon.json, validate the JSON, restart the daemon.
6. Recreate the containers so the limits take effect, then confirm with docker inspect on each one.
7. Add a disk alert. A full disk should never be discovered by a human noticing something else is broken.

Step seven is the one that actually stops this recurring. Rotation caps the damage, but a container that suddenly starts logging a hundred times more than usual is telling you something, and a disk-usage alert on the Docker data root is how you hear it.

Troubleshooting

Rotation is configured but files still grow past the cap. The container predates the change. Check docker inspect --format '{{.HostConfig.LogConfig}}' and recreate it.

The daemon will not start after editing daemon.json. Almost always invalid JSON, or unquoted values in log-opts. Read the actual error with journalctl -u docker --no-pager -n 50 rather than guessing.

max-file is set but only one file ever appears. max-file does nothing without max-size. Set both.

docker logs -f hangs after showing existing output. Something truncated the file underneath the daemon, usually an external logrotate rule. Reattach to recover, then remove the rule.

docker logs returns nothing at all. Check the driver. none discards everything, and some remote drivers only serve reads through the dual-logging cache.

Disk usage did not drop after deleting log files. The daemon still holds the descriptors. Restart the daemon to release the inodes, and use truncate next time.

Logs vanish sooner than the retention you configured. On journald, check journal rate limiting and SystemMaxUse. On a remote driver in non-blocking mode, check whether the buffer is overflowing.

Common mistakes

• Assuming a daemon restart applies the new limits to running containers. It does not.
• Setting max-file without max-size and believing rotation is on.
• Writing numeric values unquoted in log-opts.
• Overwriting an existing daemon.json instead of merging, and silently dropping storage or network settings.
• Using rm on a live container’s log file and concluding the disk report is broken.
• Specifying a logging.driver in Compose without its options, which resets that service back to the driver’s own unbounded defaults.
• Pointing host logrotate at the Docker containers directory when the driver could have handled it.
• Sizing rotation per container without multiplying by container count against the actual disk.

Best practices

• Set logging limits in daemon.json as part of host provisioning, before any container exists. This belongs in your Ansible role or cloud-init, not in a runbook.
• Prefer the local driver on standalone hosts unless something specifically needs to read the JSON files.
• Turn on compress. Log text compresses well enough that you get more retention for the same disk.
• Size per service rather than uniformly. Give the noisy proxy a tight cap and the service you actually debug a generous one.
• Do the multiplication. Worst-case host usage is max-size × max-file × container count, and it should be a fraction of your disk, not most of it.
• Alert on Docker data root usage at a threshold that leaves you time to act.
• Treat on-host logs as a short buffer, not an archive. Anything you need next quarter belongs in a log platform, not on the VPS.
• Verify with docker inspect after any change. docker info tells you the daemon’s intent, not what your containers carry.

Frequently asked questions

Does Docker rotate container logs automatically?

Not with the default driver. json-file ships with max-size unlimited and max-file set to 1, so a single file grows without bound. The local driver does rotate and compress by default. If you have not explicitly configured Docker log rotation, assume it is off.

Where are Docker container logs stored on the host?

Under the Docker data root, one directory per container, which on a standard Linux install means /var/lib/docker/containers/. Rather than assembling the path by hand, ask the daemon: docker inspect --format '{{.LogPath}}' <container>. That works regardless of a custom data-root.

How do I clear Docker logs without stopping the container?

Truncate the file to zero bytes with truncate -s 0 on the path from {{.LogPath}}. Do not use rm: the daemon holds an open descriptor, so deleting the directory entry leaves the data allocated and frees no space until the daemon restarts.

Why did my daemon.json rotation settings not apply?

Because logging config is fixed at container creation time. A daemon restart changes what new containers get; it does not rewrite existing ones, and docker restart reuses the same container object. Recreate the container, then confirm with docker inspect --format '{{.HostConfig.LogConfig}}'.

Should I use the local driver or json-file?

Use local if nothing outside Docker reads the log files directly, since it rotates and compresses by default and uses less disk for the same content. Use json-file if an agent or shipper on the host tails those files, because it cannot parse the local format. Either way docker logs behaves the same.

Can I use logrotate for Docker container logs instead?

You can, and plenty of hosts do, but the driver options are the better tool. A copytruncate rule pointed at the containers directory reaches into files the daemon treats as private, and can leave docker logs -f streams stalled after each rotation. Use logrotate for logs your applications write to mounted volumes, and let the driver handle stdout.

How much disk should I budget for container logs?

Worst case is max-size × max-file per container, multiplied by how many containers run on the host. At 10m and 3 files across thirty containers that is roughly 900 MB before compression. Pick numbers that leave the total at a comfortable fraction of the disk, and keep long-term retention off the box.

The one thing worth remembering

Docker log rotation is not a feature you turn on once at the daemon and forget. It is a property each container carries from the moment it is created, and every container that existed before your change is still running under the old rules.

So the working sequence is always the same: set the limits, restart the daemon, recreate the containers, then verify with docker inspect rather than trusting docker info. Add a disk alert on the Docker data root so the next surprise arrives as a notification and not as a failed deploy. Everything else in this post is detail around that.

Need a hand with container logging on your hosts?

Log sprawl is one of those problems that is cheap to fix in an hour and expensive to leave alone. If you would rather have someone go through it with you, this is the kind of work I take on:

• Auditing every container on a host for logging driver and retention, and producing the recreate plan that actually applies the limits
• Sizing a rotation budget against your real disk, container count and log volume, per service rather than one flat number
• Emergency recovery on a full Docker host, including the cases where deleting files did not give the space back
• Choosing between json-file, local, journald and shipping off-box, including the blocking behaviour trade-offs under load
• Setting up centralised logging with Loki, rsyslog or a hosted platform, with sane retention on both ends
• Baking logging limits into provisioning so new hosts are never born unbounded, plus the disk alerts to catch what rotation cannot

If you want a second opinion, send me your daemon.json, the output of docker ps and a du -sh of your containers directory, and I will tell you what I would change.

The post Docker Logs Ate My Disk: A Working Guide to Log Drivers and Rotation appeared first on John Nessime.

The worse version of this problem is the mirror image. You reboot a server to clear something, come back to read what led up to it, and the journal starts at the reboot. Everything before is gone. Not rotated, not compressed, not archived somewhere clever. Gone, because the journal was living in RAM and nobody ever told it otherwise.

Both of those are the same misunderstanding wearing different clothes: not knowing where your logs actually live. This post walks through the journald vs rsyslog split on a modern Linux server, what each one really stores, the three failure modes that bite in production, and how to decide which of the two is the copy you keep.

What actually happens when a process logs a line

On any systemd-based distribution, systemd-journald is the first stop for almost everything. It is not one option among several. It sits underneath.

• Anything a unit writes to stdout or stderr, because systemd wires those to the journal by default.
• Kernel messages, read from the kernel ring buffer.
• Classic syslog(3) calls from libc, which land on the /dev/log socket that journald owns.
• Native journal API calls from anything linked against libsystemd, which carry structured key-value fields instead of a flat string.
• Audit records, if Audit=yes is in effect.

journald writes all of that into indexed binary journal files. Then, on most server distributions, rsyslog gets a second copy of the same events and writes them out as text into /var/log/messages, /var/log/secure, /var/log/maillog and friends.

That handoff happens one of two ways, and which one your box uses matters more than most people expect. More on that below. First, the part that costs you an outage.

Before changing anything, look at the effective journald configuration rather than the file you think is authoritative. Distributions ship drop-ins under /usr/lib/systemd/journald.conf.d/ that quietly override upstream defaults:

# Print the merged configuration, including every drop-in, in load order
systemd-analyze cat-config systemd/journald.conf

This is the single most useful command in this whole post. Half the arguments about “what the default is” evaporate once you run it, because upstream systemd and your distribution frequently disagree about ForwardToSyslog.

Failure one: the journal that was never on disk

journald’s Storage= setting defaults to auto. That word does more work than it looks like. Under auto, journald writes to /var/log/journal/ only if that directory already exists. If it does not, journald falls back to /run/log/journal/, which is tmpfs. Memory. Wiped on reboot.

Nothing warns you. journalctl works fine, colours are pretty, filters work. You just silently have no history.

The quickest test is boot history. If the journal is persistent you see multiple boots; if it is volatile you see exactly one:

journalctl --list-boots

To make it persistent, create the directory, let systemd-tmpfiles apply the correct ownership and ACLs, then restart the daemon and flush anything still held in RAM:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
sudo journalctl --flush
journalctl --list-boots

The systemd-tmpfiles step is the one people skip. The journal directory needs specific group ownership and ACLs so that unprivileged users can read their own entries and members of the systemd-journal group can read everything. Creating the directory with a bare mkdir and walking away usually works, but it is the kind of “usually” that produces a confusing permissions ticket six months later.

If you want the behaviour to be explicit rather than inferred from a directory’s existence, set it in /etc/systemd/journald.conf.d/00-storage.conf:

[Journal]
Storage=persistent

A drop-in file is better than editing the main config, because package upgrades will not fight you over it.

Failure two: paying for the same log line twice

Once the journal is persistent, you are storing everything twice on a default server build: once as compressed binary in the journal, once as text under /var/log/. Two stores, two completely separate retention policies, neither of which knows the other exists.

The journal is capped by SystemMaxUse= and SystemKeepFree=. Left unset, they default to 10 percent and 15 percent of the filesystem respectively, and journald honours whichever is stricter. Recent systemd also caps those calculated defaults so they do not grow without bound on a large disk, which is worth knowing if you have ever wondered why a huge volume did not produce a proportionally huge journal.

The text copy is capped by logrotate, configured somewhere completely different, usually /etc/logrotate.d/rsyslog, on a weekly or daily schedule with its own rotate count.

Measure both before you tune either:

# What the journal is actually consuming on disk
journalctl --disk-usage

# What everything under /var/log costs, sorted, biggest last
sudo du -sh /var/log/* | sort -h

On a small VPS this stops being academic quickly. A modest instance from a provider like InterServer, Hetzner or DigitalOcean often has /var sharing a single root volume of 20 to 40 GB. A journal allowed to take 10 percent of that, plus four weeks of rotated text logs, plus a database, plus a container image cache, is how you end up paged at two in the morning for something that is not a real incident.

If you decide the text files are the copy you keep, put the journal on a short leash:

[Journal]
# Hard ceiling on total journal size
SystemMaxUse=200M
# Never let the journal push free space below this
SystemKeepFree=1G
# Discard entries older than this regardless of size
MaxRetentionSec=2day

Those three do different jobs and you generally want at least two of them. SystemMaxUse bounds the journal itself. SystemKeepFree protects the rest of the filesystem from the journal. MaxRetentionSec bounds it in time, which is what auditors and data-retention policies actually care about.

To reclaim space immediately without waiting for the next rotation:

# Trim archived journal files until the total falls below 500M
sudo journalctl --vacuum-size=500M

# Or trim by age instead
sudo journalctl --vacuum-time=7d

One caveat that surprises people: vacuuming only removes archived journal files. The currently active file is never deleted. If almost all your usage is in one large active file, run journalctl --rotate first, then vacuum.

Failure three: two rate limiters, both silent

This is the one that is invisible until it matters, and it is the real reason to understand the journald vs rsyslog relationship rather than treating them as interchangeable.

There are two independent rate limiters in the default pipeline, and both drop messages quietly.

journald’s limiter

journald applies RateLimitIntervalSec= and RateLimitBurst= per service, defaulting to 30 seconds and 10000 messages. Exceed the burst inside the interval and the rest of that service’s messages in that window are discarded. Not queued. Discarded.

The one piece of good news is that journald tells you, in the journal itself. Grep for it:

journalctl --grep="Suppressed" -n 50 --no-pager

A chatty web server or a container using the journald log driver will hit this. Rather than disabling rate limiting globally, override it for the one unit that needs it, using a drop-in:

# /etc/systemd/system/nginx.service.d/logging.conf
[Service]
LogRateLimitIntervalSec=30s
LogRateLimitBurst=50000

Per-unit values override journald.conf for that unit only, which is exactly what you want. Turning the global limiter off means one runaway process can fill your disk and take the box down. Turning it up for the single service you know is noisy is a bounded decision.

rsyslog’s limiter

If rsyslog is reading via the imjournal module, it applies its own rate limit on top, defaulting to 20000 messages per 600 seconds. A message can therefore survive journald’s limiter, land in the journal, and still never reach /var/log/messages or your remote log host.

That is the nasty case, because your local investigation finds the events but your centralised alerting never fired. Two people look at two sources and reach opposite conclusions about whether something happened.

rsyslog exposes counters for this through its statistics module, including how many messages it read from the journal, how many it submitted onward, and how many it discarded to rate limiting. Enable impstat if you run anything log-volume-sensitive; the discard counter is the number that tells you whether your pipeline is lying to you.

How rsyslog gets its copy: imjournal or imuxsock

There are two mechanisms, they behave differently under load, and running both at once is a classic source of duplicate lines.

imuxsock: journald pushes a copy to a socket

With ForwardToSyslog=yes in journald, journald writes a classic syslog-formatted copy of each message to a dedicated socket, and rsyslog reads it with imuxsock.

• Where it wins: simple, no state file, no second rate limiter, and no possibility of rsyslog reading back its own output and looping.
• Where it does not: you get the flat syslog view only. Structured journal fields such as the originating unit are not carried across.

imjournal: rsyslog pulls from the journal

imjournal reads journal files directly and keeps a state file so it can resume where it left off after a restart.

• Where it wins: structured fields survive the trip. If you want to route or template on the systemd unit name rather than parsing it out of a message string, this is the only option of the two.
• Where it does not: extra rate limiter to reason about, a state file that can drift, and a documented risk that a corrupted journal database causes rsyslog to re-read the same entries in a loop. rsyslog’s own documentation recommends using imuxsock instead unless you specifically need the structured data.

A representative imjournal load line, with the rate limit stated explicitly rather than left implicit:

module(load="imjournal"
       StateFile="imjournal.state"
       Ratelimit.Interval="600"
       Ratelimit.Burst="20000")

Writing the defaults out by hand is worth the two extra lines. The next person to debug missing messages will see the limiter exists instead of assuming there isn’t one.

To find out which mechanism your box is using right now:

grep -R "imjournal|imuxsock" /etc/rsyslog.conf /etc/rsyslog.d/ 2>/dev/null
grep -R "ForwardToSyslog" /etc/systemd/journald.conf /etc/systemd/journald.conf.d/ /usr/lib/systemd/journald.conf.d/ 2>/dev/null

If the first command returns both modules and the second returns ForwardToSyslog=yes, you have a duplication problem. Pick one path and disable the other.

Only one of the two leaves the box

This is the argument that settles most journald vs rsyslog debates in practice. journald is a local store. In a stock server install it has no mechanism for shipping logs to another host. There are separate systemd components for journal upload and reception, and third-party projects that read the journal and forward it, but none of that is in the default path.

rsyslog forwards as a first-class feature, and it does the part that actually matters: it buffers when the destination is unreachable.

global(workDirectory="/var/spool/rsyslog")

action(type="omfwd"
       target="logs.example.net"
       port="514"
       protocol="tcp"
       action.resumeRetryCount="-1"
       queue.type="LinkedList"
       queue.filename="fwd_loghost"
       queue.maxDiskSpace="1g"
       queue.saveOnShutdown="on")

Line by line, because these are not decoration:

• workDirectory is where spool files get written. Without it, the disk queue has nowhere to go.
• protocol="tcp" rather than UDP. UDP syslog silently drops under congestion, which defeats the point of forwarding at all.
• queue.type="LinkedList" makes the action asynchronous so a slow destination does not block local processing.
• queue.filename is what actually enables disk assistance. In-memory queue first, spilling to disk when it fills. The name must be unique across every action in the config.
• queue.maxDiskSpace caps that spool. Set it, or a long outage at the far end fills the disk you were trying to protect.
• queue.saveOnShutdown="on" persists whatever is still queued across a service restart.

One honest caveat on action.resumeRetryCount="-1". Infinite retry is the right default for “never lose a log line”, but there are reported cases where a relay under sustained back-pressure stops accepting new inbound messages while retrying forever. If your host is a relay rather than a leaf, test that behaviour deliberately with the destination firewalled off before you trust it.

rsyslog is not the only way off the box. Grafana Alloy into Loki, Vector, and Fluent Bit all read the journal directly and speak modern backends, and hosted platforms such as Better Stack, Papertrail or Datadog will take a plain syslog stream if you would rather not run the storage side. The mechanism differs; the decision does not. Something has to own the write path off the machine, and journald on its own is not it.

journald vs rsyslog: how I would decide

Stop asking which is better. Ask which one is the copy you are willing to be judged on, then make the other one cheap.

1. Do logs need to leave this machine? If yes, rsyslog or a dedicated shipper owns the outbound path. That is settled before anything else.
2. Do you troubleshoot mostly by unit? If journalctl -u something -b is your muscle memory, journald is your primary read path and should get the retention budget.
3. Do existing tools read text files? Fail2ban, logwatch, older SIEM collectors and a lot of home-grown scripts parse /var/log/*. Turning rsyslog off breaks them quietly.
4. How much disk do you actually have? Under about 40 GB, keeping two full copies is an unforced error. Cap one hard.
5. Is tamper-evidence a requirement? journald supports Forward Secure Sealing, which detects after-the-fact modification of journal files. Shipping to an append-only remote store is the more commonly accepted answer, and the two are not mutually exclusive.

The configuration I reach for first on a general-purpose server: journald persistent with a hard cap for a few days of fast local querying, rsyslog kept for the syslog files that other tools expect plus reliable forwarding off-box, imuxsock as the handoff unless something genuinely needs structured fields, and logrotate retention trimmed down because the remote copy is the one that matters for anything older than a week.

Troubleshooting

• Journal only shows the current boot. Volatile storage. journalctl --list-boots confirms it, and the fix is the persistent-storage sequence above.
• Events in journalctl but not in /var/log/messages. Either rsyslog is not reading from the journal at all, or the imjournal rate limiter is discarding. Check which input module is loaded first.
• Every line appears twice in the text logs. Both imjournal and imuxsock are active while ForwardToSyslog=yes. Disable one path.
• Bursts of messages disappear during incidents. That is a rate limiter, and incidents are exactly when services get loud. Search for suppression notices and raise the limit for that specific unit.
• journalctl reports corruption or behaves oddly. Run journalctl --verify. If files are damaged, journalctl --rotate starts a fresh active file so new writes are clean, then vacuum the bad archives.
• “Journal has been rotated since unit was started.” Not an error. Rotation happened mid-session, so journalctl -u cannot map the full range. Re-run the query without the unit filter or widen the time range.
• Disk still full after vacuuming. Vacuum skips the active journal file. Rotate first, then vacuum again.

Common mistakes

• Assuming the journal is persistent because journalctl returns results. It always returns results. The question is how far back.
• Disabling rsyslog on a box where fail2ban, logwatch or a scraper still reads /var/log/secure. Nothing errors. Detection just stops.
• Setting RateLimitBurst=0 globally to “stop losing logs”, then having a crash-looping service fill the disk in an afternoon.
• Editing /etc/systemd/journald.conf directly and being surprised when a vendor drop-in overrides it. Use a drop-in of your own with a name that sorts later.
• Forwarding over UDP because it is the one-line version. It drops silently under exactly the load that produced the logs you wanted.
• Enabling a disk-assisted queue without queue.maxDiskSpace, turning a remote outage into a local disk-full outage.
• Tuning journald limits on a machine whose real problem is that one application logs every health check at info level. Fix the source first.

Best practices

• Make storage explicit. Storage=persistent or Storage=volatile, never left to auto on a server you care about.
• Always set SystemKeepFree alongside SystemMaxUse. The first bounds the journal, the second protects everything else on the volume from it.
• Decide consciously which store is authoritative, and write it in the runbook. Two stores with no stated owner means neither gets maintained.
• Use exactly one journal-to-rsyslog path. Both is duplication; neither is silent data loss.
• Override rate limits per unit, not globally.
• Monitor journalctl --disk-usage as a metric, not as something you check after the alert.
• Ship off-box with TCP and a bounded disk-assisted queue, and test it with the destination firewalled off before you rely on it.
• Keep the local copy short and the remote copy long. Local disk is the expensive place to store history.

Frequently asked questions

Can I just disable rsyslog and use journald only?

On a self-contained box with no remote logging and nothing parsing text files, yes, and it saves real disk. Before you do it, grep your configuration management and cron jobs for /var/log/ paths, and check whether fail2ban or any monitoring agent reads those files. That is where this bites people.

Can I disable journald and use rsyslog only?

Not meaningfully. journald is how systemd captures unit stdout and stderr, so it stays in the path regardless. What you can do is set Storage=volatile with a small RuntimeMaxUse so it holds a short in-memory window and rsyslog owns everything durable.

Where are journald logs stored?

/var/log/journal/ when persistent, /run/log/journal/ when volatile. They are indexed binary files, not text, so grep and tail do not work on them directly. Use journalctl, or journalctl -o json if you want to pipe structured output into something else.

Why does journalctl show entries that never reached /var/log/messages?

Three usual causes: rsyslog is not reading the journal at all, the imjournal rate limiter dropped them, or a severity filter such as MaxLevelSyslog or an rsyslog rule excluded that facility or priority. Check in that order.

Is the binary journal format a problem for log analysis?

Locally, no. Structured fields and fast filtering are genuinely better than parsing text with regex. It becomes a problem when you need to move logs somewhere else, because almost every collector expects text or JSON. That conversion step is precisely the role rsyslog or a modern shipper plays.

How much disk should the journal be allowed to use?

Work backwards from how far back you actually query locally. If you rarely look past yesterday, a few hundred megabytes with MaxRetentionSec set to two or three days is plenty. Anything older belongs in a remote store where it is cheaper and survives the machine.

Does journald compress logs automatically?

Yes. Compress= defaults to enabled and applies to entries above a size threshold, which is one reason the journal is often smaller than the equivalent text files despite carrying more metadata. It does not remove the need for a size cap.

The one thing worth remembering

The journald vs rsyslog question is not a contest. On a stock server both are running, both are storing the same events, and the defaults were chosen by your distribution rather than by you. The failure modes all come from that: a journal in RAM that vanishes on reboot, two uncoordinated retention policies eating the same disk, and two rate limiters dropping messages without anyone noticing.

Run systemd-analyze cat-config systemd/journald.conf and journalctl --list-boots on your servers. Two commands, thirty seconds, and you will know whether the logs you would reach for during an incident are actually there.

Need help sorting out your logging pipeline?

Logging is one of those areas where the defaults work well enough to hide the problem until an incident makes it expensive. If any of the above sounded familiar, these are the things I take on:

• Auditing journald and rsyslog on existing servers and telling you exactly where each log line lives, how long it survives, and what is being silently dropped.
• Fixing volatile journals, runaway /var growth and duplicate text-and-binary storage on small VPS instances.
• Setting up reliable off-box forwarding with disk-assisted queues, TCP or RELP transport, and TLS where it is required.
• Migrating from text-file scraping to a structured pipeline into Loki, Elasticsearch or a hosted log platform, without losing the alerts you already depend on.
• Tuning rate limits and retention per service so noisy applications stop hiding the messages that matter.
• Writing the runbook that says which store is authoritative, so the next engineer does not have to reverse-engineer it during an outage.

Send me your journald.conf, your rsyslog.conf, or the output of journalctl --disk-usage and I will tell you what I see before we talk about scope.

The post journald vs rsyslog: Where Your Linux Logs Actually Live appeared first on John Nessime.

Either way you end up looking at Podman, reading a Podman vs Docker comparison table that says “daemonless” and “rootless by default”, and trying to work out whether those words are worth a migration.

Here is the short version, and I will spend the rest of the post defending it. The architectural difference between the two engines is real and it matters more than the feature tables suggest. The migration cost is also real, and it is almost entirely concentrated in four places that comparison posts tend to skip. Whether the switch is worth it depends on which of those four you actually depend on.

This post covers the one architectural difference everything else follows from, an honest profile of each engine with where it wins and where it does not, the four things that genuinely break when you switch, and a decision procedure you can run against your own setup rather than against a generic one.

The difference everything else follows from

Docker is a client and a daemon. The docker CLI is a thin thing that talks to dockerd, which runs as root and owns every container on the host. Podman has no daemon. When you run podman run, the container is started as a child of your own process and supervised by a small monitor, not handed off to a privileged background service.

Two consequences follow, and they are the whole argument.

Membership of the docker group is equivalent to root. Not “close to root”, not “root if misconfigured”. The daemon runs as root and will happily create a container that bind-mounts the host’s root filesystem, so anyone who can talk to the socket can read or write anything on the box. Docker’s own documentation says as much. This is the finding that shows up in access reviews, and it is not a bug you can configure away without switching to rootless mode.

Process supervision belongs to whoever started the container. With Docker, the daemon restarts your containers according to the restart policy, and if the daemon wedges you have a bad afternoon regardless of what the containers are doing. With Podman, containers are ordinary processes, so systemd supervises them the same way it supervises everything else on the machine, with the same logging, dependency ordering and failure handling.

Podman’s rootless mode builds on user namespaces: UID 0 inside the container maps to your unprivileged UID on the host. A break-out lands in a normal user account rather than on a root shell. Docker has a rootless mode too and it works, but it is an opt-in you configure rather than the path of least resistance, which in practice means far fewer installations run it.

Docker: where it wins and where it doesn’t

Where it wins

• The ecosystem assumes it. Testcontainers, VS Code Dev Containers, Traefik’s container discovery, Portainer, most CI runners, half the SDKs on your machine: all of them expect a Docker socket. Some work fine against Podman’s compatible socket. Some do not, and finding out which is your problem.
• Compose is first-party. Docker Compose is maintained by the same people who maintain the engine, tracks the Compose specification closely, and gets features like --wait and profiles as soon as they exist. Nothing in the Podman world matches that.
• Every answer to every question already exists. Fifteen years of Stack Overflow. That is not a technical merit but it is a real operational one at 2am.
• Cross-platform experience is smoother. Docker Desktop on macOS and Windows is a polished product, and the VM it manages is largely invisible.

Where it doesn’t

• The root daemon is a standing liability. One privileged process that everything flows through, plus a group that is root by another name.
• Docker Desktop is licensed, and the terms are specific. Docker’s own license page states Desktop is free for personal use, education, non-commercial open source, and small businesses defined as fewer than 250 employees and less than ten million dollars in annual revenue. Cross either threshold and commercial use needs a paid subscription. Government entities need one regardless. Docker Engine and the Moby project are not affected. This catches people out because the software installs without asking for a key.
• systemd integration is bolted on. You can wrap a Compose stack in a unit file, but systemd is supervising a wrapper, not the containers.
• Restarting the daemon touches everything. An engine upgrade is a whole-host event rather than a per-service one.

Podman: where it wins and where it doesn’t

Where it wins

• Rootless is the default, not a project. You get the safer posture by doing nothing special, which is the only kind of security control that survives contact with a busy team.
• Quadlet is genuinely better than restart policies. You write a declarative unit file, systemd generates a real service from it, and container lifecycle becomes indistinguishable from any other service on the box. This is the feature I would actually miss if I went back.
• No licensing question to answer. Podman is Apache 2.0 with no commercial-use restriction, and on Red Hat Enterprise Linux, Fedora, Rocky and AlmaLinux it is the packaged, supported option already sitting in the repos.
• Pods and Kubernetes YAML are first-class. podman kube play runs a Kubernetes manifest locally, and podman generate kube goes the other way. If your destination is Kubernetes, the mental model transfers.
• Per-user isolation comes free. Two developers on a shared build host cannot see or stop each other’s containers, because they are running in separate user namespaces rather than as siblings under one daemon.

Where it doesn’t

• Compose is a compatibility story, not a native one. More on this below, because it is the single biggest source of migration pain.
• Rootless networking has sharp edges. Traffic goes through a userspace network stack, and the behaviour is not identical to a root bridge. Source IPs, host-loopback access and privileged ports all behave differently, and you will meet at least one of those.
• Swarm and Docker plugins do not exist. Podman does not implement Swarm mode, and Docker volume and network plugins are not supported. If you use either, the conversation is over.
• Desktop platforms mean a VM you can see. Podman on macOS and Windows runs a Podman machine, and the seams show more than Docker Desktop’s do.
• Documentation assumes Docker. Every vendor quickstart, every README, every “run this to try it” snippet is written for docker run. Most translate directly. The ones that do not cost you an hour each.

The four things that actually break

This is the part worth reading twice. Command compatibility is not the problem; podman accepts almost every docker flag you know. The problems are elsewhere.

1. Compose

There are two routes and they are not equivalent.

podman-compose is a separate Python implementation. It covers the common cases and diverges on the uncommon ones. Container naming is the one that bites scripts first: it has historically used underscores where Docker Compose v2 uses hyphens, so anything referencing project-web-1 by name stops finding it.

The Podman socket plus real Docker Compose is the route I would take. Podman exposes a Docker-compatible REST API over a systemd socket, and the actual Compose binary talks to it:

# Start the Docker-compatible API socket for your user.
systemctl --user enable --now podman.socket

# Confirm it exists before pointing anything at it.
ls -l /run/user/$(id -u)/podman/podman.sock

# Point the real Docker Compose at Podman instead of dockerd.
export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock
docker compose up -d

You keep the tool that tracks the Compose specification and swap only the engine underneath. The catch is that the API is a compatibility layer, not the Docker daemon, so behaviour can differ at the edges. Run your real compose file through it before you commit to anything.

2. Anything that mounts the Docker socket

Traefik discovering containers by label, Portainer, Testcontainers in your test suite, any CI job that does Docker-in-Docker. All of these want /var/run/docker.sock. The Podman socket serves the same API and many of them work against it unchanged, but “many” is doing work in that sentence. Inventory what mounts the socket before you plan the migration, not after.

3. Ports below 1024

A rootless container cannot bind a privileged port, because the process genuinely is not privileged. If your compose file publishes 80 and 443, it will fail, and the error will not obviously say why.

Three ways out, in the order I would try them:

1. Publish a high port and put a host reverse proxy in front. Nginx or Caddy running as a normal system service, listening on 443, proxying to 8443. No kernel tuning, and you probably wanted TLS termination on the host anyway.
2. Lower the privileged port threshold. The kernel exposes net.ipv4.ip_unprivileged_port_start, and setting it to 0 lets any user bind any port. It works, and it is a host-wide change that weakens a defence for every process on the machine, not just yours. Understand that before you write it into a playbook.
3. Run that one container rootful. Podman supports it. Mixing modes is a legitimate choice as long as it is deliberate and documented rather than something that happened.

4. Restart on boot

This is the one that catches people who tested on a laptop and shipped to a server. Rootless containers belong to a user session, and a user’s services are stopped when their last session ends. Reboot the box and nothing comes back, no matter what restart policy you set.

The fix is two parts. Enable lingering so the user’s systemd instance runs without a login, and define the container as a Quadlet unit rather than relying on a restart policy:

# Let this user's services run without an active login session.
sudo loginctl enable-linger appuser

# Verify. Linger=yes is the line that matters.
loginctl show-user appuser --property=Linger

Then the unit itself. Quadlet files look like systemd units with a Podman-specific section, and a generator turns them into real services at boot and on daemon-reload. Rootless units live in ~/.config/containers/systemd/; system ones in /etc/containers/systemd/.

# ~/.config/containers/systemd/api.container

[Unit]
Description=Application API
After=network-online.target

[Container]
# Fully qualified. Podman does not assume Docker Hub.
Image=registry.example.com/api:1.4.2
PublishPort=8080:8080
Volume=api-data.volume:/var/lib/api
Environment=NODE_ENV=production

[Service]
Restart=always

[Install]
# Starts at boot once lingering is enabled for this user.
WantedBy=default.target

# Regenerate services from Quadlet files, then start it.
systemctl --user daemon-reload
systemctl --user start api

# From here it is an ordinary service.
systemctl --user status api
journalctl --user -u api -f

Note that podman generate systemd, which you will find in older tutorials, is deprecated in favour of Quadlet. Podman’s documentation says it receives urgent bug fixes only and no new features. Do not build anything new on it.

One smaller trap while you are here: Podman does not assume Docker Hub for unqualified image names. nginx:alpine may prompt you to choose a registry or fail outright depending on configuration. Write docker.io/library/nginx:alpine, or whatever your registry is, everywhere. This is arguably the correct behaviour and it is still an afternoon of edits.

Podman vs Docker: how I would decide

Run these in order and stop at the first one that answers.

1. Do you use Swarm, or Docker volume or network plugins? Stay on Docker. Podman does not implement them and no workaround changes that.
2. Is a compliance requirement or an access review driving this? Switch, or at minimum move to rootless. “The docker group is root” is a finding you cannot argue with, and rootless Podman is the shortest path to closing it.
3. Are you on RHEL, Fedora, Rocky or AlmaLinux? Lean Podman. It is the packaged, supported, tested-by-the-vendor option on those distributions, and you are swimming with the current instead of against it.
4. Is Docker Desktop licensing a real cost for your organisation? Check the thresholds against your actual headcount and revenue before assuming either way. If you are over, this is a budget conversation with a free technical answer, which is a rare shape of problem.
5. Does your test suite or CI depend on the Docker socket? Prototype that specific dependency against the Podman socket first. It is the highest-risk item and the cheapest to test. If it fails, everything else is academic.
6. None of the above, and Docker is working? Stay. “Podman has nicer architecture” is true and it is not a business case. Migrations cost weeks, introduce risk, and the containers you are already running do not care which engine started them.

The honest summary: Podman is the better-designed engine and Docker is the better-supported one. Design wins when you are building something new or when a policy requirement forces the question. Support wins when you have a working system and a limited number of weeks.

If you do switch, do it in this order

1. Inventory the socket mounts. Grep your compose files and CI config for docker.sock. That list is your actual migration scope.
2. Stand up one non-critical service on Podman. A staging box, an internal tool, something on a spare VPS. Run it for a fortnight through at least one reboot before you draw conclusions.
3. Fully qualify every image reference. Do this in the compose files while they still run under Docker. It changes nothing there and removes a whole class of failure later.
4. Switch the engine, keep Compose. Podman socket plus DOCKER_HOST. Change one variable at a time so you know what broke.
5. Convert to Quadlet last, and only where it earns its place. Long-lived services benefit. A three-container stack you rebuild constantly is fine staying on Compose.

Arguments that don’t survive contact

• “Podman is a drop-in replacement.” The CLI nearly is. The surrounding tooling is not, and that is where the weeks go.
• “Daemonless means faster.” The engine is not your bottleneck. Rootless networking has historically been slower than a root bridge, not faster. Pick Podman for the security model, not for benchmark numbers.
• “Rootless means secure.” It means a break-out lands somewhere less catastrophic. A container running rootless with the host filesystem bind-mounted read-write is still a bad idea.
• “Docker is dying.” It is not. Docker Engine, Compose and Buildx are actively developed and hosted CI runners still ship them by default.
• “Just alias docker to podman and you’re done.” The alias handles the commands. It does not handle the socket, the ports, the boot behaviour or the image names.

Frequently asked questions

Can Podman run docker-compose files?

Yes, by two routes. Install podman-compose, which is a separate implementation with good but incomplete coverage, or enable the Podman socket and point real Docker Compose at it with DOCKER_HOST. The second gives better specification coverage because you are running the actual Compose tool. Neither is a guarantee; test your own file.

Is Podman actually more secure than Docker?

Its default posture is better. Rootless out of the box means no root daemon and no group that is root by another name. Docker can be configured to a similar posture with rootless mode, so the difference is less “Podman is secure and Docker is not” and more “Podman’s safe path is the easy path”. Defaults determine what most installations actually run.

Do I need to pay for Docker?

Docker Engine on Linux is open source and free. Docker Desktop is the licensed product. Per Docker’s terms it is free for personal use, education, non-commercial open source, and businesses with fewer than 250 employees and under ten million dollars in annual revenue; cross either threshold, or be a government entity, and commercial use requires a subscription. If you only run Docker Engine on Linux servers, the licensing question does not apply to you.

Why do my Podman containers disappear after a reboot?

Because rootless containers run under your user’s systemd instance, which is torn down when your last session ends. Run loginctl enable-linger for the service account and define the containers as Quadlet units with WantedBy=default.target. A Podman restart policy alone will not survive a reboot.

What is Quadlet and do I have to use it?

Quadlet is a systemd generator that turns declarative container definitions into real systemd services. You do not have to use it, but the alternative is hand-writing unit files, and the older podman generate systemd command is deprecated in favour of it. For anything long-lived on a server, Quadlet is the current answer.

Can Podman and Docker coexist on the same host?

Yes. They keep separate image and container stores, so nothing collides at the storage layer. Watch for port conflicts if both are running services, and be deliberate about which one a given script is talking to, because DOCKER_HOST and an alias can quietly disagree with each other.

Does Podman work with Kubernetes?

Better than Docker does, in the sense that it speaks the format natively. podman kube play runs a Kubernetes manifest on a single host and podman generate kube produces one from running containers. It is not a substitute for a cluster, but as a local development story for a Kubernetes destination it is the closest thing available.

The one thing worth remembering

The Podman vs Docker decision is not really about container engines. Both run OCI images and both will start your workload. It is about whether a root daemon and a root-equivalent group are acceptable in your environment, and whether you have the weeks to spend on the four things that break.

If a policy requirement is driving the question, switch and budget properly for the compose and socket work. If you are on an enterprise Linux distribution and starting something new, start on Podman, because you will be swimming with the current. If Docker is working and nothing external is forcing the question, the correct engineering answer is to leave it alone and go fix something that is actually broken.

Thinking about a migration and want a second opinion?

I help teams decide this question and then execute it without a surprise outage. Things I am usually brought in for:

• Auditing what on your hosts actually depends on the Docker socket, and scoping the migration honestly before anyone commits to it
• Testing your real compose files against the Podman socket and reporting what breaks, rather than guessing from a feature table
• Converting long-lived services to Quadlet units with proper systemd ordering, health gating and journald logging
• Sorting out rootless networking: privileged ports, reverse proxy placement, and the source-IP behaviour that surprises people
• Closing the “docker group equals root” audit finding, whether that means Podman, rootless Docker, or removing the group entirely
• Writing the decision up so it survives the next person who asks why you chose what you chose

If you are weighing this up, send me a compose file and a list of what mounts the socket. That is usually enough to tell you whether the switch is a fortnight or a quarter.

The post Podman vs Docker: Is the Switch Actually Worth It? appeared first on John Nessime.