No. And that question is the entire problem with this one.

This post is a reference for the PHP2Shell vulnerability: what actually broke inside PHP, which servers are exposed, how to patch each common setup, how to prove the fix is really running, and which quick fixes give you false comfort. Keep it open while you work through your servers.

What happened: the PHP2Shell vulnerability in plain terms

PHP2Shell is the nickname researchers gave to two flaws in the PHP engine itself, CVE-2026-109618 and CVE-2026-109619. PHP rates both 9.2 on CVSS v4.0, and both can let an unauthenticated attacker run code on a web server. The name is the researchers’, not the PHP project’s.

The dates matter here, because they decide if you were exposed and for how long:

• 20 October 2026: PHP announced an emergency release for two critical issues. Details leaked publicly the same day.
• 21 October 2026: fixed releases shipped: PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35, with a full advisory.
• 23 October 2026: PHP published an informational advisory for a third, much narrower issue, CVE-2026-115560, with a one-line mitigation and no emergency release.

The flaws were reported by Wiz Research, Calif and OpenAI’s Codex Security team, who used AI models to turn long-standing weaknesses into working exploits. The affected code goes back to PHP 5.4, so effectively every PHP version from the last decade is in scope.

The two bugs, without the jargon

CVE-2026-109618 (out-of-memory corruption). PHP enforces memory_limit by aborting a request the instant it crosses the limit. If that abort lands while PHP is still building internal request data, for example while parsing a multipart file upload, the data is left half-initialized. Request cleanup then treats it as valid and works with memory that can still hold attacker-controlled leftovers.

CVE-2026-109619 (stale global state). Some state owned by one request can survive after its memory has been freed, for example after a fatal error. The next request handled by the same worker can use that stale state before normal setup resets it.

Here is why this is worse than the usual plugin CVE. The bug lives in the runtime, not in your code. A site whose own code is flawless is still exposed, and updating WordPress, Laravel or Drupal changes nothing, because the PHP binary underneath comes from your OS packages, your host or your container image. The researchers also demonstrated reading memory from the PHP worker process, which can contain passwords, tokens and keys belonging to other people’s requests, without ever getting a shell.

Who is exposed, and why mod_php is the worst case

PHP’s advisory covers every installation that serves web requests: PHP-FPM, Apache with mod_php, php-cgi, FrankenPHP and LiteSpeed. The risk is not equal across them, though.

mod_php is the highest risk. Apache with mod_php runs request shutdown even when request startup already failed. That means simply running out of memory while PHP parses the request body is enough, and one unauthenticated request can trigger the attack before a single line of your application runs. PHP demonstrated this with default memory and input limits.

PHP-FPM is harder to hit, not safe. FPM shuts the worker down when a request fails early in setup, which blocks the cleanest path. Public reporting says one of the two bugs was still shown to reach code execution under FPM, through a path that depends on something the application does. Treat “we run FPM” as “we have slightly more time,” nothing more.

If you are not sure which mode you run, these two checks answer it on most Linux servers. The first lists Apache modules and shows a PHP module only if mod_php is loaded. The second lists running FPM services.

apachectl -M | grep -i php
systemctl list-units --type=service | grep -i fpm

How to fix PHP2Shell: it depends on where your PHP comes from

The fix is always the same in principle: run a patched PHP build. Getting there depends entirely on who builds your PHP. And there is a trap that runs in both directions. Linux distributions and long-term-support vendors received the patch early and backported it, so a patched package can carry an old-looking version like 8.3.6. Meanwhile, an unpatched process can keep running after the package updated. The version string answers neither question reliably.

Debian and Ubuntu packages

Debian, Ubuntu and Ondřej Surý’s widely used repository (packages.sury.org, also published as ppa:ondrej/php) all shipped fixed packages, including builds for some end-of-life branches. Update normally, then confirm the CVE is in the package changelog instead of reading the version number. Replace 8.3 with your branch.

sudo apt update && sudo apt upgrade
zgrep CVE-2026-109618 /usr/share/doc/php8.3-common/changelog.Debian.gz
sudo systemctl restart php8.3-fpm
sudo systemctl restart apache2

If zgrep prints a line, the fix is in the installed package. If it prints nothing, you are either on an older distro release that needs a paid extended-support channel, or on a repository that has not shipped the fix.

RHEL, AlmaLinux and Rocky Linux

Same idea with RPM. Update, then search the package changelog for the CVE. If you use Remi’s parallel packages, the package name follows its own pattern (for example php83-php-common), so query that name instead.

sudo dnf upgrade 'php*'
rpm -q --changelog php-common | grep CVE-2026-109618
sudo systemctl restart php-fpm httpd

One warning from PHP’s own advisory: the volunteer-run rpms.remirepo.net repository was not expected to have updates on release day, and PHP suggested switching to distribution packages or building from source if you depend on it. Check before assuming the update landed.

cPanel, CloudLinux and LiteSpeed

cPanel ships its own ea-php packages through EasyApache 4, and CloudLinux ships alt-php. CloudLinux backported the fix into supported and end-of-life branches, so most of its fixed builds keep version numbers below the php.net release. CloudLinux’s published procedure is to update both package families, then restart the services that run PHP so no worker keeps the old binary loaded.

yum clean all && yum update 'alt-php*' 'ea-php*' -y
/scripts/restartsrv_httpd

If your domains run PHP-FPM through cPanel, restart the FPM service as well. On LiteSpeed, restart the web server so every lsphp process respawns from the new build. Public analysis says LiteSpeed’s LSAPI behaves more like FPM than mod_php, which buys time but does not remove the need to patch.

DirectAdmin with CustomBuild

CustomBuild compiles PHP from source, so you get the fix when CustomBuild offers the patched release and you rebuild. Refresh the version list, confirm the fixed release appears for each PHP version you run, then build.

cd /usr/local/directadmin/custombuild
./build update
./build versions
./build update_versions

Read the ./build versions output before running the last command. If a branch you use still shows an unpatched release, apply the mitigations below for that branch until it updates. Compiling takes a while on small servers, so plan for it rather than starting it at peak traffic.

Docker and container images

A container never patches itself. If your image is built FROM an official php tag, rebuild with a fresh pull of the base image and redeploy. If your Dockerfile pins the base by digest, the digest has to change or nothing happens.

docker compose build --pull
docker compose up -d

Then run php -v inside the running container, not on the host, to confirm the version the app actually uses.

Managed hosting versus unmanaged VPS

On managed hosting, the host owns the PHP runtime, and several large hosts announced they had patched their fleets. Your job there is to get written confirmation, not to assume. On an unmanaged VPS from Contabo, Interserver or any similar provider, nobody patches PHP but you. That is the trade you accepted for the lower price and full root access, and this week is when it gets collected.

End-of-life PHP: 8.1 and older

The PHP project released no fixed builds below 8.2. Because of the severity, its security team published minimal source backports for PHP 7.4, 8.0 and 8.1 for people who build from source. Beyond that, your options are a distribution or vendor that backports (Ubuntu’s own PHP 8.1 on 22.04, the Surý repository, Freexian ELTS for older Debian, CloudLinux and TuxCare, HeroDevs), or upgrading. Note that PHP 8.2 itself reaches end of life on 31 December 2026, so if you are touching an 8.2 server now, plan the next upgrade at the same time.

Prove the patch is actually running

This is the failure mode that will bite most people. The package manager says “upgraded,” the ticket gets closed, and the old PHP code keeps serving traffic because the processes that loaded it were never restarted. Linux happily keeps running a binary whose file has been replaced on disk.

1. Confirm the fix is in the package with the changelog check above, not the version string.
2. Restart everything that runs PHP: every FPM pool, Apache (for mod_php), LiteSpeed, and any long-running PHP workers such as queue consumers managed by Supervisor or systemd.
3. Check the restart time is after the upgrade. systemctl status php8.3-fpm shows when the service started. If that is earlier than your package install, it is still running old code.
4. Ask the system what still needs a restart. On Debian and Ubuntu, sudo needrestart -r l lists services still using outdated files, without restarting anything.
5. Check the right binary. php -v on the command line reports the CLI build, which can differ from the FPM build serving your site. On Debian and Ubuntu, php-fpm8.3 -v asks the FPM binary directly.

Mitigations if you cannot patch today, and what they don’t do

PHP is blunt about this: only an update is a complete fix. The settings below reduce exposure to the specific attack paths it has confirmed. They do not make a server safe, and some of them have real costs. Use them to buy hours, not weeks.

Set them where applications cannot override them: php_admin_value or php_admin_flag in the FPM pool, or the Apache configuration for mod_php. Do not rely on ini_set() or an FPM .user.ini file, because those apply too late in the request. Here is the FPM pool version on a Debian-style path (adjust the file and branch to yours):

; /etc/php/8.3/fpm/pool.d/www.conf
php_admin_value[memory_limit] = -1
php_admin_value[post_max_size] = 8M
php_admin_value[upload_max_filesize] = 6M
php_admin_flag[session.upload_progress.enabled] = off

And the Apache equivalent for mod_php, inside the virtual host or a Directory block:

php_admin_value memory_limit -1
php_admin_value post_max_size 8M
php_admin_value upload_max_filesize 6M
php_admin_flag session.upload_progress.enabled off

Restart every worker afterwards. Now the reasoning behind each line, because each one protects a different path:

• memory_limit = -1 targets CVE-2026-109618. The confirmed exploit needs a predictable per-request limit to hit, so removing the limit removes the trigger. The cost: one request can now consume all available memory. A genuine allocation failure can also still reach the vulnerable code.
• Low post_max_size shrinks the request bodies an attacker can use. PHP suggests 8M where uploads are not needed and 32M where they are, with upload_max_filesize kept below it.
• session.upload_progress.enabled = off targets CVE-2026-109619. It blocks the confirmed session paths but not other variants. It does nothing at all for CVE-2026-109618.
• On PHP 8.5 only, fatal_error_backtraces = off blocks the confirmed fatal-backtrace path for CVE-2026-109619, and nothing else.

Because removing the memory limit opens a denial-of-service risk, PHP recommends a container or cgroup memory limit as the availability boundary, ideally one that kills only the affected worker. It specifically warns against RLIMIT_AS or RLIMIT_DATA, since those make allocations fail at the limit, which is exactly the condition the bug needs. Enforcing request-size, field-count and multipart-part limits at your reverse proxy helps too.

A web application firewall is a reasonable stopgap, not a fix. Cloudflare deployed rules before the patch shipped and reported attackers changing payloads to get around them. If your site already sits behind Cloudflare, those rules help. They do not replace the update.

The third issue: CVE-2026-115560

Two days after the release, PHP disclosed one more possible code-execution path. It needs an unusual setup: the application sets session.serialize_handler to php_binary or php_serialize in configuration, then switches it to php at runtime with ini_set() before calling session_start(), with session upload progress still enabled. The fixed releases above do not cover it. PHP’s advice is to keep session.upload_progress.enabled=0 until the next regular bugfix release. If you already set it for CVE-2026-109619, leave it in place and you are done.

After patching: assume someone tried

Amazon reported a honeypot hit against these flaws on 17 October, four days before disclosure, and exploitation attempts climbed fast once details were public. Most reports so far describe scanning and failed attempts rather than confirmed break-ins, but “nobody has reported it” is not evidence about your server. If you ran an unpatched, internet-facing PHP site during that window, do three things.

• Rotate secrets a PHP worker could have held in memory: database passwords, API keys, payment provider keys, SMTP credentials and, for WordPress, the salts in wp-config.php (which logs every user out, so warn people first).
• Read your logs for the patterns researchers flagged: unusual multipart requests followed by memory-limit errors or bursts of HTTP 500 responses, oddly large PHPSESSID cookies in responses, and PHP workers restarting with no explanation.
• Look for anything new and executable in your web roots and upload directories, especially PHP files that no deploy put there.

Troubleshooting

“php -v still shows the old version.” Either it is a distro backport (check the changelog for the CVE), or you are looking at the CLI binary while a different build serves the site. Check the FPM binary directly and the container, not the host.

“The site throws 500 errors after the update.” The fix changes how PHP handles running out of memory. Crossing memory_limit now raises a soft error at a safe point, and a second, higher threshold ends the process outright, without running shutdown handlers. A plugin or import job that used to squeak by can now fail differently. Check the error log, then review memory_limit and post_max_size against what the application genuinely needs.

“Cleanup or logging code stopped running on big jobs.” Same cause. If anything important lives in a shutdown function, it will not run when the hard limit is hit. Fix the memory usage rather than raising limits blindly.

“The package manager says there is nothing to update.” You are probably on a repository that has not shipped the fix, an end-of-life distro release, or an end-of-life PHP branch with no vendor backport. That is a migration problem, not a patching problem.

“I set the mitigation but the old value still shows.” Look for overrides in other pool files, per-directory configuration or application code, and make sure you used the admin variants and restarted the workers.

Common mistakes

• Updating WordPress, plugins or the framework and calling it done. None of that touches the PHP runtime.
• Trusting the version number in either direction, and declaring a backported package “vulnerable” or a stale process “patched.”
• Upgrading the package and never restarting FPM, Apache or long-running workers.
• Treating session.upload_progress.enabled=0 as the fix. It was circulated as a workaround before the release and only covers part of one CVE.
• Setting mitigations with ini_set(), .htaccess overrides that apps can change, or .user.ini, which take effect too late.
• Leaving memory_limit=-1 in place long after patching, with no cgroup boundary behind it.
• Forgetting the boxes nobody owns: the staging server, the old client site on PHP 7.4, the forgotten container in a side project.

Best practices

• Keep an inventory of every PHP install, its source (distro, panel, container, source build) and its branch. This week shows why.
• Verify fixes by CVE in the package changelog, and verify the running process by start time.
• Patch mod_php servers first, then public FPM sites, then internal tools.
• Once patched, restore a sane memory_limit, but keep session upload progress off until PHP’s next regular release covers CVE-2026-115560.
• Move servers off mod_php to PHP-FPM where you can. It would not have saved you here, but it narrowed the easiest path.
• Stay on a supported PHP branch, or pay for a vendor that backports. An unsupported branch turns every emergency into a migration.

FAQ

What is the PHP2Shell vulnerability?

It is the nickname for two memory-safety flaws in the PHP engine, CVE-2026-109618 and CVE-2026-109619, both rated 9.2 on CVSS v4.0. They can allow unauthenticated remote code execution on servers that run PHP for web requests.

Which PHP versions fix PHP2Shell?

PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35 from php.net. Distribution and vendor packages may carry the fix under different, older-looking version numbers, so check the package changelog for the CVE.

Is my WordPress site affected?

If it runs on an unpatched PHP build, yes, regardless of how up to date WordPress and its plugins are. The flaw is in PHP itself, so the fix comes from your server, host or container image.

Is disabling session.upload_progress enough?

No. It blocks the confirmed session paths for one of the two bugs and covers the narrow third issue, CVE-2026-115560. It does nothing for CVE-2026-109618. Only an update is a complete fix.

I’m on PHP 7.4 or 8.1. What now?

php.net has no release for you, but PHP published source backports for 7.4, 8.0 and 8.1, and several vendors and distributions shipped backported packages. Use one of those immediately, then plan the upgrade to a supported branch.

Do I need to restart anything after updating PHP?

Yes. Restart every PHP-FPM pool, Apache if you use mod_php, LiteSpeed, and long-running PHP workers. Until they restart, they keep running the vulnerable code already loaded in memory.

Should I rotate passwords and keys?

If an internet-facing server ran an unpatched build after details went public, it is a sensible precaution. Researchers showed worker memory could be read remotely, and that memory can hold credentials from other requests.

Conclusion

The PHP2Shell vulnerability lives below your application, so the only real fix is a patched PHP build that is actually running. Everything else in this post is either a way to get there or a way to buy time. If you remember one thing, make it this: verify the fix by CVE in the package, and verify the process by its restart time. The version string will mislead you in both directions.

Need help patching PHP2Shell across your servers?

If you have more PHP servers than certainty, I can help you close this out properly:

• Inventory every PHP install across your VPS, cPanel, DirectAdmin and container hosts, with branch and package source.
• Patch each one through the right channel and prove the fix by CVE and process restart time.
• Apply and later roll back the interim mitigations safely, with a memory boundary in place.
• Rebuild and redeploy PHP container images, including pinned-digest pipelines.
• Review logs for the published indicators and plan secret rotation without breaking production.
• Plan the move off end-of-life PHP branches and off mod_php.

Send me the output of php -v, your package changelog check, and how PHP runs on the server (FPM, mod_php or LiteSpeed), and I’ll tell you where you stand.

The post PHP2Shell Vulnerability: What Happened, How to Patch, and How to Prove It Worked appeared first on John Nessime.

Then you open the ModSecurity audit log and find eleven rule matches on a single request, none of them obviously the culprit.

Dealing with ModSecurity false positives on WordPress is not hard once you understand how the OWASP Core Rule Set decides to block. It is very hard if you go looking for “the rule that blocked me,” because under the default configuration that rule does not exist. This post covers reading the log, choosing the right exclusion mechanism, the WordPress failure families that generate most of the noise, and the tuning knobs that quietly gut your coverage.

The invisible part: no single rule blocked the request

The Core Rule Set runs in anomaly scoring mode by default. Individual detection rules do not deny anything. They match, they log, and they add to a running score with setvar. At the end of the request phase, a separate blocking evaluation rule compares the accumulated score against a threshold and returns the 403.

Scores come from rule severity. A CRITICAL match adds 5, ERROR adds 4, WARNING adds 3, NOTICE adds 2. The inbound threshold ships at 5 and the outbound threshold at 4. So one critical match is enough on its own, and so are two notices plus a warning.

That is the failure mode nobody warns you about. The log line that produced your 403 lives in REQUEST-949-BLOCKING-EVALUATION.conf and it says the inbound anomaly score was exceeded. It does not name what matched. Take that ID and disable it and you have not fixed a false positive, you have turned off blocking for the whole site while leaving every detection rule running and logging. The site will look fixed. It is not protected.

First rule of tuning, then: the blocking rule is a symptom. Read backwards from it.

Read the log before you touch a rule

Every alert carries a unique_id. That value ties the error log line to the full audit log record for the same transaction, and it is the only reliable way to see every rule that fired on one request instead of a scattering of lines from different visitors.

# Paths vary by build. Check SecAuditLog in modsecurity.conf for yours.
tail -n 200 /var/log/httpd/error_log | grep -i modsecurity

# Then pull the whole transaction using the unique_id from that output
grep -F 'Zm9vYmFyCg' /var/log/httpd/modsec_audit.log

From each match, four fields matter and the rest is noise:

• The rule ID, as [id "941320"]. This is what you exclude.
• The matched variable, at the end of the pattern-match message as something like at ARGS:content. This is what you exclude it from.
• The URI, which tells you how narrowly you can scope the exclusion.
• The paranoia level tag, as [tag "paranoia-level/2"]. If every alert is tagged level 2 or higher, the real fix may be a paranoia level change rather than an exclusion.

One warning most guides skip: audit logs of false positives contain the payload that triggered them. If a user’s password happens to look like a shell command, it lands in your log in plaintext. Treat these files as sensitive, rotate them, and redact before pasting anything into a public bug report.

Pick the right exclusion mechanism

ModSecurity gives you four ways to write an exclusion, differing on two axes: applied once at startup or evaluated per request, and removing the whole rule or just one variable from it.

• SecRuleRemoveById and SecRuleRemoveByTag delete a rule or a whole tagged category at configure time. Blunt, cheap, global.
• SecRuleUpdateTargetById and SecRuleUpdateTargetByTag strip one variable from a rule at configure time, leaving the rule live for everything else.
• ctl:ruleRemoveById and ctl:ruleRemoveByTag do the same removal, but only for transactions matching a condition you write.
• ctl:ruleRemoveTargetById and ctl:ruleRemoveTargetByTag strip one variable, conditionally. This is the surgical option and the one to reach for first.

Runtime exclusions cost a little, since each is an extra rule evaluated on every request. That overhead is small next to the regex work the rule set already does, and the precision is worth it.

Placement decides whether an exclusion works at all

This is where most people lose an afternoon. The two types go in opposite places, for mechanical reasons rather than arbitrary ones.

• Configure-time exclusions go after the CRS include. They remove rules, and a rule has to exist before you can remove it.
• Runtime exclusions go before the CRS include. They change how rules behave for this transaction, and changing a rule after it has already run does nothing.

The Core Rule Set ships two files for exactly this, both with an .example suffix you drop before they load. REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf holds runtime exclusions, and RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf holds configure-time ones. Use them. Local files that survive a rule set update are the entire point.

Five failure families behind most ModSecurity false positives

Almost everything you will hit on WordPress falls into one of these. Recognizing the family tells you the shape of the fix before you read a single log line.

1. Post content that legitimately contains markup

WordPress lets editors put HTML in posts. It maintains its own allowed-tag list, sanitizes server side, and that mechanism has been reviewed hard over the years. The Core Rule Set knows none of this. It sees an <h1> or an inline style attribute in a POST parameter and matches an XSS rule.

This produces the “it saved yesterday” ticket, because it is content-dependent. Plain paragraphs pass. A pasted table with attributes does not. The fix is a target exclusion scoped to the save endpoint and the specific parameter, never a global rule removal:

# Runtime exclusion. Goes in REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf
# Replace 941320 and the parameter with what your audit log actually shows.
SecRule REQUEST_URI "@beginsWith /wp-admin/post.php" 
    "id:1000,
    phase:1,
    pass,
    t:none,
    nolog,
    ctl:ruleRemoveTargetById=941320;ARGS:content"

Pick your own IDs from a range CRS will never use. Reusing an existing ID is a startup error rather than a silent one, which is the good kind of mistake.

2. REST and admin-ajax calls that fail with no visible error

This family is nastier, because there is no error page. The browser hits /wp-json/ or /wp-admin/admin-ajax.php, gets a 403, and the JavaScript either swallows it or shows a generic “updating failed” toast. Autosave stops. An editor panel stays empty. A form plugin silently drops submissions. Users report this as flakiness rather than a security block, so it sits unfixed for months. When someone says the editor is “just slow sometimes,” open the browser network tab and look for 403s before going anywhere near PHP.

Scoping differs here. REST routes are structured, so use @rx against REQUEST_FILENAME to hit a route family rather than one path. For admin-ajax.php, the useful discriminator is usually the action parameter, since that one endpoint multiplexes dozens of unrelated plugin handlers.

3. Media uploads that break at a size boundary

Multipart uploads have two independent failure paths that look identical from the browser. The first is a genuine rule match on the multipart structure, or on binary content containing a byte sequence some rule dislikes. That shows up with a rule ID and you tune it normally.

The second is an engine limit, not a rule. SecRequestBodyLimit caps the total body ModSecurity will buffer, SecRequestBodyNoFilesLimit caps the non-file portion, and SecRequestBodyLimitAction decides whether an oversized body is rejected outright or inspected partially. If uploads fail above a consistent size and no CRS rule ID appears anywhere in the log, stop tuning rules. That is a limit, and the fix is in engine config.

Raising them costs memory per concurrent request, and partial inspection leaves part of the body unscanned. Decide which trade you are making rather than bumping the number until the ticket closes.

4. Cookies with dynamic names

WordPress authentication cookies carry a hash in the name, so a literal exclusion is impossible. Configure-time directives accept a regular expression in the target if you wrap it in forward slashes:

# Configure-time exclusion. Goes in RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf
SecRuleUpdateTargetById 942440 "!REQUEST_COOKIES:/^wordpress_logged_in_.*/"

The catch is significant: the ctl action supports no regular expressions at all. If you need a pattern match on a variable name, you are forced into the configure-time form, which applies site-wide rather than to one path. That single constraint drives more design decisions than people expect.

5. Chained rules that refuse to be tuned

Some rules are written as a chain of conditions. Target exclusions only apply to the first rule in the chain, so if the variable you want to strip is referenced further down, nothing you write has any effect and you will burn an hour convincing yourself the config file is not loading.

When a target exclusion visibly does nothing on a rule you can watch matching, check whether it is chained. If it is, your only option is removing the whole rule under a narrow condition. Accept the wider blast radius, or change the request instead.

Install the WordPress plugin before writing anything custom

The Core Rule Set maintains a WordPress rule exclusions package. In CRS 3 it was built in and switched on with a variable in crs-setup.conf. In CRS 4 it moved out into a separate plugin published in the CRS plugin registry, and it is not installed by default.

That catches people upgrading. Exclusions that used to work vanish, WordPress starts throwing 403s again, and the setup variable they remember is gone from the config file. The reasoning behind the split is sound: fewer rules loaded means less code that can carry a bug, and one of the old exclusion packages did carry a bypass.

Plugins load through wildcard includes in a specific order. Get the order wrong and the plugin silently does nothing:

Include /path/to/crs/crs-setup.conf

Include /path/to/crs/plugins/*-config.conf
Include /path/to/crs/plugins/*-before.conf

Include /path/to/crs/rules/*.conf

Include /path/to/crs/plugins/*-after.conf

Two limits before you lean on it. The plugin covers vanilla WordPress only, so false positives from third-party plugins, page builders and form plugins are still yours to tune. And the comment field is deliberately left out of the exclusions, because comment forms have a long history as a real attack surface. If your comment form throws false positives, that is a decision the maintainers made on purpose, and overriding it is a genuine risk trade rather than a gap in their package.

On a server hosting more than one application, scope the plugin to the WordPress vhost using WebAppID or a Host header condition. WordPress exclusions applied to your unrelated API are a hole you did not mean to open.

The knobs that look like a fix and are not

When exclusions get tedious, two shortcuts start looking attractive. Both cost more than they appear to.

Raising the anomaly threshold. Going from 5 to 20 makes most false positives vanish, and it also means any attack category that scores low can never trigger a block. Detections that only ever produce a single critical match become unblockable. The CRS documentation is direct here: keep the threshold at the default and change the paranoia level instead. Raising it is a temporary tuning aid, not a configuration.

Dropping the paranoia level. Legitimate, but often applied backwards. Level 1 is designed to produce few false positives. Levels 2 and up add progressively more aggressive rules and progressively more tuning work. If your alerts are all tagged level 2, dropping to level 1 is the correct fix and it is honest about what you gave up.

The better tool for moving up is the split between blocking and detection paranoia levels. Set tx.blocking_paranoia_level to what you actually enforce and tx.detection_paranoia_level higher. Rules from the higher level execute and log, but only rules up to the blocking level count toward the score. You get to see what level 2 would have caught, in production, on real traffic, without anyone getting blocked. Then you tune, then you promote. The performance cost lands as soon as you raise the detection level, though, since those rules run either way.

Two smaller settings help here. tx.sampling_percentage sends only a fraction of requests into the rule set, a gentler introduction for a busy site than blocking everything on day one. And tx.reporting_level can log requests that would have been blocked at a higher paranoia level, which is exactly the signal you want while climbing.

Troubleshooting

The exclusion has no effect. Check placement first. A runtime exclusion loaded after the rules, or a configure-time exclusion loaded before them, fails silently. Then check for a chained rule. Then confirm the file is being included at all by breaking it deliberately with a syntax error and seeing whether the server refuses to start.

It works on one site and not another. Different vhosts load different config. On control panels, per-domain overrides usually sit in a separate file from the global rule set.

It worked and then stopped. Something updated the rule set. If you edited CRS rule files directly, your edits are gone, which is the whole argument against editing them.

Nothing appears in the log at all. Confirm the rule engine is on, and that auditing is enabled with enough log parts to include the request body. An engine in detection only mode logs without blocking, which is confusing if you did not know you were in it.

You are on shared hosting and cannot edit any of this. Some hosts expose a per-domain exclusion box, some let you disable rule IDs, some give you nothing. That is a real ceiling. Providers that hand you root, such as InterServer or Contabo, let you own the rule set and its update cadence, which on a site with genuine tuning needs beats the panel convenience you trade away.

Common mistakes

• Disabling the blocking evaluation rule because it showed up in the log. It reports the score, it detects nothing, and removing it disables blocking site-wide.
• Editing CRS rule files directly. Every update reverts you, and you now maintain a fork by hand.
• Excluding by message rather than by ID. Messages carry dynamic data and special characters, so the match is fragile.
• Removing an entire attack tag because two rules misfired. SecRuleRemoveByTag attack-sqli is defensible when nothing behind the WAF touches SQL. On WordPress it is not.
• Excluding /wp-admin/ wholesale, or stacking multiple rule set vendors. Admin is where a compromised session does the most damage, and overlapping rule sets make attribution guesswork.
• Leaving known false positives in place. Real attacks get lost in the noise, and the logs accumulate user data you never intended to collect.

Best practices

• Start in detection only. Collect real traffic, tune what you find, then enable blocking. Enforcing first is how you learn about broken checkout at two in the morning.
• Install the WordPress exclusions plugin before writing custom rules. Do not hand-write what upstream already maintains.
• Write the narrowest exclusion that resolves the alert: one rule, one variable, one path. Comment it with the rule ID, the endpoint, and why you judged it safe. Six months later that comment is the only thing between you and a blind revert.
• Keep exclusion files in version control and deploy them with the rest of your config. Test rule set updates on staging, since new signatures land regularly.
• Ship audit logs somewhere queryable. Grepping a growing text file stops scaling fast, and a platform like Grafana Cloud turns “which rule fires most” into a query instead of an afternoon.
• If an edge WAF such as Cloudflare sits in front of the origin, decide deliberately which layer owns which rule. Two WAFs scanning the same body doubles your false positive surface for no extra coverage.

Frequently asked questions

How do I find which ModSecurity rule blocked my WordPress request?

Take the unique_id from the 403 entry in the error log and grep the audit log for it. That gives you every rule that fired on the transaction. The blocking evaluation rule is what returned the 403, but the detection rules listed alongside it are what you tune.

Should I disable ModSecurity for WordPress instead of tuning it?

It is a defensible choice if the alternative is blocking mode with unresolved false positives, since a WAF everyone has learned to route around is worse than none. But detection only mode gives you the logs with none of the breakage, and that is almost always the better middle ground while you work through the tuning.

Is the WordPress rule exclusions plugin enough on its own?

For a stock install on a standard theme, usually. It covers vanilla WordPress and both block-based and classic themes. It does not cover third-party plugins, page builders, or theme frameworks, and those generate most of the false positives on a real site.

Why do false positives come back after a rule set update?

Either you edited rule files directly and the update overwrote them, or the update shipped new signatures your existing exclusions do not cover. The first is avoidable by keeping every change in the exclusion files. The second is the cost of running a maintained rule set, and it is why updates belong on staging first.

Does ModSecurity slow WordPress down?

It adds per-request work, mostly regular expression evaluation against request data, and the cost scales with paranoia level and body size. The settings that matter are the paranoia level, the request body limits, and whether response body inspection is on. Measure on your own traffic rather than trusting someone else’s benchmark.

Is ModSecurity still maintained?

The engine moved to the OWASP Foundation after Trustwave stepped back from commercial support, and it still receives fixes, though feature work is slower than it once was. The Core Rule Set is a separate project on its own cadence and is actively developed. Coraza is a compatible engine running the same rule language, worth evaluating for new deployments.

Wrapping up

The thing to hold on to about ModSecurity false positives on WordPress is that the rule which returned your 403 is almost never the rule you need to change. Anomaly scoring separates detection from blocking on purpose, and reading the log backwards from the blocking decision to the detection rules that fed it is the whole skill.

Everything else follows. Install the maintained WordPress exclusions instead of reinventing them. Write the narrowest exclusion you can, in the file that survives updates. Move the paranoia level, not the threshold.

Need help tuning your WAF?

Most of the WAF work I take on is untangling exactly this: a rule set that is technically installed and practically ignored, because nobody had time to work through the false positives. I can help with:

• Triaging your audit logs and mapping every recurring false positive to the rule, variable and endpoint that produced it
• Writing scoped exclusion sets for WooCommerce, form plugins, page builders and custom REST routes that upstream packages do not cover
• Moving a site off direct rule file edits and onto exclusion files that survive rule set updates
• Planning a CRS 3 to CRS 4 migration, including the exclusion package to plugin transition and the paranoia level variable renames
• Running a staged rollout from detection only to blocking, with the reporting and sampling settings that make it safe
• Shipping ModSecurity audit logs into a queryable log platform so rule tuning becomes a dashboard instead of a grep

If you have a blocked request you cannot explain, send me the audit log entry and your exclusion file and I will tell you what is firing and why.

The post Your WordPress 403 Isn’t the Rule You Think: Tuning ModSecurity False Positives appeared first on John Nessime.

The same output showed roughly 11 GB in the available column. Nothing was wrong, and nothing had been wrong for months. The only thing broken was the alert that fired.

Low free memory on Linux is one of the most reliably misread signals in server operations. It looks like an emergency, it triggers scaling decisions, and it is almost always noise. This post covers what free is really telling you, which four signals actually indicate memory trouble, the one situation where a low free number genuinely matters, and what to put in your alert rules instead.

What the free command is actually reporting

free does not measure anything. It parses /proc/meminfo and prints a handful of those fields in columns. Every number it shows comes straight from the kernel and is readable directly if you want to skip the formatting.

A typical reading on a busy server looks roughly like this:

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            15Gi       3.4Gi       231Mi       128Mi        12Gi        11Gi
Swap:          2.0Gi          0B       2.0Gi

Read that left to right and the story is boring. Applications hold a few gigabytes. The kernel filled almost everything else with page cache and buffers, because every file that gets read or written passes through memory on the way, and directory metadata lands in reclaimable slab. What is left over, genuinely untouched, is the free column.

That last part is the bit people miss. Free memory is not a reserve. It is waste. A page holding a cached copy of a file your application reads every few seconds is doing useful work. A page sitting empty is doing nothing, and the kernel converts it into cache the moment it has a reason to. A server that has been up for a week and still shows plenty of free memory is a server that is barely touching its disk.

If you want the raw source rather than the formatted view:

$ grep -E '^(MemTotal|MemFree|MemAvailable|Buffers|Cached|SReclaimable|SwapTotal|SwapFree|Dirty):' /proc/meminfo

Two options are worth knowing. free -w switches to wide mode and splits buffers and cache into separate columns, which helps when attributing memory to a specific I/O pattern. free -h -s 5 reprints every five seconds, a crude but effective trend view while you reproduce a problem.

Signal one: available memory, not free memory

The available column maps to MemAvailable in /proc/meminfo, and it is the kernel’s own estimate of how much memory a new workload could allocate without pushing the system into swap. It has been present since kernel 3.14, so every distribution you are likely to be running exposes it.

What matters is that it is not free plus cache. That naive sum is what people calculated before MemAvailable existed, and it overstates the truth. The kernel starts from free pages, subtracts the reserves it refuses to dip below, then adds back the portion of file-backed page cache and reclaimable slab it believes it could actually release. Both additions are capped, because some cache is pinned and some reclaimable slab objects are in use.

The kernel documentation is explicit that this is an estimate whose accuracy varies from system to system, so treat it as a strong hint rather than an exact figure. It is still far better than anything you can compute yourself, and it is the number your dashboards should be built on.

If you change one thing after reading this: point your memory alerts at available, not at free. Everything else in this post is refinement on top of that.

Signal two: swap activity, not swap usage

The second most common false alarm is swap. Someone sees a few hundred megabytes of swap in use and concludes the machine is thrashing. It usually isn’t.

Swap usage is a stock: pages moved to disk at some point, possibly during a backup weeks ago, and never needed since. Leaving them there is correct, and pulling them back to satisfy a dashboard would evict something useful. Swap activity is a flow, and that is the part that hurts, because a page fault serviced from disk is orders of magnitude slower than one serviced from RAM.

vmstat gives you the flow. The si and so columns are swap-in and swap-out, in blocks per second over the sampling interval:

$ vmstat 5 6

Discard the first row. vmstat reports averages since boot on its first line, which is almost never what you want. Read the rows after that. If si and so are both consistently non-zero, the machine is simultaneously reading pages back to serve current work and writing pages out to make room for new allocations. That is thrashing, and it is a real signal with no ambiguity attached.

A single non-zero sample is not that. Brief swap activity during a log rotation or a backup window is normal. Sustained activity across several consecutive samples is what to escalate.

Signal three: memory pressure, which is the one that catches things early

Pressure Stall Information is the signal I reach for first now, and it is badly underused. It has been in the kernel since 4.20 and it answers a question none of the other tools answer directly: how much time are my processes losing because memory is contended?

$ cat /proc/pressure/memory
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0

The some line is the share of wall-clock time in which at least one task was stalled waiting on memory. The full line is the share in which every non-idle task was stalled, meaning nothing useful happened at all. The averages cover ten seconds, one minute and five minutes, and total is a cumulative counter in microseconds that updates continuously, so it catches a two-second stall the running averages smooth away entirely.

Here is why this beats a percentage threshold. Utilization tells you how much of a resource is consumed; pressure tells you whether that consumption is costing you anything. A machine can sit at high utilization for months with zero pressure, which means the sizing is fine. It can also sit at moderate utilization while full climbs, which means reclaim is working hard and latency is already suffering with no threshold crossed.

With cgroup v2 mounted, the same data exists per cgroup in each directory’s memory.pressure file, so you can attribute pressure to a specific service rather than the whole machine. Prometheus node_exporter collects the system-level PSI counters, and if you already ship metrics to something like Grafana Cloud these graph well beside your latency panels.

Signal four: the OOM log, which is the only unambiguous one

When memory really does run out, the kernel writes a record. It does not do this quietly and it does not do it conditionally. If there is no OOM record, no process was killed for memory, full stop.

$ sudo journalctl -k --since "24 hours ago" | grep -i "oom"
$ sudo dmesg -T | grep -i "killed process"

The kill line names the victim and reports its memory breakdown at the moment of death: total virtual size, anonymous resident set, file-backed resident set, shared memory resident set, and the oom_score_adj that was in effect. The line above it, where a process “invoked oom-killer”, tells you which allocation triggered the whole event, and that process is frequently not the one that got killed.

That distinction is the most useful thing in the log. The kernel picks a victim by badness score, driven largely by how much memory a process holds relative to what is available to it, adjusted by oom_score_adj. A slow leak in a small helper can push the machine over the edge and get your database killed instead, because the database is simply the largest thing running. Read both lines before deciding who the culprit is.

If you have a service that should survive at almost any cost, systemd exposes OOMScoreAdjust= in the unit file, which persists across restarts in a way that writing to /proc/<pid>/oom_score_adj does not. Use it sparingly. Protecting a process that is itself leaking just moves the crash somewhere less convenient.

When low free memory on Linux really is the problem

Everything above assumes you are looking at a whole machine. Inside a container, the accounting changes, and this is the failure mode that is invisible until it isn’t.

The cgroup memory controller charges page cache to the cgroup that caused it. Clean, evictable, perfectly reclaimable cache still counts toward memory.current, and that is what gets compared against memory.max. A container reading a large file has just charged that file to its own budget.

$ cat /sys/fs/cgroup/memory.current
$ cat /sys/fs/cgroup/memory.max
$ grep -E '^(anon|file|slab_reclaimable) ' /sys/fs/cgroup/memory.stat

The split in memory.stat is what you actually want. anon is memory your process genuinely holds; file is page cache charged to you that the kernel can take back under pressure. A container at its limit on mostly file is in a very different position from one at its limit on mostly anon.

Two things follow from this that catch people out repeatedly:

• Reclaim is not instant. Under heavy pressure the reclaim path itself runs slowly, so a container can be OOM-killed while still holding clean page cache that was theoretically available.
• Running free inside a container lies to you. Unless something virtualizes /proc, you are reading the host’s /proc/meminfo, which knows nothing about your limit. The cgroup files are the only truth here.

cgroup v2 also has memory.high, a soft limit. Crossing it triggers reclaim and throttling rather than a kill, which turns a hard failure into a slow one. That is usually the better trade, and it is worth setting deliberately rather than leaving it at its default.

What to alert on

The alert on the ticket at the top of this post was watching the wrong number. Here is what a memory alert set looks like when it watches the right ones:

• Available memory below a percentage of total, sustained. Ten percent for ten minutes is a reasonable starting shape. A single thirty-second dip is noise and should never page anyone.
• Sustained swap-in and swap-out together. Rate of change on the swap counters, not the absolute amount of swap in use.
• Memory PSI rising. The some line moving off the floor is your early warning. The full line moving at all is your late one.
• Any OOM kill, ever. This one is binary and should always generate a ticket, even if the service recovered on its own.
• Per-container memory events. The cgroup memory.events file counts limit hits, and a rising count is a limit that needs revisiting.

Notice that free memory does not appear anywhere in that list. There is no threshold on the free column that produces a useful alert, because the number is a byproduct of the kernel’s caching behaviour rather than a measure of anything you care about.

Troubleshooting: which tool answers which question

When a memory question lands on you, the order you check things in matters more than the tools.

1. Did anything actually die? Check the kernel log for OOM records first. If there are none, this is a performance question and not a capacity failure, which changes everything about what you do next.
2. Is the machine stalling? Read /proc/pressure/memory. Zero pressure with low free memory means you are done. Go close the ticket.
3. Is it swapping right now? Run vmstat 5 for half a minute and watch si and so. Ignore the first row.
4. Who is holding the memory? ps aux --sort=-%mem | head -15 gets the shortlist. Resident set size double-counts shared pages, so summing the column overstates the total.
5. Is it the kernel rather than a process? If the process list does not account for the usage, check slabtop and SUnreclaim in /proc/meminfo. Unreclaimable slab growth is a different problem, usually a driver, filesystem or network workload creating enormous numbers of kernel objects.
6. Is it growing? A leak is a slope, not a level, and one reading tells you nothing. This is the whole argument for having the graph before you need it.

One more that saves time on crowded boxes: resident set size is the wrong measure of what a process really costs, because shared libraries are counted against every process that maps them. Proportional set size divides shared pages among the processes sharing them, and tools reporting PSS give a far more honest answer when you are deciding what to move off a busy VPS.

Common mistakes

• Dropping caches to make the number look better. Writing to /proc/sys/vm/drop_caches discards clean page cache and reclaimable slab. The kernel documentation says plainly that using it outside testing or debugging is not recommended, because everything dropped has to be re-read from disk. You trade a nicer-looking free output for an I/O spike and worse latency. It is a benchmarking tool, not an operations tool.
• Setting vm.swappiness to zero as a reflex. It does not disable swap and it does not make the machine faster. It makes the kernel strongly prefer evicting page cache over swapping anonymous pages, which on a database host means throwing away exactly the cache you wanted. Change it for a measured reason, not by default.
• Running with no swap at all. Swap is not just an overflow area. Without it the kernel loses the option of paging out genuinely cold anonymous memory, which means the only lever left under pressure is evicting cache or killing something. A modest swap file gives reclaim somewhere to go.
• Summing RSS across processes and comparing it to total RAM. Shared pages are counted once per process. The sum will exceed reality, sometimes dramatically.
• Resizing the instance before finding the slope. More RAM buys time against a leak. It does not fix one, and the bill arrives every month afterwards.

Best practices

• Graph available memory, swap rate and memory PSI on one panel. Together they read as a story; individually each one misleads.
• Keep the raw counters, not just derived percentages. A percentage of an instance size that has since changed cannot answer “was this happening last week”.
• Set container limits from observed anon usage plus headroom, not a round number someone picked, and put memory.high below memory.max so pressure shows up as slowness before it shows up as a kill.
• Alert on rate of change for anything that can leak. The steady climb is the signal; the level it reached is just when you happened to look.
• Right-size deliberately. On providers such as Contabo or InterServer, where RAM is a meaningful share of the monthly bill, the gap between a real requirement and a panicked upgrade is money you keep every month.
• Write the explanation down where your team will find it. Every organization rediscovers the free-versus-available confusion independently, usually at three in the morning.

Frequently asked questions

Why is free memory so low on my Linux server?

Because the kernel used it for page cache and buffers, which is the intended behaviour. Any file read or written passes through memory, and the kernel keeps those pages in case they are needed again. It reclaims them automatically the moment an application needs the space. Check the available column instead.

What is the difference between free and available in free -h?

Free is memory that is completely unused. Available is the kernel’s estimate of how much a new process could allocate without swapping: free memory plus the portion of cache and reclaimable slab it believes it could release. Available is almost always much larger, and it answers the question you are actually asking.

Should I clear the page cache to free up memory?

No. The kernel reclaims cache automatically under pressure, and more selectively than a blanket drop. Writing to drop_caches forces everything to be re-read from disk, and the kernel documentation advises against using it outside testing or debugging. Doing it on a schedule is a strong hint something else is misconfigured.

Is swap being used a bad sign?

Not by itself. Pages that were swapped out during a quiet period and never needed again are fine where they are. What matters is swap traffic: sustained non-zero si and so in vmstat at the same time means the machine is thrashing. Alert on the rate, not the amount.

How do I tell if a process was killed for running out of memory?

Search the kernel log. journalctl -k filtered for “oom”, or dmesg -T filtered for “killed process”, shows a record naming the process, its PID and its memory breakdown at the time. The kernel always logs this, so if there is no record the process died for some other reason.

Why does my container get OOM-killed when memory looks fine inside it?

Because free inside a container usually reads the host’s memory information, not the cgroup limit being enforced. Read memory.current, memory.max and memory.stat instead. Page cache charged to the container counts toward its limit, so a read-heavy workload can hit the ceiling without any process holding much anonymous memory.

What is a good memory alert threshold?

Available memory below roughly ten percent of total, sustained for ten minutes, is a defensible starting point to tune against your own workload. Pair it with memory PSI and an unconditional alert on any OOM kill. A threshold on the free column, at any value, will page you for nothing.

Conclusion

Low free memory on Linux is the expected steady state of a healthy, working server. The kernel fills memory with cache because empty memory earns nothing, and hands it back the instant an application asks. The free column is a byproduct of that behaviour, not a health metric, and alerting on it produces exactly the kind of urgent ticket that wastes an afternoon and ends in a change request nobody needed.

The one thing worth remembering: free tells you what is idle, available tells you what you can have, and pressure tells you whether any of it is costing you. Watch the second and third, log the OOM events, and read the cgroup files rather than free whenever a container is involved.

Need help reading your memory signals?

Most of the memory work I get asked for is not “add more RAM”. It is working out whether the number someone is worried about means anything. I can help with:

• Rebuilding memory alerts around available memory, swap rate and PSI so they stop paging on normal cache behaviour
• Diagnosing repeated OOM kills, including the cases where the process being killed is not the one causing the problem
• Setting container and pod memory limits from observed anonymous usage rather than guesses, with a soft limit so pressure surfaces before a kill does
• Separating a genuine leak from normal cache growth using retained metrics rather than a single reading
• Right-sizing servers and instances honestly, up or down, based on what the workload actually needs
• Building the dashboard that turns the next memory question into a five-minute conversation instead of an incident

If you have a free -h output, a vmstat capture or an OOM log line that is bothering you, send it over and I will tell you what it means.

The post Low Free Memory on Linux Isn’t a Problem. Here’s What Actually Is. appeared first on John Nessime.

They’re right. The database is fine. That’s exactly why this incident drags on for an hour instead of ten minutes.

Connection pool exhaustion on AWS is rarely a capacity problem. It’s a queueing problem, and the queue forms in one of three places: inside your application’s pool, at the database’s connection limit, or inside a proxy sitting between the two. Each layer has its own limit, its own error message and its own fix. Resizing the instance fixes none of them reliably, and sometimes makes one worse.

Below: how to tell which layer ran out, the SQL and metrics that prove it, and fixes that survive the next deploy. Examples use RDS and Aurora for PostgreSQL and MySQL, plus HikariCP, SQLAlchemy and node-postgres where their defaults matter.

First question: which layer ran out?

Before touching anything, figure out where requests are waiting. The error text usually tells you, if you read it literally instead of pattern-matching on “connection”.

• The application pool is empty. The error comes from your pool library, not the database. HikariCP says the connection is not available and the request timed out. SQLAlchemy raises a QueuePool limit ... reached timeout. The database never saw these requests at all.
• The database refused new connections. The error comes from the server. PostgreSQL returns a FATAL about too many clients or about the remaining connection slots being reserved (the exact wording varies by major version). MySQL returns error 1040, Too many connections.
• The proxy is holding the queue. With RDS Proxy in the path, clients connect fine but wait for a database connection to be borrowed, then time out after the proxy’s borrow timeout.

Then compare what the database is actually holding against its limit. On PostgreSQL:

SHOW max_connections;

SELECT count(*) AS client_connections
FROM pg_stat_activity
WHERE backend_type = 'client backend';

On MySQL, Max_used_connections is the high-water mark since the last restart, which is useful when the spike has already passed:

SHOW VARIABLES LIKE 'max_connections';
SHOW GLOBAL STATUS LIKE 'Threads_connected';
SHOW GLOBAL STATUS LIKE 'Max_used_connections';

If the app is throwing pool timeouts while the server is well under its limit, you’re in the first failure family. If the server is at the limit, it’s the second. That single comparison saves you from the most common wrong move, which is raising max_connections when the database was never the bottleneck.

Pool exhaustion with an idle database: the one that hides

This is the failure that fools people, because every database graph looks healthy while the application is effectively down.

A pool hands out a fixed number of connections per process. A request borrows one, and nobody else can use it until it’s returned. The pool doesn’t care whether the request is running SQL during that time. If your code opens a transaction, runs a query, then calls a payment API, uploads to S3 or sends an email before committing, the connection sits borrowed and idle for the whole round trip.

Now let that third-party API slow down. Each request holds its connection longer, the pool drains, new requests queue for a connection, and they time out. The database sees a handful of sessions doing nothing. CPU drops, because less SQL is running than usual. The dashboard gets greener as the outage gets worse.

So a slow dependency that has nothing to do with the database can exhaust your database pool, and database metrics will never show it.

Prove it with pg_stat_activity

Group sessions by state and by who owns them. The state column is the evidence; idle in transaction means a client opened a transaction and then went quiet without committing.

SELECT state,
       application_name,
       count(*) AS sessions,
       max(now() - xact_start) AS oldest_transaction
FROM pg_stat_activity
WHERE backend_type = 'client backend'
GROUP BY state, application_name
ORDER BY sessions DESC;

Then list the worst offenders. The query column shows the last statement the session ran, not what it’s doing now, which is exactly what you want here: it tells you where in the code the transaction was left open.

SELECT pid,
       application_name,
       client_addr,
       now() - xact_start AS transaction_age,
       left(query, 80) AS last_statement
FROM pg_stat_activity
WHERE state = 'idle in transaction'
ORDER BY xact_start;

On MySQL, open transactions live in information_schema.INNODB_TRX. The thread ID joins back to the process list so you can see the host and user:

SELECT trx_mysql_thread_id, trx_started, trx_state
FROM information_schema.INNODB_TRX
ORDER BY trx_started;

One thing makes all of this much faster: set application_name per service in the connection string. Without it, every session looks the same and you’re guessing from IP addresses that change with every task restart.

Leak or slow holder?

Two different bugs produce the same symptom. A leak is a code path that borrows a connection and never returns it, usually an exception path that skips the close. A slow holder returns the connection eventually but keeps it through non-database work.

The tell is recovery. Leaks never recover without a restart; the pool ratchets down one connection at a time until it’s empty. Slow holders recover when the slow dependency recovers. If restarting the service “fixes” the incident every few days, suspect a leak.

HikariCP has leakDetectionThreshold, which is off by default. Set it to a value longer than your slowest legitimate request, and it logs the stack trace of wherever a connection was borrowed and held past that threshold. That stack trace is usually the whole diagnosis. Other pools have their own hooks; if yours doesn’t, log borrow duration yourself around the checkout.

Fixes that actually hold

1. Stop holding connections across network calls. Do the external call first, then open the transaction, write, and commit. If you need the call’s result inside the transaction, restructure so the transaction is short. This is the real fix; everything else is a guardrail.
2. Put a server-side ceiling on idle transactions. PostgreSQL’s idle_in_transaction_session_timeout terminates a session that sits idle inside a transaction past the limit. The app will see an error, and that’s the point: a loud failure beats a silent pool drain. You can set it in the parameter group or scope it to the application role.
3. Fail fast on checkout. A web request that waits 30 seconds for a connection is worse than one that fails in two. Match the pool’s acquisition timeout to what the caller will tolerate.

ALTER ROLE app_user SET idle_in_transaction_session_timeout = '60s';

The role-level setting applies to new sessions for that role, so existing pooled connections keep the old value until they’re recycled. Pick a limit comfortably above your longest legitimate pause, or you’ll kill batch jobs that were behaving.

MySQL has no direct equivalent. wait_timeout closes idle sessions, but it’s a blunt setting that also affects pooled connections sitting legitimately idle, so tune it alongside your pool’s own idle and max-lifetime settings rather than instead of fixing the code.

Watch the node-postgres default specifically. Its pool connectionTimeoutMillis defaults to 0, which means no timeout: when the pool is empty, requests wait forever. You don’t get an error to alert on; you get hung requests and upstream timeouts that point everywhere except the pool.

Server slots run out while CPU idles

The second family is the one people expect: the database refuses connections. The surprise is that it happens at low CPU, and that the cause is arithmetic, not load.

Pool multiplication

Every pool size is per process. The number the database actually faces is the product of everything that opens a pool:

worst case = pool size
           x worker processes per container
           x containers at the autoscaler's maximum
           x 2 during a rolling deploy (old and new running together)
           + cron jobs, migrations, BI tools, admin sessions

Take an illustrative case: a pool of 10, four worker processes per ECS task, 30 tasks at the scaling maximum. That’s 1,200 potential connections before a deploy briefly doubles it. Nobody chose that number. It emerged from three settings owned by three different people, and it only shows up on the day traffic and a deploy coincide.

Use the autoscaler’s maximum, not today’s task count. Today’s count is what makes the math look safe.

The limit moves when you resize

On RDS and Aurora, the default max_connections in the parameter group is a formula based on the instance class’s memory, not a fixed number. So a rightsizing exercise that drops the instance a size to save money also quietly lowers the connection ceiling. The CPU graph said it was safe. The connection budget didn’t get a vote.

Don’t calculate the value by hand; the memory figure RDS uses excludes what its own processes consume. Run SHOW max_connections on the actual instance, and re-run it after every class change.

Why raising max_connections is the wrong reflex

You can override max_connections in a custom parameter group, and sometimes a modest bump is fine. But it treats the symptom. Each PostgreSQL connection is a separate backend process with its own memory, and sorts or hashes inside it can each use up to work_mem. Thousands of connections all trying to work at once contend for the same CPU cores, and throughput often drops as concurrency rises past what the hardware can run in parallel.

Also keep in mind that the last few slots are reserved for administrative roles. The application starts failing before the counter reaches the limit, and when it does, you may struggle to get your own session in to diagnose it. Keep headroom for humans.

Budget connections on purpose

1. Read the real max_connections from the instance and subtract a margin for admin sessions, monitoring agents and migrations.
2. List every consumer: each service, its worker count, its scaling maximum, and its batch jobs.
3. Divide the budget and set each pool’s maximum from that, including the deploy overlap.
4. Cap the autoscaler maximum at something the database can serve. An autoscaler that can add tasks faster than the database can accept connections just converts a traffic spike into a connection storm.
5. If the budget can’t cover the fleet, add a pooler in front instead of more slots.

Smaller pools feel scary, but a connection is only useful while SQL is running. A small pool with short transactions usually beats a large pool with long ones.

Zombie connections

Some slots belong to clients that are gone. A container killed without a clean shutdown, or a Lambda execution environment frozen between invocations, can leave server-side sessions open until something times them out. With Lambda, each concurrent environment holds its own connection, so cap concurrency and put a pooler in front; the serverless API work I do starts exactly there.

For long-lived services, set the pool’s idle timeout and max lifetime below any idle timeout on the network path, so connections are retired by the pool rather than cut silently in the middle.

When the proxy becomes the queue

RDS Proxy is the managed answer to pool multiplication. Many client connections share a smaller set of database connections, and the proxy hands a database connection to a client per transaction. When that works, the server-side limit stops being your problem.

It stops working in two ways. First, pinning: when a session changes state the proxy can’t safely share, such as setting a variable on PostgreSQL, the proxy ties that client to one database connection until the session ends. Enough pinned sessions and the proxy behaves like no proxy at all. Second, the borrow queue: when every database connection the proxy is allowed to open is busy, clients wait up to ConnectionBorrowTimeout (120 seconds by default) and then fail. Your application sees slow requests, not refused connections, and the database graphs look calm again.

The proxy’s own CloudWatch metrics tell you which it is. DatabaseConnectionsCurrentlySessionPinned shows pinning. DatabaseConnectionsCurrentlyBorrowed against DatabaseConnections shows how close the proxy is to its share of the database. And MaxConnectionsPercent on the target group caps that share as a percentage of max_connections, so a conservative setting there creates the borrow queue by itself.

The proxy is billed on the capacity of the database behind it (per vCPU-hour for provisioned instances, per ACU-hour for Aurora Serverless), so run your current rates through something like the AWS Cost Estimator before assuming it’s free relief. PgBouncer in transaction mode on a small EC2 instance is the self-managed alternative: cheaper to run, but it’s one more thing to patch, monitor and make highly available.

Metrics that warn you before connection pool exhaustion

Database CPU is the wrong alarm for this problem. By the time connection pressure shows up there, if it ever does, users already noticed. The signals that lead are on the client side:

• Threads waiting for a connection. HikariCP exposes pending threads, active and idle counts, and acquisition time through its Micrometer integration. Sustained pending threads above zero is the earliest reliable warning you get.
• Connection acquisition time. Healthy pools hand out connections almost instantly. A rising acquisition time is the queue forming, long before timeouts start.
• Server connections as a ratio of the limit. Alarm on CloudWatch DatabaseConnections relative to the instance’s real max_connections, not an absolute number that goes stale after a resize.
• Oldest open transaction. A scheduled query against pg_stat_activity exporting the age of the oldest xact_start catches slow holders and leaks early.

Where these land matters less than having them next to each other. Grafana with a Prometheus scrape of the application’s pool metrics works well, as do Datadog or New Relic APM, which can show pool wait time inside the request trace. Sentry is useful for grouping the pool-timeout exceptions so you see the first one, not the thousandth. The goal is one screen where app pool state and database connection counts share a time axis. If you’d rather not build that, it’s the kind of thing covered under AWS monitoring and alerting.

Troubleshooting by symptom

• Pool timeouts, database far below its limit, many idle-in-transaction sessions: slow holder or leak. Find the last statement in pg_stat_activity and trace the code path that follows it.
• Pool timeouts, few database sessions at all: the pool is too small for the worker count, or connections are borrowed and held without a transaction open. Check borrow duration, not the database.
• Refused connections only during deploys: the old and new versions overlap. Budget for the overlap or slow the rollout.
• Refused connections right after an instance class change: the formula-derived limit dropped. Check SHOW max_connections.
• Requests hang with no error at all: a pool with no acquisition timeout, such as node-postgres at its default. Set one.
• Everything was fixed by a restart, then came back days later: a leak. Turn on leak detection and wait for the stack trace.
• RDS Proxy in the path, slow requests, calm database: check pinned and borrowed connection metrics before anything else.

Common mistakes

• Scaling the instance up because “it’s the database”, when the database was idle the whole time.
• Raising max_connections to absorb pool multiplication, then losing throughput to contention.
• Making the pool bigger to fix pool timeouts caused by long transactions. It buys minutes and moves the failure to the server limit.
• Sizing pools from today’s task count instead of the autoscaler’s maximum.
• Adding RDS Proxy without checking pinning, and ending up with a paid queue instead of a pool.

Best practices

• Keep transactions short and never hold a connection across an external call.
• Write the connection budget down, with every consumer, and revisit it when scaling limits or instance classes change.
• Set an acquisition timeout that matches what callers will tolerate, and alert on pending threads, not on timeouts.
• Use idle_in_transaction_session_timeout on PostgreSQL as a safety net, scoped to application roles.

FAQ

What causes connection pool exhaustion when database CPU is low?

Usually connections held while no SQL runs: long transactions, external API calls inside a transaction, or leaked connections that were never returned. The pool is empty, but the database is doing very little.

Should I increase max_connections on RDS?

Only after confirming the server limit is actually the one being hit, and only modestly. If the cause is pool multiplication, budget the pools or add a pooler. More connections means more memory per backend and more contention, not more throughput.

How big should my connection pool be?

Smaller than most people set it. Start from the connection budget divided across every process at maximum scale, then load test. A pool large enough to cover every thread usually just moves the queue from the application to the database.

Does RDS Proxy fix pool exhaustion?

It fixes server-side slot exhaustion from many clients, as long as sessions aren’t pinned. It does nothing for an application pool drained by long transactions, because those transactions hold the proxy’s database connections just as firmly.

How do I find leaked connections in PostgreSQL?

Look for sessions in pg_stat_activity that stay idle or idle in transaction for far longer than any request should, grouped by application_name. Then turn on your pool’s leak detection to get the stack trace of where the connection was borrowed.

Does this happen on self-hosted databases too?

Exactly the same way. On PostgreSQL running on a VPS from providers like Hetzner, Contabo or InterServer, the pool math and idle transactions behave identically; the difference is that you run PgBouncer yourself instead of paying for RDS Proxy.

Conclusion

Connection pool exhaustion on AWS almost never means the database is too small. It means a queue formed somewhere, and the fix depends entirely on where. Read the error literally, compare the database’s connection count against its real limit, and within minutes you’ll know if you’re dealing with a drained application pool, a server limit reached by arithmetic, or a proxy holding the line.

If you remember one thing: a connection costs you for as long as it’s borrowed, not for as long as SQL runs. Shorten the borrow and most of these incidents stop happening.

Need help tracking down a connection problem on AWS?

This is work I do for teams running RDS, Aurora or self-managed PostgreSQL and MySQL. Typical engagements look like this:

• Working out which layer is exhausting during an incident, from pool logs, pg_stat_activity and CloudWatch.
• Building a written connection budget across services, workers, autoscaling limits and deploy overlap.
• Finding leaks and long-held connections in application code, with leak detection and transaction timeouts set sensibly.
• Evaluating RDS Proxy or PgBouncer for your workload, including pinning and cost.
• Dashboards and alerts that put application pool state and database connections on the same screen.
• Checking what an instance resize will do to your connection limit before you make it.

Send me the pool timeout from your logs and the output of the pg_stat_activity query above, and I’ll tell you which layer ran out.

The post Plenty of CPU, Zero Connections: Debugging Connection Pool Exhaustion on AWS appeared first on John Nessime.

That is the shape of most EKS upgrade incidents. The part everyone books a maintenance window for, the control plane bump, is the part AWS handles and almost never goes wrong. The damage happens in the data plane rollout, because of decisions made months earlier that nobody connected to upgrades.

This post covers the three failure families that actually break EKS version upgrades: removed APIs you did not know you were calling, add-on updates that silently discard your configuration, and pod disruption budgets that deadlock the drain. For each one: what it looks like, how to catch it beforehand, and what the escape hatch costs.

Why the control plane is the easy half

An EKS cluster has two versions that must stay compatible. AWS manages the control plane; you manage the data plane, which in practice means the kubelet version on your nodes.

Two constraints shape everything else. In-place upgrades move one minor version at a time, so being three versions behind means three sequential upgrades and three data plane rollouts. And the kubelet cannot run ahead of the API server, so once the control plane moves, the clock starts on getting nodes to follow.

Failure one: the API calls you did not know you were making

Deprecation and removal are different events, and the gap between them is where people get comfortable. An API version gets deprecated, everything keeps working, the warning scrolls past in CI, and several releases later the version is gone and whatever depended on it starts getting 404s.

Your own manifests are usually fine; someone fixed those when the warning appeared. What breaks is the third-party agent nobody owns: a metrics collector, an ingress controller, a Datadog or New Relic agent installed by a Helm chart pinned two years ago. These call the API on a watch loop, so they hit removed APIs immediately, and they tend to fail quietly rather than crash-loop.

Cluster insights, and what it is actually reading

aws eks list-insights --cluster-name my-cluster

aws eks describe-insight 
    --cluster-name my-cluster 
    --id a1b2c3d4-5678-90ab-cdef-EXAMPLE11111

EKS runs upgrade readiness checks and exposes them through the insights API. The deprecated API check reads your control plane audit logs on a thirty-day rolling window, scanned about once a day. That has a consequence people trip over constantly: after you fix a deprecated call, the insight stays in ERROR until the last matching log entry ages out. The work is done, the check still fails, and if you gate automation on insight status your pipeline is blocked for weeks on a solved problem. Force a fresh scan rather than waiting for the daily one:

aws eks start-insights-refresh --cluster-name my-cluster
aws eks describe-insights-refresh --cluster-name my-cluster

The refresh re-reads the window; it does not shorten it. Audit-log detection also only sees APIs that were actually called, so a monthly CronJob or a chart sitting unapplied in a repo never appears. Insights tells you what your cluster did, not what it could do.

Ask the API server directly

The API server keeps its own counter, and it names the release that removes each API. Fastest check available:

kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis

Read the removed_release label first: it tells you whether this is urgent for the next upgrade or a warning for the one after. Scrape the metric into Prometheus, Grafana Cloud or Amazon Managed Grafana and you get a standing signal instead of a scramble.

For the callers, go to the audit logs. With control plane audit logging on, CloudWatch Logs Insights gives you the identity behind each deprecated call:

fields @message
| filter `annotations.k8s.io/deprecated` = "true"

The user.username field is what you need. It usually points at a service account in a namespace you did not install, which is your cue to find the chart’s owner. Static scanners like kubent and pluto cover the other half, the manifests that have not run recently. Use both.

Failure two: add-ons and the conflict resolution you picked once

Managed add-ons cover what a cluster cannot run without: the VPC CNI, CoreDNS, kube-proxy, the EBS CSI driver. They are version-pinned separately from the cluster, and their compatible ranges move with the Kubernetes version. The failure here is rarely the version, though. It is the conflict resolution mode.

When you update an add-on, EKS wants to apply its own field values. If you edited those fields in-cluster, that is a conflict, resolved by what you pass:

• NONE leaves your value alone and the update may simply fail. Safe, loud, and it stops your upgrade.
• OVERWRITE reverts your customization to the EKS default. The update succeeds. Your custom Corefile, your CNI tuning, your tolerations, gone, with no error.
• PRESERVE keeps your value through the update.

OVERWRITE is the one that hurts, because it is also the one people reach for when an update fails and they want it to stop failing. It is the standard fix when migrating a self-managed add-on to a managed one, so it gets learned there and carried into the update path, where it means something different. A CoreDNS update with OVERWRITE will replace a Corefile carrying your stub zones and forwarders, and resolution for those zones stops cluster-wide while every health check stays green.

aws eks update-addon 
    --cluster-name my-cluster 
    --addon-name coredns 
    --addon-version <compatible-version> 
    --resolve-conflicts PRESERVE

One more trap: the same value means different things on create and on update. On create, PRESERVE behaves close to NONE, leaving an existing self-managed add-on alone rather than adopting your settings. Check the operation, not the word.

The durable fix is to stop editing add-on resources with kubectl at all. Put customizations in the add-on’s configuration values so EKS owns them and there is nothing left to conflict over. An afternoon of work, and this failure family goes away permanently.

Failure three: PDB deadlocks and the fifteen-minute drain clock

This is the one that stalls upgrades halfway, and the mechanics matter because the fix depends on where you are stuck. A managed node group update runs in four phases: setup, scale up, upgrade, scale down. On the default strategy, scale up launches replacements before anything is terminated, which is what preserves capacity. Then the upgrade phase does this, per node:

1. Pick a node to replace, up to the configured maxUnavailable.
2. Cordon it once a new node reaches Ready.
3. Drain the pods off it.
4. Wait sixty seconds after eviction, then ask the Auto Scaling group to terminate it.
5. Repeat until nothing is left on the old launch template version.

Step three has a fifteen minute budget. If pods have not left in that time and you did not pass the force flag, the whole update fails with PodEvictionFailure and you are left with a half-migrated node group.

The deadlock nobody sees coming

A PDB says how many pods must stay available during voluntary disruption, and eviction is voluntary disruption. So the arithmetic is simple and, once you see it, obvious: a deployment with two replicas and a budget requiring two available pods can never be evicted. Not slowly. Never. The drain retries until the clock runs out.

One replica with a one-pod minimum is the same deadlock in friendlier clothing, as is a percentage budget that rounds up to the full replica count. And if two PDBs select the same pod, the more restrictive one wins, producing deadlocks invisible in either manifest read alone. The pre-flight check is cheap, and I run it before every upgrade:

kubectl get pdb --all-namespaces

Read the ALLOWED DISRUPTIONS column, not the configured minimum. Any budget sitting at zero is a drain that will hang. Fix those before touching the node group, by raising replicas or loosening the budget so at least one disruption is permitted.

There is a second cause of PodEvictionFailure with nothing to do with PDBs. The upgrade taints nodes as unschedulable, so after eviction the node should be empty. Anything tolerating every taint keeps it non-empty, and the drain never satisfies its exit condition. If your budgets look fine and eviction still stalls, look at what tolerates NoSchedule unconditionally.

What the force flag actually costs

aws eks update-nodegroup-version 
    --cluster-name my-cluster 
    --nodegroup-name my-nodegroup 
    --force

Force does not negotiate with budgets. It deletes the pods and terminates the node regardless. For a stateless service behind a load balancer with several replicas, that is fine and the upgrade completes. For a quorum-based system it is a bad idea: if the budget in front of your etcd, Consul, Kafka or ZooKeeper pods is doing its job, forcing past it can take the cluster below quorum. The budget was the protection; force is the decision to ignore it.

My rule is that force is reasonable on a node group I know holds only stateless workloads, and never a first response on a mixed one. Reaching for it because you do not know what is on the node is the actual problem.

If you run Karpenter, the levers are different

Karpenter has no update phases. It detects drift when a NodePool or EC2NodeClass stops matching a node’s configuration, then replaces the drifted nodes. Disruption budgets control the pace and can be scheduled, which is how you build a maintenance window without booking one:

spec:
  disruption:
    budgets:
      - nodes: "0"
        reasons: ["Drifted"]
        schedule: "0 6 * * mon-fri"
        duration: 12h

Two things bite people. Schedules are evaluated in UTC, so a window that looks like business hours to you may not be. And any period not covered by a budget is unrestricted, so gaps are not “no disruption”, they are “unlimited disruption”.

The other lever, terminationGracePeriod on the NodePool, overrides everything. Once it elapses, remaining pods are force-deleted regardless of PDBs or the karpenter.sh/do-not-disrupt annotation. It is a deadlock breaker, not a safety setting, and if it is shorter than your budget schedule you can get forced terminations inside a window you thought was protected.

The rollback window changes the calculation

Upstream Kubernetes has no control plane rollback. That single fact is why so many teams built heavyweight approval processes around upgrades. EKS now supports reverting an in-place control plane upgrade to the previous minor version within a seven day window, preserving etcd data and running workloads. It covers in-place upgrades only, which makes sense: with blue/green the old cluster is still there and rollback is a DNS change.

What changes is the shape of the risk. An upgrade used to be a one-way door, so the rational response was exhaustive testing beforehand. Now the first week is the test, under real traffic, with a way back. That argues for upgrading more often on smaller jumps, and treating the week after as an active watch period rather than a closed ticket.

It is not a general undo. Once the window closes, or you upgrade again, the previous version is gone, and it does not roll back add-ons or application state. Blue/green still has a place when you are several versions behind, since sequential in-place hops multiply exposure. The cost is running two clusters during migration, worth modelling first in Cost Explorer with both tagged, or in a tool like Vantage or CloudZero if you already run one.

Troubleshooting a stalled or failed upgrade

When a node group update fails, the reason comes back as an error code. Start there rather than guessing:

aws eks describe-nodegroup 
    --cluster-name my-cluster 
    --nodegroup-name my-nodegroup

• PodEvictionFailure. A budget blocked the drain, or something tolerates all taints. Check ALLOWED DISRUPTIONS first, then tolerations.
• NodeCreationFailure in scale up. New nodes never reached Ready. Usually no capacity for your instance type in an Availability Zone, an EC2 instance limit in the account, or custom user data breaking the bootstrap.
• Slow bootstrap. A node has fifteen minutes to join, measured from when the Auto Scaling group size increases. Heavy user data, large images pulled at boot or a slow init path push past it.
• Pods stuck terminating with no budget in sight. Look for blocked I/O. A pod with an NFS mount whose server is unreachable will not terminate, and eviction pressure changes nothing.
• Insights still in ERROR after the fix. The thirty day audit window. Refresh, confirm another way, move on.

One structural note: if the Cluster Autoscaler scales the node group during the scale down phase, the upgrade workflow exits without restoring the group to its original size. Check your desired and max counts afterwards.

Common mistakes

• Treating insights as the only pre-flight check. It reads audit logs, so it cannot see code paths that have not run.
• Reaching for OVERWRITE to make a failing add-on update succeed. It succeeds by discarding your configuration.
• Upgrading add-ons and the control plane in one unbroken pipeline run, so you cannot tell which change broke things.
• Leaving maxUnavailable at its default of one on a large node group, then wondering why the rollout takes hours.
• Forgetting self-managed components. Cluster Autoscaler, the AWS Load Balancer Controller and metrics-server are versioned against Kubernetes too, and nothing reminds you.

How I sequence EKS version upgrades

1. Read the release notes for the target version. Not the summary post, the removals section.
2. Run all three deprecation checks: insights for what the cluster did, the API server metric for what it does now, a static scanner for what sits in the repo.
3. Audit PDBs and fix every zero. Highest-value ten minutes in the process.
4. Update add-ons with PRESERVE to versions compatible with both the current and target Kubernetes version, so components talking to the API server are ready before it moves.
5. Upgrade the control plane and stop. Watch error rates and API server latency long enough to mean something.
6. Roll the least critical node group. Confirm pods reschedule cleanly and nothing depended on a node label or an AMI-specific detail.
7. Roll the rest, confirm every kubelet matches the control plane, then treat the following week as the real test. That is what the rollback window is for.

FAQ

Can you do EKS version upgrades with zero downtime?

For stateless workloads with sensible replica counts, spread across zones, with budgets allowing at least one disruption, yes. The default node group strategy launches replacements before terminating anything, so capacity holds throughout. Stateful workloads can too, but only if the application handles rolling leadership changes and the budget permits exactly one disruption at a time.

Why does the update fail with PodEvictionFailure when my PDBs look fine?

Two likely causes. Multiple budgets may select the same pod, in which case the most restrictive applies and neither manifest reveals it alone. Or something on the node tolerates every taint, so the node never becomes empty and the drain never completes. Check ALLOWED DISRUPTIONS across all namespaces, then check tolerations.

Is the force update option safe?

Safe for stateless workloads with multiple replicas. Not safe for quorum-based systems, because it deletes pods regardless of the budget protecting quorum. Use it when you know what is on the node group and the answer is nothing that minds being killed.

Does Karpenter make upgrades easier than managed node groups?

More automatic, which is not the same thing. Drift detection replaces outdated nodes without you running a command, but you inherit a different set of controls to get wrong: budgets evaluated in UTC, uncovered schedule gaps that mean unlimited disruption, and a grace period that overrides PDBs entirely. Neither is easier. They fail differently.

The one thing worth remembering

EKS version upgrades do not fail at the moment you run the command. They fail on decisions already sitting in the cluster: a budget written for a two-replica deployment, an add-on customized with kubectl rather than through its configuration values, a chart pinned before an API was removed.

Which means the pre-flight checks are the upgrade. Ten minutes reading ALLOWED DISRUPTIONS and the deprecated API metric tells you more about how tomorrow goes than any amount of care during the rollout. The rollback window gives you a second chance on the control plane. It will not un-delete a pod that force-eviction pulled out of a quorum.

Need a second pair of eyes on your upgrade path?

Most of this work is unglamorous and specific. What I would actually do on a cluster facing a version bump:

• Audit every PDB against its replica count and flag the ones that will deadlock a drain.
• Trace deprecated API calls back to the chart, controller or service account making them, from audit logs rather than guesswork.
• Move add-on customizations into managed configuration values, so conflict resolution stops being a decision made under pressure.
• Write a repeatable upgrade runbook: pre-flight checks, sequencing, and the rollback path for your specific workloads.
• Alert on the deprecated API metric and node group update events, so the next version is a scheduled task rather than an incident.
• Review Karpenter disruption budgets and grace periods for the gaps that turn into unplanned rollouts.

If you are staring at a failed node group update right now, send me the output of kubectl get pdb --all-namespaces and the error code from describe-nodegroup. That is usually enough to say what is holding the drain.

The post EKS Version Upgrades Without a Maintenance Window: APIs, Add-Ons and PDB Deadlocks appeared first on John Nessime.

That gap is where the real cost lives, and it is almost never on anyone’s dashboard. The build duration graph looks fine. The team still says shipping is slow. Somebody proposes buying bigger runners, and six weeks later the graph has moved a little and the complaints are identical.

Most CI/CD pipeline bottlenecks are not the thing people go looking for. Compute is usually the smallest slice. The expensive parts are minutes spent waiting for a runner, a cache that reports a hit and delivers nothing, a stage blocking four jobs that do not depend on it, and a required check sitting at “Pending” until someone notices. This post is organized by those failure families rather than by tool. Examples use GitHub Actions and GitLab CI, but the shapes apply to Jenkins, Buildkite, CircleCI and Argo Workflows too.

First, split the number you are staring at

“The build takes eleven minutes” is not a measurement. It is three measurements added together and then thrown away:

• Queue time. From job creation to a runner picking it up. You pay this in wall clock, not compute.
• Compute time. The job actually running. The only part most dashboards show.
• Gate time. Waiting on a human, an approval, a shared environment, or a check that never fired.

Until you split those, every optimization is a guess. On GitHub Actions the jobs API hands you the split directly, because each job carries a creation timestamp and a start timestamp:

gh api repos/OWNER/REPO/actions/runs/RUN_ID/jobs 
  --jq '.jobs[] | {name, created_at, started_at, completed_at}'

Creation to start is queue. Start to completion is compute. Run that across your last fifty runs on the default branch and the answer is usually sitting there in plain sight. GitLab reports the same idea on its jobs API, with a queued duration alongside the run duration. If the two numbers are roughly equal, you have a capacity problem rather than a build problem, and faster CPUs will fix half of nothing.

Queue time: the minutes you pay for twice

Queue time has two causes and they need different fixes.

Not enough runners

On hosted runners you are bounded by the concurrency your plan allows. On self-hosted you are bounded by whatever fleet you provisioned, which for most small teams is a fixed number of always-on boxes that sit idle overnight and saturate at 4pm.

If you run your own fleet on Kubernetes, Actions Runner Controller is where most people land. It creates runner scale sets that scale on workflow demand and can drop to zero when the queue is empty, with ephemeral pods so every job starts clean. That last part matters more than the scaling: a persistent runner accumulates state, and state is where “works on my machine” comes back as a CI flake.

If Kubernetes is more machinery than you want, a couple of dedicated boxes from a provider like Contabo or InterServer running the runner agent under systemd gets you a long way, with hosted runners as overflow when the queue backs up. Managed runner services such as Depot and BuildJet sell faster hardware behind the same interface. Their honest pitch is that you stop maintaining a fleet. The honest counterpoint is another vendor in the path of every merge, and a bill that scales with commit volume.

Runners busy on work that is already obsolete

Check this one first because it is free. Push three commits to a branch in five minutes and by default you get three pipelines. The first two are answering a question nobody is asking anymore, and they are holding the runners the third one needs.

On GitHub Actions, a concurrency group keyed on the ref does it:

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

Put that at workflow level on PR builds. Do not put it on a deploy workflow with cancellation enabled, or you will eventually kill a release halfway through and spend an afternoon working out what actually shipped.

GitLab splits the same job across two settings that have to agree. The interruptible keyword marks a job as safe to kill and defaults to false. The workflow-level auto-cancel setting decides what happens on a new commit:

workflow:
  auto_cancel:
    on_new_commit: interruptible

test:
  interruptible: true
  script:
    - make test

deploy:
  interruptible: false
  script:
    - make deploy

The default mode is conservative, which means once any non-interruptible job has started, the whole pipeline stops being cancelable. Setting it to interruptible cancels only the jobs that opted in and leaves the rest running. That distinction is the difference between a policy that works and one that quietly does nothing, and it is the usual reason teams turn auto-cancel on and see no change.

Caches that look like they are working

My favorite family of CI/CD pipeline bottlenecks, because the failure is completely silent. The step is green. The log says the cache was restored. The install takes exactly as long as before.

Branch scoping

GitHub Actions caches are scoped to the branch that created them, plus the default branch as a fallback. A feature branch reads its own caches and the default branch’s caches. It cannot read a sibling branch’s cache.

The consequence catches a lot of people: if your caching step only runs on pull requests, the default branch never populates a cache, so every new branch starts cold forever. Make sure at least one workflow on the default branch writes it. Runs triggered from fork pull requests get read-only cache access, so those jobs restore but never save.

Keys that never hit, or hit and never update

Cache entries are immutable. Once a key exists, writing to it again does nothing. So a key like node-modules-v1 gets populated once, on some machine, in some state, and then freezes. A key that hashes the lockfile is correct but creates a brand new entry on every dependency change, which burns through storage.

The pattern that survives contact is an exact key plus a prefix fallback: hash the lockfile for the primary key, then use restore keys to fall back to the most recent entry sharing that prefix. You get a partial restore and a fast install instead of a cold one.

Then watch the ceiling. Repositories have a default cache storage limit, entries untouched for a week are removed, and once you hit the limit the least recently accessed entries get evicted. Push enough large caches and you get thrashing, where entries are created and evicted at roughly the rate you create them. The tell is a hit rate that looks random. Check the repository’s cache usage in Actions settings before assuming the config is wrong.

Docker layer cache that does not cross machines

On ephemeral runners the local BuildKit cache dies with the machine, so it has to be exported somewhere. Registry-backed cache is the one I reach for first:

docker buildx build 
  --cache-from type=registry,ref=registry.example.com/app:buildcache 
  --cache-to type=registry,ref=registry.example.com/app:buildcache,mode=max 
  --tag registry.example.com/app:${GIT_SHA} 
  --push .

Two details do the work. mode=max exports intermediate stage layers, not just the ones that landed in the final image. If the expensive work happens in a builder stage that gets discarded, the default mode caches none of it, which is exactly the case you were trying to fix. Inline cache, the kind that ships alongside the image, only supports the minimal mode, so it cannot help multi-stage builds either.

The second detail is a trap. BuildKit cache mounts, the RUN --mount=type=cache kind, are not part of the exported layer cache. They live on the builder. Restoring a registry cache does not restore them, so if the instruction runs again the package manager starts from an empty download cache:

# syntax=docker/dockerfile:1
FROM node:22-alpine
WORKDIR /app
COPY package*.json ./
RUN --mount=type=cache,target=/root/.npm npm ci

That line is excellent on a persistent builder and close to useless on a runner destroyed after every job. Know which one you have before you credit it with a speedup.

Pipelines shaped like a staircase

Stages are a scheduling constraint most pipelines inherit by accident. In a stage-based model nothing in the test stage starts until everything in the build stage finishes. Put a forty-second lint job in the same stage as a six-minute compile and your lint feedback arrives six minutes late for no reason.

GitLab’s fix is needs, which lets a job start as soon as its own dependencies finish and ignore stage order entirely:

lint:
  stage: test
  needs: []
  script:
    - make lint

unit:
  stage: test
  needs: ["build"]
  script:
    - make unit

needs: [] means start immediately, wait for nothing. Lint now reports in under a minute while the compile is still running.

GitHub Actions has the opposite default: jobs run in parallel unless you add needs. So the equivalent problem there is over-declared dependencies, usually added early to force a readable graph and never revisited. Go read your needs lines and ask which are real data dependencies and which are narrative ordering.

The trade-off is real. A wide graph surfaces every failure at once and costs more compute per run. A narrow graph fails fast and saves minutes, but developers make several round trips to find three problems. For PR pipelines, wide is usually right. For expensive nightly jobs, narrow is.

The test suite is not one bottleneck

“Tests are slow” gets treated as a single problem. It is usually three, and they want different fixes.

• The suite is serial. Split by shard across parallel jobs. The cost is per-job setup overhead paid N times, so past a point more shards makes things slower. Find your own break-even instead of assuming more is better.
• The suite is flaky. A retry costs more than one rerun. It costs attention, a context switch, and eventually the team’s belief that red means anything. Quarantine flaky tests into a non-blocking job with an owner and a deadline. Leaving them in the required path while everyone knows to just re-run is worse than deleting them.
• The suite tests the wrong layer. A test that boots a database and a message broker to assert a validation rule is a unit test wearing a costume. Slowest fix, largest payoff.

Before touching any of it, get per-test timing out of your test runner and sort descending. Most suites have a handful of tests that dominate, and they are rarely the ones people suspect.

The wait that nobody is measuring

Here is the one that costs days rather than minutes, and it appears in no CI metric because from CI’s point of view nothing is happening.

If a workflow is skipped because of path filtering, branch filtering, or a skip directive in the commit message, its checks stay pending. Not skipped. Pending. And if that check is required, the pull request cannot merge. Nothing fails, nothing alerts, the PR just sits there looking almost ready until someone investigates by hand.

This bites hardest in monorepos, which is exactly where path filters are most attractive. You add paths to skip the service test suite on documentation-only changes, you save real minutes, and you accidentally build a merge trap.

The distinction that matters: a job skipped by an if condition inside a workflow that did run reports success. A workflow that never ran at all reports pending. So either keep the workflow triggering on everything and move the filtering inside, or pair each filtered workflow with a companion using the same name and inverted path filters that does nothing and passes. GitHub documents that second pattern and it works, but it means two files that must stay in sync.

The other gate-time offenders are less subtle and just as expensive: one shared staging environment serializing every team, and approvals routed to a single person asleep in another time zone. Plot merge timestamps against last-commit timestamps for a month. The shape of that distribution tells you which problem you have.

Troubleshooting: finding your bottleneck in an afternoon

1. Pull the last fifty runs on your default branch and compute queue time and compute time separately. If queue is a meaningful share, fix capacity and cancellation before anything else.
2. Read your slowest run as a graph, not a list. Wide gaps between a dependency finishing and its dependent starting are either queue or bad ordering.
3. Compare install duration on a cache hit against a cache miss. If they are similar, the cache is decorative.
4. Check repository cache storage usage. Thrashing looks like an erratic hit rate with no config changes behind it.
5. Count retries. The rate of second attempts on the same commit is your flakiness tax, and it is usually higher than anyone guesses.
6. Measure merge latency, last commit to merge, across a month. The difference between that and pipeline duration is gate time, and that is where the days are.

Pipeline run data is worth shipping somewhere queryable. Most CI platforms emit webhooks on job completion; feeding those into Prometheus and Grafana, or a hosted platform like Grafana Cloud or Datadog if you would rather not run the storage, turns “the build feels slow lately” into a graph you can argue with.

Common mistakes

• Buying bigger runners before splitting queue time from compute time. The most common wasted spend in CI.
• Turning on auto-cancel without setting the per-job interruptible flags, so the setting is technically on and functionally off.
• Caching a directory the tool does not read from, then declaring victory because the step went green.
• Using the default Docker cache export mode on a multi-stage build and wondering why the builder stage rebuilds every time.
• Adding path filters to workflows that back required checks, creating pull requests that can never merge.
• Optimizing the pipeline that runs on merge instead of the PR pipeline developers actually wait on.

Practices that hold up

• Treat queue time as a first-class metric with its own threshold, not a footnote on build duration.
• Cancel superseded runs on branches, never on the default branch and never on deploys.
• Make the default branch populate every cache other branches depend on.
• Order jobs by how fast they fail, not by how the file reads top to bottom. Cheap checks first, always unblocked.
• Give every flaky test an owner and an expiry date, or take it out of the blocking path.
• Re-measure after every change and keep the number. Pipelines rot, and the bottleneck moves as soon as you fix the current one.

Frequently asked questions

How do I find the bottleneck in my CI/CD pipeline?

Split every job into queue time and compute time using the timestamps your platform’s API already exposes, then compare pipeline duration against real merge latency. Those two comparisons put you in one of three buckets: capacity, build work, or gates. Fixing the wrong bucket is why most optimization efforts show no result.

Are self-hosted runners faster than hosted ones?

Per job, often yes, because you pick the hardware and can keep caches warm. Across a working day, frequently no, because a fixed fleet queues under load while a hosted pool does not. The right answer depends on which number is hurting you, compute or queue, which is why measurement comes first. A hybrid setup, self-hosted by default with hosted overflow, avoids committing to either.

Why is my cache not speeding anything up?

Usual suspects in order of frequency: the cache is scoped to a branch that never populated it, the key is static so it froze at whatever went in first, the cached directory is not the one the tool reads, or entries are evicted before the next run. Compare install duration on a hit against a miss. Identical duration means the restore is doing nothing regardless of what the log says.

Should I parallelize tests or make them faster?

Parallelize first because it is cheap and reversible, then fix the slow tests, because parallelism has a floor. Every shard pays setup cost, so a suite split too many ways spends more time booting containers than running assertions. Sharding buys time to do the real work. It is not the real work.

Why does my pull request say pending forever?

Almost certainly a required status check attached to a workflow that was skipped by path filtering, branch filtering, or a commit message directive. A skipped workflow reports pending rather than skipped, so the merge requirement is never satisfied. Either drop the path filter and skip inside the workflow instead, or add a companion workflow with the same name that always passes.

Do bigger runners actually help?

They help work that is genuinely CPU or memory bound, such as compilation and container builds. They do nothing for queue time, network-bound dependency downloads, tests waiting on external services, or gate time. Before upgrading, confirm the slow job is saturating the resource you are about to buy more of.

The one thing worth remembering

Most CI/CD pipeline bottlenecks are waiting, not working. Queue time, cache misses that report as hits, jobs blocked behind stages they do not depend on, and checks that sit pending because a workflow was never triggered. None of them show up on a build duration graph, and all of them are cheap to fix once you can see them. Split the number before you spend anything; the measurement takes an afternoon and usually points at a config change rather than a purchase order.

Need someone to find where your pipeline is losing time?

This is a large part of what I do as a freelance DevOps engineer. Typical work looks like:

• Instrumenting your pipeline so queue time, compute time and merge latency become three separate numbers you can watch over time
• Auditing GitHub Actions and GitLab CI caching to find the restores that report success and deliver nothing
• Restructuring stage-based pipelines into dependency graphs so cheap checks report first
• Setting up autoscaling self-hosted runners, on Kubernetes with Actions Runner Controller or on plain VMs, with hosted overflow
• Fixing Docker build caching for multi-stage images on ephemeral runners, including registry-backed export
• Untangling required checks and path filters in monorepos so pull requests stop getting stuck

Send me a workflow file, a job timeline, or the output of your slowest run, and I will tell you where the minutes are going before we talk about anything else.

The post CI/CD Pipeline Bottlenecks: Where Your Build Minutes Actually Go appeared first on John Nessime.

That asymmetry, sending works and receiving doesn’t, is the single most useful clue in the whole problem. It narrows the search enormously. Yet most guides on Zoho Mail not receiving email open by telling you to check your SPF record, which has nothing whatsoever to do with inbound delivery. People lose a full day tuning an SPF string while their MX records quietly point at a data center that has never heard of their domain.

This guide walks the inbound path in the order a sending mail server actually walks it, groups the real causes by failure family, and gives you the exact commands to prove which one you’re looking at. Everything here applies whether your DNS lives at Cloudflare, Namecheap, GoDaddy, Route 53, or a cPanel box at InterServer or Contabo.

Sending and receiving are two separate systems

This is worth internalizing before you touch a single DNS record, because it eliminates most of the internet’s advice on this topic in one move.

When you send from Zoho, your client authenticates against Zoho’s outbound SMTP server with a username and password. Your domain’s DNS plays no part in that handshake. SPF, DKIM and DMARC records exist so that the recipient’s server can decide whether to trust mail claiming to be from you. They are judgements other people make about your outbound mail.

When someone sends to you, none of that matters. Their server looks up your MX records, connects to whatever hostname it finds, and hands the message over. Your SPF record is never consulted. Your DKIM key is never consulted.

So if outbound works and inbound doesn’t, the fault is in the MX lookup, the connection that follows it, or Zoho’s decision about what to do with the message once it arrives. Not SPF. There is one narrow exception, covered later, where Zoho’s own inbound spam checks reject mail because the sender’s SPF fails. That is a different record on a different domain.

The inbound path, in the order it actually happens

Six things happen between someone hitting send and a message appearing in your Zoho inbox:

1. The sending server splits the recipient address and takes the domain part.
2. It asks DNS for the MX records of that domain, following delegation from the root down to whichever nameservers the registry says are authoritative for you.
3. It sorts the results by preference number and takes the lowest one first.
4. It resolves that hostname to an address record.
5. It opens an SMTP connection and offers the message with MAIL FROM and RCPT TO.
6. The receiving server decides whether that recipient exists in its organization and where to file the message.

Steps two through four are pure DNS. Step five is network. Step six is Zoho configuration. Diagnosing this well is mostly a matter of finding out which step you’re stuck on instead of guessing.

Five DNS failure families behind Zoho Mail not receiving email

Nearly every inbound failure I’ve seen falls into one of these. They present almost identically from the user’s chair, which is why guessing goes badly.

1. You’re editing records at a provider that isn’t authoritative

This is the most common one and the most frustrating, because the records look perfect in the control panel you’re staring at. They’re just not the records the internet reads.

It happens whenever the domain is registered in one place and the nameservers point somewhere else. You buy at Namecheap, delegate to Cloudflare, then add MX records back in the Namecheap DNS panel out of habit. Or a previous developer moved DNS to a hosting account you no longer log into. The registrar panel accepts your edits and stores them faithfully. Nobody queries them.

Find out where authority actually sits:

# Which nameservers does the parent zone delegate to?
dig +short NS example.com

# What do those nameservers say about your MX, directly?
dig MX example.com @ns1.yourprovider.com +norecurse

# Watch the full delegation chain from the root
dig +trace MX example.com

The +trace output is the one to read carefully. It shows each referral step, so you can see exactly which nameserver set the world is being pointed at. If those names don’t match the provider where you’ve been editing, you’ve found your problem and the fix takes thirty seconds.

2. Correct-looking MX records for the wrong Zoho data center

This is the failure mode that stays invisible longest, and it’s the reason copying MX values off a random blog post is dangerous.

Zoho runs regional data centers, and your account lives in exactly one of them. The mail hostnames differ by region: the international service uses .com hostnames, and the European and Indian deployments use their own regional equivalents. Your domain’s mailboxes exist only in the data center where your organization was created.

Point your MX at the wrong region and everything looks healthy. The records resolve. The hostnames are genuine Zoho servers. A lookup tool will show green. But mail arrives at infrastructure that has no record of your domain, and senders get a rejection saying the address doesn’t exist, or nothing at all.

For the international service, Zoho’s documentation specifies three records, in this order of preference:

Priority 10   mx.zoho.com
Priority 20   mx2.zoho.com
Priority 50   mx3.zoho.com

Do not take that as gospel for your account. The authoritative source is the DNS Mapping section of your own Zoho Mail Admin Console, which shows the exact hostnames for the data center your organization sits in. Read them there, copy them from there. If you’re unsure which region you’re on, the domain in the URL when you log into the admin console tells you.

Verify what the world sees, not what your panel shows:

dig +short MX example.com

# Confirm the target hostname itself resolves
dig +short mx.zoho.com

3. Leftover MX records from whatever came before

MX preference is not a fallback list in the sense most people assume. A sending server tries the lowest number first, and if that server accepts the message, the transaction is over. It never touches the others.

So a stale record from your old host sitting at priority 0 or 5 doesn’t compete with Zoho. It wins outright, every time. The mail is delivered successfully, to a mailbox nobody checks, on a server that may still be quietly accepting for your domain. The sender sees no error. You see nothing at all.

This is also why partial migrations bite. Two providers both accepting mail for one domain means your messages split between them based on preference and retry timing, which looks like random message loss and is nearly impossible to reason about from the inside.

The rule is simple: after cutover, the only MX records in the zone should be Zoho’s. Delete everything else, including records for subdomains you forgot about. Check the full answer rather than a truncated view:

dig MX example.com +noall +answer

4. A CNAME sitting where the MX records live

Two related problems here, and both are structural rather than typos.

The first: an MX record must point to a hostname with an address record. Pointing an MX at a name that is itself a CNAME is forbidden by the DNS specifications, and Zoho documents this explicitly. Some resolvers cope, others don’t, and the ones that don’t will fail intermittently in ways that look like a network problem.

The second: a CNAME at the zone apex. Classic DNS forbids a CNAME coexisting with any other record at the same name, and MX records at the apex are exactly that. Providers like Cloudflare work around this with CNAME flattening, which synthesizes an address record and does keep your MX records functional. Providers with a naive implementation may shadow or drop them. If someone added an apex CNAME to point the bare domain at a hosting platform or a site builder, that’s worth checking early.

dig CNAME example.com +short
dig A example.com +short

One thing that is not a problem, despite frequent claims otherwise: Cloudflare’s orange-cloud proxy setting. Proxying applies to HTTP traffic on address records. It has no effect on MX records, and toggling it grey will not fix inbound mail.

5. DNSSEC left broken after a nameserver change

Less common, but spectacular when it happens, because it takes the entire domain down rather than just mail.

If DNSSEC was enabled and the DS record at the registry no longer matches the signing keys at your current DNS provider, validating resolvers refuse to return any answer at all. They return SERVFAIL. Non-validating resolvers still work, which is why the site loads fine for you and mail from major providers, who mostly validate, vanishes.

The test is a one-liner. +cd disables validation checking:

# Fails with SERVFAIL if validation is broken
dig MX example.com

# Same query, validation bypassed
dig +cd MX example.com

If the second command returns your records and the first doesn’t, you have a DNSSEC mismatch. The fix is at your registrar: either update the DS record to match your current provider’s keys, or remove DNSSEC entirely, confirm mail flows, and re-enable it properly afterwards.

When DNS is clean and mail still doesn’t arrive

If your MX records resolve correctly to the right data center and nothing else is in the zone, DNS has done its job. The remaining causes sit on either side of it.

Your own server is intercepting the mail

This catches people running a site on cPanel or DirectAdmin, and it’s genuinely counterintuitive. Control panels have an email routing setting per domain, typically offering local, remote, or automatic delivery. If it’s set to local mail exchanger, the mail server on that box delivers messages for your domain into its own accounts and never performs an MX lookup.

The effect is narrow but confusing: external senders reach Zoho normally, while anything sent from a script or a form on that server goes to a local mailbox instead. Contact form submissions disappear while ordinary email works. Set routing to remote mail exchanger on any host where the site lives but the mailboxes don’t.

The address doesn’t exist in your Zoho organization

Zoho will only accept mail for addresses it knows about. A user account, an alias on a user, or a group. If someone hands out an address that was never created, Zoho rejects it and the sender gets an address-not-found bounce that they may not think to forward to you.

Zoho supports a catch-all address for exactly this, and it’s worth setting one during a migration so misaddressed mail lands somewhere recoverable rather than bouncing into the void.

Zoho accepted it and quarantined it

Under Security and Compliance in the Admin Console, Zoho lets you decide what happens to mail that fails SPF, DKIM, DMARC or DNSBL checks. The available actions include delivering it, quarantining it, or rejecting it outright, with or without a bounce.

Set to reject-without-bounce, a sender whose own SPF is slightly wrong gets silently dropped, and neither party learns anything. This is the narrow SPF connection I mentioned earlier: it’s the sender’s SPF being judged, not yours. Check the incoming quarantine before concluding mail never arrived, and check the organization blocked list while you’re there, since a blocked domain or TLD produces exactly the same silence.

The domain lapsed or lost verification

If a domain expires, its DNS stops resolving and every associated record becomes unreachable, which also invalidates the domain verification Zoho performed. Zoho’s documentation notes that after renewal you have to reconfigure MX records as part of reverification. Renewing the domain alone doesn’t restore mail.

A troubleshooting sequence that ends in an answer

Work these in order. Each step either clears a layer or hands you the cause, so you never end up changing three things at once and losing track of which one worked.

1. Confirm delegation. Run dig +short NS example.com and check it matches the provider you’re editing. If not, stop here and fix that first.
2. Read the MX records from an authoritative nameserver. Use dig MX example.com @ns1.yourprovider.com so you’re seeing the source of truth rather than a cached copy.
3. Compare against the Admin Console. Open the DNS Mapping page in Zoho Mail and match hostnames character for character, including the regional suffix.
4. Confirm nothing else is in the answer. Any non-Zoho MX record is a suspect, especially one with a lower preference number.
5. Test resolution from a public resolver. Try dig MX example.com @1.1.1.1 and @8.8.8.8. Differences between them mean you’re mid-propagation and waiting is the fix.
6. Rule out DNSSEC. Compare dig MX example.com against dig +cd MX example.com.
7. Prove the SMTP conversation. Send a real test and watch what the server says.
8. Check quarantine and blocked lists in the Zoho Admin Console before concluding anything was lost.

For step seven, swaks is the cleanest tool. It speaks SMTP for you and prints the full exchange, so you see the actual response code rather than guessing from a bounce message:

# Stop after RCPT TO so no message is actually delivered
swaks --to you@example.com --server mx.zoho.com --quit-after RCPT

A 250 after RCPT TO means Zoho recognises the address and would accept the mail. A 550 means it doesn’t know that recipient, which points straight at a missing mailbox or the wrong data center. A connection timeout usually isn’t Zoho at all: many residential and cloud providers block outbound port 25 by default, so run this from a machine you know has egress on 25. If you only have a laptop on a consumer connection, testing through a VPN endpoint that permits SMTP, or from a small VPS, avoids chasing a phantom.

Common mistakes

• Fixing SPF to solve an inbound problem. Your SPF record has no role in receiving mail. If you can send but not receive, it is not SPF.
• Copying MX values from a tutorial. Regional deployments have different hostnames. Use your own Admin Console every time.
• Adding Zoho’s records without removing the old ones. A lower preference number beats Zoho outright and nothing warns you.
• Trusting the DNS panel over a query. A saved record and a published record are different things. Query it.
• Cutting MX over before creating the mailboxes. Every address that doesn’t exist yet bounces, and those messages are gone.
• Blaming propagation on day three. Propagation is bounded by the TTL that was in effect before the change. If it’s been longer than that, something is genuinely wrong.
• Trailing dot confusion. Some panels want mx.zoho.com. with the dot, others append the zone themselves and turn mx.zoho.com into an unresolvable name. Always confirm with a query afterwards.

Best practices

• Drop the TTL before you migrate. Lower the MX TTL to a few minutes a day ahead of cutover, do the change, then raise it back. This turns a rollback from an overnight wait into a coffee break.
• Create every mailbox, alias and group before touching MX. Zoho’s own guidance is to configure users first and change MX last.
• Set a catch-all during the transition. It converts “we lost an email” into “it’s in the catch-all mailbox”.
• Keep DNS in one place. One authoritative provider, one panel, no ambiguity about where records live.
• Monitor the MX record itself. An uptime or synthetic monitoring service that checks DNS answers, or a scheduled dig comparison in your own pipeline, catches an accidental zone edit before a client does.
• Enable DKIM and set DMARC after inbound is confirmed working. They matter for your outbound reputation. Doing them first just adds variables while you’re debugging.
• Keep a copy of the zone. Export it before major changes. Reconstructing a zone from memory during an outage is miserable.

Frequently asked questions

Why can I send from Zoho Mail but not receive?

Because sending authenticates directly against Zoho’s outbound servers and doesn’t consult your domain’s DNS, while receiving depends entirely on your MX records. Working outbound with broken inbound is a strong signal that the problem is an MX record, a nameserver delegation issue, or a leftover record from a previous provider.

How long should MX changes take to work?

Propagation is governed by the TTL that was set on the old record, not by any fixed schedule. If the previous TTL was one hour, resolvers holding a cached copy will refresh within an hour. If it was set to a day, expect a day. Nothing you do afterwards speeds up a cache that has already been populated.

Do I need all three Zoho MX records?

Mail will flow with only the primary, but the backups exist so that senders have somewhere to deliver if the primary is unreachable. Without them, sending servers queue and retry, which delays delivery and eventually bounces. Add the full set your Admin Console lists.

Does SPF affect whether I receive email?

Your own SPF record does not. It’s evaluated by servers receiving mail from you. The one indirect connection is that Zoho evaluates the sender’s SPF on incoming mail, and if your Admin Console is set to reject or quarantine on SPF failure, legitimate mail from badly configured senders can disappear.

Why do lookup tools say my MX records are fine when mail still fails?

Lookup tools verify that records exist and resolve. They can’t know which Zoho data center your organization lives in, whether the mailbox has been created, or whether a message is sitting in quarantine. A green result rules out one layer, not all of them.

Can I run Zoho Mail alongside another mail provider on one domain?

Not by simply listing both sets of MX records. That splits mail unpredictably. Zoho does offer routing arrangements designed for gradual migration, configured on the Zoho side rather than by stacking MX records in DNS. If you need a genuine split, use the supported mechanism.

My contact form emails vanish but normal email works. Why?

Almost always the control panel email routing setting on the server hosting the site. If it’s on local mail exchanger, mail generated on that machine is delivered into a local account rather than sent out to Zoho. Switch it to remote.

Conclusion

If you take one thing from this: when Zoho Mail is not receiving email but sending works, stop looking at authentication records. Inbound mail is decided by your MX records, and those records have to be published by the nameservers that are actually authoritative for your domain, pointing at the data center your Zoho organization lives in, with nothing else in the answer competing with them.

Query, don’t assume. Read the answer from the authoritative nameserver rather than the panel, compare it against your own Admin Console instead of a tutorial, then work down through SMTP and Zoho’s own quarantine. Almost every case resolves in that order, and the ones that don’t are usually a mailbox that was never created in the first place.

Need someone to untangle your mail DNS?

I work on DNS and mail delivery problems like this regularly, usually for people who have already tried three fixes and made things harder to reason about. Things I can help with:

• Diagnosing inbound mail failures end to end, from delegation through SMTP response codes to Zoho’s quarantine
• Planning and executing a mail migration to Zoho with a TTL schedule and a rollback path that actually works
• Auditing and cleaning up a zone that several people have edited over the years
• Setting up SPF, DKIM and DMARC properly once inbound is stable, including staged DMARC enforcement
• Fixing DNSSEC mismatches after a nameserver move without taking the domain down again
• Adding DNS and mail-flow monitoring so the next zone edit doesn’t go unnoticed for a week

Send me the output of dig +trace MX yourdomain.com and a rejection message from a sender, and I can usually tell you where it’s breaking before we’ve agreed on anything.

The post Zoho Mail Not Receiving Email? A DNS Troubleshooting Guide appeared first on John Nessime.

The rule was fine. The query was fine. The evaluation ran on schedule and the state transition happened exactly when it should have. What failed was everything after that, and none of it made a sound.

That is the failure mode worth designing around. Rule evaluation and notification delivery are two different systems, and a red rule only proves the first one works. Grafana alert notifications run against SMTP servers, Slack’s API and Telegram’s bot endpoint, all of which fail in their own quiet ways. A rejected login, a revoked webhook, a message over a length limit: each produces a log line and nothing else. The alert list still shows the rule firing, so everything looks healthy from the one place you’d think to check.

This post covers the practical setup for the three channels most small and mid-sized teams actually use: email, Slack and Telegram. It covers what breaks in each of them, how to route alerts so the right ones interrupt people and the rest don’t, how to provision the whole thing as files instead of clicking through the UI, and how to detect the case where delivery has stopped entirely.

Where a firing alert actually goes

Before touching any integration, it helps to know how many hops sit between “the rule fired” and “my phone buzzed”.

• The rule evaluates on its own schedule and produces alert instances, each carrying labels.
• Those instances go to an Alertmanager. Grafana ships its own and can also use an external Prometheus Alertmanager.
• The Alertmanager applies mute timings, groups instances, and walks the notification policy tree for a match.
• The matched policy names a contact point, which holds one or more integrations.
• Each integration makes its own outbound call to a third-party service.

Steps one and two are visible in the Grafana UI. Steps three through five are where things go wrong, and they are visible only in logs. That asymmetry is the whole problem.

One thing worth knowing early: a contact point can hold several integrations at once, so a single contact point named critical can fire email, Slack and Telegram together. Building one contact point per channel instead makes routing much harder than it needs to be.

Email: the channel everyone assumes works

Email is the one people configure first and verify least. The catch is that the email contact point holds only the recipient addresses. The actual SMTP credentials live in grafana.ini, so a perfectly valid contact point can sit there doing nothing because the server underneath was never configured.

[smtp]
enabled = true
host = smtp.example.com:587
user = alerts@example.com
# Wrap the password in triple quotes if it contains # or ;
password = """your-smtp-password"""
from_address = alerts@example.com
from_name = Grafana Alerts
skip_verify = false
# Default is OpportunisticStartTLS
startTLS_policy = MandatoryStartTLS

Several details in that block matter more than they look.

• The port is part of host. There is no separate port setting. Port 587 with STARTTLS is the normal choice; 465 is implicit TLS and behaves differently, which is usually why one fails cleanly and the other hangs.
• startTLS_policy decides how hard Grafana insists on encryption. The default is opportunistic: it upgrades the connection if the server offers it and quietly continues in the clear if not. On a relay you control, mandatory turns a silent downgrade into a visible failure.
• from_address has to be something the relay accepts. Managed relays reject mail from unverified domains after the connection succeeds, which is why the Grafana side looks fine.
• Leave skip_verify false. It exists for internal relays with self-signed certificates. Turning it on to clear an error removes the check that was telling you the certificate is wrong.

Changes to this file need a Grafana restart. Nothing in the UI reloads SMTP settings.

Deliverability is a separate problem from delivery

Sending directly from a VPS is the fastest way to get alerts filed as spam. A fresh IP range from a provider like Contabo or InterServer has no sending reputation, and a mail server that emits one message a week during incidents never builds one.

Point Grafana at a transactional relay instead. Amazon SES, Postmark, Mailgun and Brevo all do the same job at this volume. Whichever you pick, publish SPF and DKIM records for the sending domain and set a DMARC policy, then send a real alert to a Gmail address and a Microsoft 365 address and check the spam folder. That is the only test that tells you anything.

Slack: webhook or bot token

Grafana supports two ways of posting to Slack, and the choice has consequences beyond setup time.

Incoming webhooks are the quick path. Slack generates a URL under hooks.slack.com/services/ that is bound to a single channel, and Grafana posts to it. There’s no OAuth flow and no scopes to reason about. The trade-off is that the URL is the credential and it maps to exactly one channel, so a team with six alert channels ends up managing six secrets with no central view of them.

A Slack app with a bot token takes longer. You create the app, add the chat:write.public scope so it can post to public channels without being invited to each one, then copy the bot user OAuth token, which starts with xoxb-. In Grafana you supply that token plus the channel ID as the recipient. One credential covers every channel, permissions are visible in one place, and revoking it kills all posting at once.

Webhooks win when you have one or two channels and want to be done in five minutes. The bot token wins as soon as channels start multiplying or someone asks what the integration is permitted to do. I reach for the webhook on small setups and regret it around the fourth channel.

Two constraints either way. A Slack integration targets one channel, so several channels means several integrations. And notification templates control the message title and body text, not its visual structure, so rich block layouts aren’t on the table.

Telegram: cheap paging, one nasty edge

Telegram is genuinely useful for out-of-hours alerts because it pushes to a phone without anyone paying for a paging product. The setup is two values: a bot API token and a chat ID.

1. Message BotFather and send /newbot. The username has to end in bot or _bot.
2. Copy the HTTP API token it returns. That’s the bot API token Grafana wants.
3. Create a group for alerts and add the bot to it. A bot cannot message a chat it has never seen.
4. Send a slash command in the group, such as /start@your_bot_name. An explicit command is the reliable way to make the group visible to the bot.
5. Fetch the chat ID from the Telegram API.

curl -s "https://api.telegram.org/bot<YOUR_BOT_TOKEN>/getUpdates" | jq '.result[].message.chat'

The id field in that output is what goes in Grafana’s Chat ID field. Group chat IDs are negative. Copy the minus sign; leaving it off is the single most common reason a Telegram contact point tests as failing.

Now the edge case, and it’s the reason Telegram belongs in a post about invisible failures. Telegram caps messages at 4096 UTF-8 characters. If you’re using a parse mode other than none, truncating at that boundary can leave the markup unbalanced, and Telegram rejects the whole message as malformed. You don’t get a shortened alert. You get nothing.

This is exactly backwards, because the message most likely to blow the limit is a large grouped alert during a real outage. The channel works perfectly for months of single alerts, then goes quiet the first time forty instances fire together. Three ways to deal with it, in the order I’d try them:

• Write a deliberately short notification template for Telegram: alert name, severity, instance count and a dashboard link. Treat it as a pager, not a report.
• Narrow group_by on the policy that feeds Telegram so a single notification covers fewer instances.
• Set the parse mode to none, which makes truncation harmless at the cost of formatting.

Routing: split by severity, not by tool

The most common design mistake is a policy tree organised around channels: a Slack policy, an email policy, a Telegram policy. That gives you no way to express the thing you actually care about, which is that some alerts should wake someone up and most should not.

Put severity on the alert rule as a label and route on that instead. Contact points then become bundles of channels chosen to match urgency.

• severity=critical goes to a contact point holding Telegram and Slack. Something is broken and someone needs to look now.
• severity=warning goes to Slack only, in a channel people read during working hours.
• Everything else falls through to the default policy and lands in email, where it accumulates without interrupting anyone.

Four timing settings control how noisy this is, and each one does something specific:

• group_by decides what counts as the same notification. Grouping by alertname and a service label means one cascading failure produces one message instead of forty.
• group_wait (default 30 seconds) is how long to hold the first notification for a new group, so related instances that fire a few seconds apart arrive together.
• group_interval (default 5 minutes) is the minimum gap before a group that has already notified sends again because new instances joined it.
• repeat_interval (default 4 hours) is how often an unresolved alert re-notifies with nothing new to say.

The temptation is to drop repeat_interval to fifteen minutes so nothing gets forgotten. Resist it. Teams that do this stop reading the channel within a month, which converts a routing problem into a much worse cultural one. Longer intervals plus a real escalation path beats short intervals every time.

Grafana also offers a simpler path: pick a contact point directly on the alert rule form and skip label matching entirely. Reasonable when a rule has one obvious destination. The trade-off is that routing knowledge moves into individual rules, so changing where a class of alerts goes means editing every rule instead of one policy.

Provision it as files, not clicks

Contact points built through the UI are invisible to review, easy to delete by accident, and impossible to recreate after a bad restore. If Grafana alert notifications matter enough to build, they matter enough to keep in Git. Grafana reads alerting configuration from files in the provisioning/alerting directory under its provisioning path at startup.

Before writing any YAML by hand, build one contact point in the UI, test it, then use the export option on the contact points list to dump it as YAML. That gives you the exact setting keys for that integration type instead of the ones you half-remember from a blog post. Setting names are not consistent between integration types, and a guessed key produces a config that loads without error and sends nothing.

apiVersion: 1

contactPoints:
  - orgId: 1
    name: critical-oncall
    receivers:
      # Telegram receiver goes here. Export it from the UI first:
      # the setting key names differ from what you would guess.
      - uid: critical-slack
        type: slack
        settings:
          url: https://hooks.slack.com/services/REPLACE/WITH/YOURS
        disableResolveMessage: false

  - orgId: 1
    name: warnings-slack
    receivers:
      - uid: warnings-slack
        type: slack
        settings:
          url: https://hooks.slack.com/services/REPLACE/WITH/YOURS

  - orgId: 1
    name: email-digest
    receivers:
      - uid: email-digest
        type: email
        settings:
          addresses: ops@example.com
          singleEmail: false

And the policy tree that uses them:

apiVersion: 1

policies:
  - orgId: 1
    receiver: email-digest
    group_by: [grafana_folder, alertname]
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - receiver: critical-oncall
        matchers:
          - severity = critical
        group_wait: 10s
        repeat_interval: 1h
      - receiver: warnings-slack
        matchers:
          - severity = warning
        repeat_interval: 12h

Two things to know about provisioned resources. By default they become read-only in the UI, which is the point but surprises people the first time. And a contact point that a policy references cannot be deleted until the policy stops referencing it, so removals have to happen in the right order.

Keep tokens out of the files themselves. Grafana supports environment variable substitution in provisioning files, which means the YAML can live in Git and the secrets can come from your existing secret store.

Detecting the silence

Everything above still leaves the original problem unsolved. If the Slack app gets revoked or the SMTP password rotates, notifications stop and Grafana does not tell you.

The fix is a heartbeat alert, and it’s worth the twenty minutes. Create a rule whose condition is always true, give it a low severity label, route it to each contact point with a short repeat_interval, and point the resulting notification at an external dead-man’s-switch service. Healthchecks.io, Cronitor and Better Stack all offer this, and Healthchecks.io can be self-hosted if you’d rather not add a dependency. The service expects a signal on a schedule and alerts you when it stops arriving.

The logic is inverted on purpose. You aren’t monitoring infrastructure here, you’re monitoring the path that tells you about your infrastructure, and it has to be watched by something that isn’t itself.

As a second layer, Grafana exposes its own internal metrics on a Prometheus-compatible endpoint, including counters for notification attempts and failures. If you’re already scraping Grafana, alert on that failure counter increasing. It won’t catch a credential that works but delivers to a dead mailbox, which is why the heartbeat comes first.

Troubleshooting: the rule fires, nothing arrives

Work down this list in order. Each step rules out one hop, and stopping early is how people spend an afternoon on the wrong layer.

1. Test the contact point in isolation. The Test button sends a notification without touching rules or policies. If it fails, the problem is credentials or network and nothing else matters yet.
2. Watch the Grafana logs during the test. On a systemd install, run journalctl -u grafana-server -f while you click Test. SMTP, Slack and Telegram delivery errors all surface here, usually specific enough to fix directly.
3. Compare the instance’s labels against your matchers. A policy matching severity = critical never sees severity = Critical. Matchers are exact and case-sensitive. Read the firing instance’s real labels, not the ones you think the rule sets.
4. Look for a policy earlier in the tree that swallowed it. Evaluation stops at the first match unless the policy continues. A broad route near the top silently absorbs everything below it.
5. Check active silences and mute timings. A silence created during last month’s maintenance with a generous matcher is the classic. So is a mute timing in the wrong timezone.
6. Check whether you’re inside repeat_interval. A red alert now does not mean a notification is due now. This looks identical to a broken integration and isn’t.
7. For email, read the relay’s own logs. Grafana handing the message off is not delivery. Bounces and spam filing happen after Grafana’s involvement ends.

Common mistakes

• Configuring the email contact point and never touching grafana.ini. The contact point is only a recipient list. Without the SMTP block, it has nothing to send with.
• Routing by channel instead of severity. It feels tidy and it makes urgency impossible to express.
• Dropping the minus sign from a Telegram group chat ID. Group IDs are negative and the field takes the value literally.
• Sending verbose templates to Telegram. The 4096-character limit turns your biggest incident into your quietest one.
• Testing with the Test button and calling it done. Test bypasses the policy tree entirely. It proves the credential works, not that a real alert would ever reach it.
• Leaving resolved notifications on for every channel. Resolution messages in a pager channel double the volume and train people to ignore it.
• Storing webhook URLs and bot tokens in the UI only. They exist in one place, are invisible to review, and disappear with the database.

Best practices for Grafana alert notifications

• Set a consistent severity label on every rule from day one. Retrofitting labels across dozens of rules is miserable work.
• Use one contact point per urgency level, holding as many integrations as that level needs.
• Put a dashboard or runbook link in an annotation your templates render. An alert that says what broke but not where to look costs the responder five minutes every time.
• Write channel-specific templates: short for Telegram, richer for email, in between for Slack.
• Provision contact points and policies from files, with secrets injected from the environment, and run a heartbeat through every channel.
• Re-test every integration after credential rotation, Slack workspace changes or a Grafana upgrade. Those are the three events that quietly break delivery.

FAQ

Can one contact point send to email, Slack and Telegram at the same time?

Yes. A contact point holds multiple integrations and every one of them fires when that contact point is selected. This is the right way to build an escalation tier: one contact point per severity, with the channel mix chosen to match how urgently that severity needs attention.

Why does the Test button work but real alerts never arrive?

Test sends straight to the contact point and skips the policy tree. If it works and real alerts don’t, the break is in routing: matchers that don’t match, a broader policy earlier in the tree absorbing the alert, an active silence, or a mute timing. Open the firing instance, read its real labels, and compare them character by character against your matchers.

Do I need Grafana Cloud for any of this?

No. Contact points, notification policies, templates and all three integrations ship in the open source build. Grafana Cloud handles hosting and adds on-call tooling on top, which matters if you need rotations and escalation chains. For a team routing alerts to three channels, a self-hosted instance on a modest VPS does everything described here.

Should I use a Slack webhook or a bot token?

Webhook for one or two channels where speed of setup matters. Bot token once you have several channels, want one revocable credential, or need to see in one place what the integration is allowed to do. Both deliver identically; the difference is entirely in credential management.

Why did my Telegram alerts stop during a big outage?

Almost certainly the 4096-character limit. A large grouped notification exceeded it, truncation broke the message markup, and Telegram rejected the send outright. Shorten the template for that channel, group more narrowly, or drop the parse mode. It’s the failure mode most worth guarding against, because it only shows up when you need the channel most.

Can I keep using the Prometheus Alertmanager I already run?

Yes. Grafana can send to an external Alertmanager instead of its built-in one, and the contact point selector at the top of the alerting pages lets you choose which one you’re configuring. If you already have a mature Alertmanager routing config, keeping it and pointing Grafana at it is usually less work than migrating the tree.

The one thing to remember

A firing rule proves evaluation works. It proves nothing about whether anyone was told. Grafana alert notifications cross three or four systems after the rule turns red, and every one of them fails quietly by default.

So build the channels carefully, route by severity rather than by tool, then run a heartbeat through every path and let something outside Grafana watch it. The alternative is finding out during the incident, which is the one time you have no attention to spare.

Need alerting that reaches people?

Most of the Grafana setups I get asked to look at have good dashboards and alerting nobody trusts. Work I take on here:

• Fixing email delivery end to end: SMTP config, relay selection, SPF, DKIM and DMARC so alerts stop landing in spam.
• Setting up Slack and Telegram contact points properly, including bot apps, scoped tokens and channel-appropriate templates.
• Designing a policy tree around severity labels, with grouping and timing tuned so people keep reading the channel.
• Converting click-built alerting into provisioned YAML under version control, with secrets kept out of Git.
• Adding heartbeat monitoring so a broken notification path announces itself instead of waiting for an outage.
• Auditing alert rules for missing labels, missing runbook links, and rules that fire constantly and are therefore ignored.

Send me your notification policy export and the output of journalctl -u grafana-server from a period when an alert should have gone out, and I’ll tell you which hop is dropping it.

The post Grafana Alert Notifications That Actually Arrive: Email, Slack and Telegram appeared first on John Nessime.

That gap is the entire problem. Monitoring n8n workflow failures in Grafana is not hard because the tooling is missing. It’s hard because the built-in metrics endpoint answers a question you didn’t ask. It tells you the process is alive and busy. It does not tell you that a workflow stopped producing correct output.

This post covers what n8n’s Prometheus endpoint actually exposes, the three families of failure it handles badly, and the specific PromQL and alert rules that close each gap. There’s also a troubleshooting section for the case where /metrics returns nothing at all, which is the first wall most people hit.

What the n8n metrics endpoint actually gives you

n8n uses the prom-client library and serves metrics on the same port as the editor, at /metrics. The endpoint is off by default, and almost everything interesting inside it is off by default too, behind its own environment variable.

N8N_METRICS=true
N8N_METRICS_INCLUDE_MESSAGE_EVENT_BUS_METRICS=true
N8N_METRICS_INCLUDE_WORKFLOW_ID_LABEL=true
N8N_METRICS_INCLUDE_NODE_TYPE_LABEL=true
N8N_METRICS_INCLUDE_QUEUE_METRICS=true

The first line opens the endpoint. The second turns on the workflow and node event counters, which is the part people assume is included and is not. The third attaches a workflow_id label to workflow metrics, without which you get a single instance-wide number and no way to tell which automation broke.

Those labels are opt-in for a reason worth understanding rather than working around. Every distinct label value creates a separate time series. Turn on workflow_id and you get one series per workflow per counter; add node_type and node counters fan out across every integration you use. On a few dozen workflows that’s fine. On several hundred with per-node labels, you’re signing up for a cardinality problem that surfaces later as slow queries and a Prometheus process that keeps getting OOM-killed.

With the event bus metrics enabled, the counters you get include n8n_workflow_started_total, n8n_workflow_success_total, n8n_node_started_total and n8n_node_finished_total. Queue mode adds n8n_scaling_mode_queue_jobs_active, _completed, _failed and _waiting, gathered from Bull and exposed on the main instance only.

Prometheus side, the scrape job is unremarkable:

scrape_configs:
  - job_name: n8n
    static_configs:
      - targets:
          - n8n-main:5678
          - n8n-worker-1:5678
    metrics_path: /metrics

Note both main and workers. Each process exposes its own endpoint and its own counters. Scrape only the main and you lose everything the workers did, which in queue mode is nearly all the real execution work.

One thing the n8n docs are emphatic about, and they’re right: don’t put /metrics on the public internet. It leaks workflow IDs, execution volumes and instance topology to anyone who asks. Keep the scrape path on a private network segment, a WireGuard tunnel, or your provider’s internal networking. If you’re running the whole stack on a single VPS from somewhere like Contabo or InterServer, binding n8n to the Docker network and scraping it by container name is enough. Publishing port 5678 to the host and firewalling it afterwards is not, because you will forget.

Failure family one: the execution that never happened

This is the one that costs the most and shows up the least. A schedule trigger gets deactivated during a deploy. A credential expires and the workflow is switched off rather than fixed. An upstream service stops calling a webhook. Nothing errors. Nothing increments. The workflow simply stops existing as far as your metrics are concerned.

Every error-rate alert you write is blind to this, because an error rate needs a denominator. Zero executions divided by zero executions is not an alert, it’s a NaN.

The instinct is to write this:

sum by (workflow_id) (increase(n8n_workflow_started_total[6h])) == 0

It looks correct and it does not work. When a workflow stops running, its series stops being exported. Prometheus marks it stale after a few missed scrapes and instant queries stop returning it. There is no series left for == 0 to match against. The alert never fires, because the thing you wanted to alert on is exactly the thing that removed the data.

What you want is a function that returns something when a series is absent:

absent_over_time(
  n8n_workflow_started_total{workflow_id="AbC123XyZ"}[6h]
)

absent_over_time returns 1 if the matched series produced no samples in the window and nothing at all if it did. That inversion is the point. The alert fires on silence.

The cost is that absent_over_time needs an exact matcher, so you can’t write it once for everything. That’s more acceptable than it sounds. You don’t need a heartbeat on all two hundred workflows, you need one on the six a human would notice within an hour. Write those six by hand and treat the list as part of the runbook.

One Grafana-specific trap. When a query returns nothing, Grafana’s alert rule enters a No Data state, which is configurable separately from the alert condition and defaults to firing in most versions. Set it deliberately. For an absent_over_time rule, No Data means the whole scrape target is gone, which is a different incident from one workflow going quiet, and you probably want it routed differently.

Failure family two: the failure you can see but can’t attribute

You have the counters. Something is failing. Which workflow?

Without N8N_METRICS_INCLUDE_WORKFLOW_ID_LABEL, the answer is: no idea. The counters are instance-wide totals. You know the number moved and that’s it. That’s a fine signal for capacity planning and useless during an incident.

Turn the label on and you get IDs, which are opaque strings like AbC123XyZ. A dashboard full of those is technically attributable and practically unreadable at 3am. Newer n8n versions solve this with a gauge that maps IDs to names, enabled with N8N_METRICS_INCLUDE_WORKFLOW_INFO:

n8n_workflow_info{workflow_id="VaQPuPmx9tPpo6BX",workflow_name="Invoice sync"} 1

It’s a constant-1 gauge that exists only to be joined against. The pattern is a standard Prometheus vector match:

sum by (workflow_name) (
  rate(n8n_workflow_started_total[5m])
  * on(workflow_id) group_left(workflow_name)
  n8n_workflow_info
)

The multiplication is arithmetically a no-op because the gauge is always 1. Its only job is to carry the workflow_name label across with group_left. Do this once in a Grafana variable or a recording rule rather than in every panel, or you’ll be copy-pasting the join for the rest of your life.

On an older build without the info gauge, the fallback is a hand-maintained relabel config or a Grafana value mapping. Both drift, and the relabel version drifts silently.

Failure family three: the failure that counts as a success

This is the invisible one, and it’s why a green n8n dashboard is not evidence of anything.

n8n gives you several deliberate ways to make a node’s failure not be an execution failure. They’re all reasonable features. Together they mean the success counter and “the automation worked” are different statements.

• Continue on error. A node set to continue passes its error downstream instead of stopping the run. The execution finishes. It finishes as a success.
• Error output branches. A node with a separate error output routes failures to a different path. That path might log and stop. The execution still completes normally.
• Never Error on HTTP responses. The HTTP Request node can be configured to treat any status code as a valid response. A run of 500s from an upstream API becomes a workflow that processed zero useful records, successfully.
• Empty result sets. A query that returns nothing is not an error. A sync that syncs nothing, every night, looks exactly like a sync that has nothing to do.

There are also user reports on the n8n community forum of started and success counters matching even when a node failed outright, in queue mode. I have not reproduced that and I’d treat it as unconfirmed rather than a known behavior. But it’s the right instinct: don’t assume the success counter is a strict complement of failure. Verify it on your own instance before you build alerting on the assumption.

For webhook-triggered workflows, newer n8n versions give you a real answer here. The n8n_webhook_request_duration_seconds histogram carries a status_code label, so you can alert on what the caller actually received rather than on what n8n thought about the run:

sum by (workflow_id) (
  rate(n8n_webhook_request_duration_seconds_count{status_code!~"2.."}[5m])
)

Worth flagging: the n8n documentation shows this example with status_code!="2..", using the equality operator. In PromQL, != compares against a literal string, so that matcher excludes only a status code whose literal value is the three characters 2.., which never occurs. It matches everything, including your 200s. Use !~ for the regex form. Test any label matcher you copy from documentation by running it in Prometheus and checking the series count against what you expect.

Getting a second source of truth

For everything that isn’t a webhook, the metrics endpoint alone won’t close this gap. You need a second signal that reads execution outcomes from n8n’s own store.

The public REST API exposes exactly that. Executions can be filtered by status, and the failed ones are one request away:

curl -s -H "X-N8N-API-KEY: $N8N_API_KEY" 
  "https://n8n.example.com/api/v1/executions?status=error&limit=50"

Two ways to get that into Grafana. Write a small exporter that polls the API on a timer and serves its own /metrics, or run a scheduled job that pushes counts to a Prometheus Pushgateway. The exporter is more work and behaves correctly. Pushgateway is faster to build and carries the usual caveat: pushed metrics persist until deleted, so a job that stops running leaves a stale value sitting there looking healthy. That reintroduces failure family one through the back door.

Two caveats on the API. There’s an open report that the status=crashed list filter returns executions whose stored status is something else entirely when re-fetched by ID, which would make crash alerting built on the list endpoint unreliable. Check the behavior on your version before depending on it. And n8n’s log streaming feature, which forwards execution events to syslog or a webhook in real time, is an Enterprise capability. If you’re on the community edition, the API poll is your path.

The third option, and often the best one for a small setup, is an Error Trigger workflow. n8n lets you designate a workflow that runs whenever another workflow fails. Point it at Pushgateway, at Loki, or at a Grafana webhook contact point. It’s native, it needs no API key, and it gives you the error message rather than just a count. The trade-off is that it only fires on real execution failures, so it inherits every blind spot in this section.

The Grafana panels that earn their space

Four panels cover most of what you need. Everything past that is usually decoration.

1. Executions per workflow, by name. A time series of rate(n8n_workflow_started_total[5m]) joined to n8n_workflow_info. Its real job is showing you the lines that flatten to zero.
2. Success ratio. rate(n8n_workflow_success_total[15m]) / rate(n8n_workflow_started_total[15m]). Anything below 1 is worth a look. Anything above 1 means your window is catching a restart and the counters reset.
3. Queue depth and failed jobs. n8n_scaling_mode_queue_jobs_waiting next to increase(n8n_scaling_mode_queue_jobs_failed[1h]). A waiting count that climbs and never drains means workers are dead or starved, which looks identical to “no failures” on every other panel.
4. Webhook non-2xx rate. The query from the previous section. This is the only panel that reports what the outside world experienced.

Skip the node-level breakdown unless you have a specific reason. It is the most expensive thing you can enable in cardinality terms and it answers a question you can answer faster by opening the execution in the n8n UI.

Alert rules worth paging on

1. Named workflow silent. absent_over_time on each business-critical workflow, window set to roughly twice its expected interval. This is the rule that catches the invoice sync.
2. Success ratio below threshold. Sustained for 15 minutes, not instantaneous. A single failed run is not a page.
3. Queue waiting count rising. Compare the current value to fifteen minutes ago rather than to a fixed number, so the threshold survives traffic growth.
4. Webhook error rate. Per workflow, so a single noisy endpoint doesn’t mask the rest.
5. Scrape target down. The up metric. Obvious, and routinely forgotten, and it invalidates every other rule on this list.

Troubleshooting: the endpoint returns nothing useful

• “Cannot GET /metrics”. N8N_METRICS isn’t set on the process you’re hitting, or it’s set as a string that n8n doesn’t read as true. In Docker Compose, quote it as "true" and confirm with docker compose exec n8n env | grep N8N_METRICS.
• Endpoint responds but has no n8n_ counters. You have the default Node.js process metrics and nothing else. Set N8N_METRICS_INCLUDE_MESSAGE_EVENT_BUS_METRICS=true and restart.
• Counters exist but no workflow_id label. N8N_METRICS_INCLUDE_WORKFLOW_ID_LABEL is off. Restart is required; it’s read at startup.
• Queue metrics are zero or missing. They’re gathered from Bull and exposed on the main instance, not the workers. Querying the worker endpoint gives you nothing. Confirm you’re looking at the right target.
• Counters look wrong after a deploy. They reset to zero when the process restarts. Always wrap them in rate() or increase(), which handle counter resets. Raw counter values on a dashboard will lie to you every time you deploy.
• Prometheus can’t reach the target. Usually a Docker network mismatch. Scrape by container name on the shared network rather than by published host port, and check the target page in Prometheus for the actual error rather than guessing.

Common mistakes

• Treating the success counter as proof the automation worked. It proves the run finished.
• Building only error-rate alerts, so the workflows that stop entirely go unmonitored.
• Enabling every N8N_METRICS_INCLUDE_* variable on day one and discovering the cardinality bill a month later.
• Scraping only the main instance in queue mode and losing all worker-side execution data.
• Copying label matchers out of documentation without running them and checking the returned series count.
• Exposing /metrics publicly because it’s “just numbers”.

Best practices

• Decide which workflows are business-critical and write a silence alert for each one by name. Six good heartbeats beat two hundred generic panels.
• Run a second source of truth. Metrics tell you about volume, the executions API tells you about outcomes. You need both.
• Push the ID-to-name join into a recording rule so panels stay readable and cheap.
• Break a staging workflow on purpose and confirm the alert fires. An untested alert is a decoration.
• If you’d rather not run Prometheus and Grafana yourself, Grafana Cloud will scrape a remote endpoint and gives you alerting without a control plane to maintain. Weigh that against your metrics leaving your network.

FAQ

Does n8n expose a failed-executions counter?

Not a reliable per-workflow one. You get started and success counters from the event bus, and a failed-jobs counter for queue mode. Deriving failures by subtracting success from started works only if you’ve verified on your own instance that a failing run genuinely doesn’t increment success, which is exactly the assumption that breaks when nodes are set to continue on error.

Can I monitor n8n Cloud this way?

No. The /metrics endpoint isn’t available on n8n Cloud. On Cloud you’re limited to the public API and the built-in insights views, so the executions-API approach in this post is the only one that transfers.

Why does my workflow error rate look like zero when I know things are failing?

Almost always one of three things: the event bus metrics aren’t enabled so you’re only seeing Node.js process metrics, the failures are being absorbed by continue-on-error settings and recorded as successes, or you’re scraping the main instance while the work happens on workers.

How do I alert when a scheduled workflow stops running entirely?

Use absent_over_time against that workflow’s started counter with an exact workflow_id matcher, and set the window to about twice the expected interval. A threshold comparison won’t work, because a workflow that stops running stops producing the series you’d compare against.

Is enabling the workflow_id label safe on a large instance?

It multiplies your n8n series count by the number of workflows. On a few dozen it’s negligible. Past a few hundred, combined with node-type labels, watch your Prometheus memory and head series count before and after. Enable the workflow label, leave the node label off, and only turn the node one on when you have a specific question it answers.

Should I use OpenTelemetry instead of Prometheus for n8n?

They solve different problems. Metrics answer “how many, how often, how fast” cheaply and forever. Traces answer “where did this run spend its time and what did it call”, which is better for debugging a slow multi-service workflow. Tracing n8n usually means a custom image with the OpenTelemetry SDK loaded before startup, so it’s a project rather than an environment variable. Start with metrics; add tracing when you have a latency question metrics can’t answer.

Where do the n8n counters go when the container restarts?

They reset to zero. They’re in-process counters with no persistence. Prometheus handles this correctly as long as your queries use rate() or increase(). Any panel showing a raw counter value will appear to lose all its history on every deploy.

The one thing to take away

Monitoring n8n workflow failures in Grafana works, but only once you stop asking the metrics endpoint a question it can’t answer. It reports activity. Activity and correctness are different things, and the space between them is where the expensive outages live.

So build for absence first. A silence alert on the handful of workflows a human would miss will catch more real incidents than any error-rate panel, because errors are loud and stopped schedules are not. Then add attribution with the workflow label, then close the false-success gap with the executions API or an Error Trigger workflow. In that order, because each one is cheap and the value drops off fast after the third.

Need help wiring this up?

I build and maintain observability stacks for self-hosted automation platforms, and n8n is a common one. If any of the above sounds like a week you’d rather not spend, here’s what I can take off your plate:

• Enabling and securing the n8n metrics endpoint on a private network path, including queue-mode setups with multiple workers
• Building Prometheus scrape configs, recording rules and the ID-to-name join so your Grafana panels are readable rather than a wall of hashes
• Writing silence and error-rate alert rules that survive counter resets, Grafana No Data states and traffic growth
• Adding a second source of truth from the n8n executions API, either as a small exporter or an Error Trigger workflow feeding Loki or Pushgateway
• Auditing an existing setup for cardinality problems, stale Pushgateway series and alerts that quietly stopped being able to fire
• Running a failure drill against staging so you know which alerts actually work before you need them

Send me your Compose file, your prometheus.yml, or a screenshot of the panel that looks fine while things are on fire, and I’ll tell you what’s missing.

The post Monitoring n8n Workflow Failures in Grafana: Why Your Green Dashboard Is Lying appeared first on John Nessime.

Sometimes the disk really is dying. More often it isn’t, and the number you’re staring at is not measuring what you think it’s measuring.

This post covers troubleshooting high I/O wait the way I’d actually do it on a production box: what the metric means, why it lies in both directions, and how to work through the five failure families that produce it.

What high I/O wait actually measures

Here’s the definition that matters: iowait is a subcategory of idle time. For a given CPU, it’s the time that CPU spent doing nothing while at least one task scheduled on it had an outstanding block I/O request.

Two consequences fall out of that, and both will burn you. High iowait can mean nothing is wrong. If the box has spare CPU and one process is pulling a large file off disk, the CPU genuinely has nothing else to do. Copying a backup will push iowait toward 100 percent on an idle host, and that’s a working system, not a fault.

Low iowait can hide a severe problem. This is the one that costs people days. If the CPU has other runnable work, the kernel context switches to it and that time gets counted as user or system, not iowait. A busy application server can be stalling badly on storage while wa reads two percent, because the CPU is never actually idle.

Add the averaging problem on top. The wa figure in top is averaged across every core. One thread pinned to one core and blocked hard on disk shows up as roughly three percent on a 32-core machine. Press 1 in top to break out per-CPU lines before you conclude anything from the aggregate.

So iowait is a hint, not a verdict. Treat it as the thing that starts the investigation, never the thing that ends it.

Start with pressure, not percentages

Pressure Stall Information is the metric I reach for first, and it exists precisely because iowait is a poor proxy for pain. It answers the question you actually care about: how much time did tasks lose because they couldn’t get I/O?

cat /proc/pressure/io

You get two lines. some is the share of time at least one task was stalled waiting on I/O. full is the share of time every non-idle task was stalled at once, which is CPU capacity being burned for nothing. The avg10, avg60 and avg300 fields are percentages over the last 10, 60 and 300 seconds. total is a cumulative microsecond counter since boot, which is what you want for graphing because averages smooth away short spikes.

The practical rule I use: a rising full line on io is a real user-visible problem almost every time. A high some line with a flat full line usually means one noisy job is waiting while everything else gets on with its life.

If /proc/pressure doesn’t exist, the kernel was built without CONFIG_PSI, or it was built with PSI compiled in but disabled by default, in which case you need psi=1 on the kernel command line. On cgroup v2 systems you also get per-cgroup pressure, which is how you narrow from “the host is stalling” to “this container is stalling” in one command:

# Pressure for one systemd slice, useful on any modern distro
cat /sys/fs/cgroup/system.slice/io.pressure

# Pressure for every leaf cgroup that has it, sorted by nothing,
# just eyeball the full lines
find /sys/fs/cgroup -name io.pressure -exec sh -c 
  'printf "%s: " "$1"; sed -n 2p "$1"' _ {} ;

If you run Prometheus, node_exporter has exposed these since it gained a pressure collector. The counters are node_pressure_io_waiting_seconds_total for some and node_pressure_io_stalled_seconds_total for full. Rate those and you have a far better storage alert than any threshold on %iowait will ever be. Grafana Cloud and Netdata both chart PSI out of the box if you’d rather not build the panels yourself.

Failure family one: the device really is saturated

This is the case everyone assumes, so get it confirmed or eliminated quickly.

iostat -xy 1

The -x gives extended per-device stats. The -y matters more than people realise: without it, the first report is a since-boot average, and reading that first block is the single most common iostat mistake there is. It will tell you the disk has been fine all year, which is true and useless.

Now the columns worth your attention:

• r_await and w_await: average milliseconds a read or write took, queue time included. This is the closest thing to “how slow does storage feel”. Split reads from writes, because they fail very differently.
• aqu-sz: average queue depth. Requests either queued or in service. A depth that climbs while throughput stays flat is the signature of a genuinely saturated device.
• r/s, w/s, rkB/s, wkB/s: the actual workload. Compare against what the device or volume class is rated for.
• %util: treat with suspicion, see below.

The %util trap. That column counts the fraction of time at least one request was in flight. On a spinning disk that serviced one request at a time, it was a fair saturation signal. On NVMe and on cloud volumes that handle many requests in parallel, a device can sit at 100 percent while running at a fraction of its real capacity. Provisioning a bigger volume off the back of a 99 percent %util reading is a common and expensive way to change nothing. Use aqu-sz and the await columns instead. Similarly, svctm was deprecated and dropped from newer sysstat output because it was unreliable on parallel devices; ignore it if an old build still prints it.

Once you know the device is busy, find the culprit. pidstat -d 1 gives per-process read and write rates from the same sysstat package and needs nothing special enabled. iotop -o is friendlier, but on kernels from 5.14 onward per-task delay accounting is off by default, so the SWAPIN and IO> columns come back empty until you turn it on:

sudo sysctl kernel.task_delayacct=1
sudo iotop -oPa
# turn it back off when you're done, it costs a little scheduler overhead
sudo sysctl kernel.task_delayacct=0

And check the boring things before the clever ones. dmesg -T | grep -iE 'i/o error|reset|medium' catches a failing drive or a flapping controller. smartctl -a /dev/sda catches reallocated sectors on hardware you own. If either of those turns something up, stop reading and replace the disk.

Failure family two: memory pressure in an I/O costume

This is the failure family I’d check second, and it’s invisible if you only look at storage.

When RAM gets tight, the kernel starts evicting page cache and swapping. Both generate disk traffic. The disk gets busy, iowait climbs, and every graph points at storage while the actual constraint is memory. Give the box more RAM and the “disk problem” disappears.

# si and so are swap-in and swap-out in KB/s.
# Sustained non-zero values are your answer.
vmstat 1

# Are we thrashing writeback? Dirty pages waiting to be written.
grep -E 'Dirty|Writeback' /proc/meminfo

# The direct question
cat /proc/pressure/memory

If memory pressure and I/O pressure rise together, fix memory first. Everything else is downstream.

There’s a related trap on the write side. The kernel buffers writes in page cache and flushes them in the background, governed by vm.dirty_background_ratio and vm.dirty_ratio. Cross the background threshold and flusher threads start writing. Cross the hard vm.dirty_ratio threshold and writing processes are forced to do writeback themselves, synchronously. That transition is brutal, and on a box with plenty of RAM and a modest disk the defaults let you accumulate gigabytes of dirty pages and then pay for all of them at once. Lowering the ratios makes writeback more frequent and less spiky, trading away a little write batching. Measure before you commit to it.

Failure family three: small synchronous writes

Throughput can be trivial and the system still crawls. A workload doing thousands of tiny fsync calls a second is bound by round-trip latency, not bandwidth, and no amount of extra megabytes per second will help.

Databases are the usual source, because durability has a cost and that cost is a flush. MySQL and MariaDB expose it through innodb_flush_log_at_trx_commit; PostgreSQL through synchronous_commit. Loosening either buys real throughput and trades away durability on power loss or crash. That can be a perfectly reasonable trade for a cache or an analytics replica, and a terrible one for anything holding orders or payments. Decide it explicitly, write down what you decided, and don’t let it be a thing somebody changed once at 2am.

The other common source is logging. Application logs, access logs, container JSON logs and a database sharing one volume will starve each other. Splitting hot write paths onto separate devices is unglamorous and works.

Failure family four: throttled, not saturated

A throttled device and a saturated device look almost identical from inside the guest: high await, growing queue, miserable latency. The difference is that no amount of tuning inside the box will fix a throttle. Three flavours are worth knowing.

• Cloud volume limits. Managed block storage gives you a baseline of IOPS and throughput, sometimes with a burst allowance on top. Burn the burst and you drop to baseline, and everything that was fine in testing falls over in production. The tell is a latency cliff that arrives at a consistent point into a workload rather than tracking load smoothly. On AWS, watch the volume-level CloudWatch metrics such as BurstBalance on burstable volume types and VolumeQueueLength; other providers publish equivalents. The fix is a volume class change, not a sysctl.
• cgroup limits you set yourself. On cgroup v2, io.max caps a cgroup at a given bytes-per-second or IOPS figure, and io.weight shares bandwidth proportionally under contention. A limit that made sense when it was written can become the bottleneck a year later, and nothing in the container’s own metrics announces it.
• Neighbours on shared hardware. On oversubscribed VPS plans your storage is shared, and someone else’s backup job is your latency spike. This is where cheap and expensive hosting genuinely differ. Budget providers like Contabo and InterServer are fine value for a lot of workloads, but if storage latency is load-bearing for you, look for plans with dedicated NVMe rather than shared pools, and test before you migrate anything that matters.

# Per-device I/O accounting for a cgroup, including any throttling
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/io.stat

# Is a cap set? An empty file means no limit on that cgroup.
cat /sys/fs/cgroup/system.slice/docker-<id>.scope/io.max

One more signal while you’re here. In top, the st column is steal time, meaning the hypervisor gave your vCPU’s slice to someone else. Steal and iowait rising together on a shared host is a strong hint that the problem is the neighbourhood, not your configuration.

Failure family five: the latency isn’t in the block layer

Here’s the failure mode that wastes the most time, because every storage metric you check will look healthy.

iostat measures the block layer. It cannot see anything above it. So if your latency is in NFS, in a network filesystem, in an overlay or FUSE mount, in filesystem lock contention, or in a journal that’s serialising writes, the device will report sub-millisecond service times while your application waits tens of milliseconds per operation.

The diagnosis is the gap. Measure latency at the application’s layer and at the block layer, and if the two numbers disagree wildly, the problem lives in between.

# Block layer latency as a histogram. Look for a bimodal shape
# or a long tail rather than the average.
sudo biolatency-bpfcc -m 10 1      # Debian/Ubuntu naming
sudo /usr/share/bcc/tools/biolatency -m 10 1   # RHEL family

# Filesystem operations slower than 10ms, with the process and file
sudo ext4slower-bpfcc 10
sudo xfsslower-bpfcc 10

# Everything at the VFS layer, higher overhead, use briefly
sudo fileslower-bpfcc 10

If ext4slower is full of hundred-millisecond operations and biolatency shows everything completing in microseconds, you have your answer and it is not the disk. For NFS specifically, nfsiostat splits out RTT from execution time, which separates network cost from server cost. Install the tooling with bpfcc-tools on Debian and Ubuntu, or bcc-tools on RHEL, AlmaLinux and Rocky.

A working order for troubleshooting high I/O wait

On an unfamiliar box, this is the sequence I work through. Each step either finds the problem or eliminates a family.

1. Read /proc/pressure/io. If full is near zero, the stall is not hurting the whole system and you should widen the search before you tune storage.
2. Check /proc/pressure/memory at the same time. If it’s rising too, fix memory first.
3. Run iostat -xy 1 for thirty seconds. Note r_await, w_await and aqu-sz per device. Ignore %util.
4. List processes stuck in uninterruptible sleep: ps -eo state,pid,comm | awk '$1 ~ /^D/'. That’s your blocked set, and it names the workload.
5. Attribute the traffic with pidstat -d 1. Confirm the process you suspect is the process actually doing the I/O.
6. If await is high, decide saturated versus throttled. Check cgroup io.max, check provider volume metrics, check steal time.
7. If await is low but the application is slow, jump to the layer gap. Run biolatency and ext4slower together and compare.
8. Only now consider tuning: dirty ratios, I/O scheduler, flush settings, splitting volumes.

Common mistakes

• Reading iostat’s first report. It’s a since-boot average. Use -y, or discard the first block.
• Trusting %util on NVMe or cloud volumes. Parallel devices break the assumption it was built on.
• Alerting on a fixed %iowait threshold. It fires on healthy backups and stays silent during real stalls on busy hosts.
• Changing the I/O scheduler first. Going from mq-deadline to none on NVMe is sometimes right, but it’s a tuning step, not a diagnosis, and it fixes nothing if you’re throttled.
• Adding swap to fix swapping. More swap on a memory-starved box means more disk traffic, not less.
• Stopping at the block layer. If every storage metric looks clean and the app is still slow, you’re measuring at the wrong layer, not measuring a healthy system.
• Testing with the wrong benchmark. A sequential dd tells you nothing about a workload doing random 4K synchronous writes. Use fio and match the real access pattern.

Best practices worth the effort

• Record PSI continuously. Storage incidents are short and bursty. Without history you’re guessing about something that already ended.
• Capture a baseline while things are healthy. Save an iostat -xy sample and a PSI reading from a normal day. “Is 8ms bad?” is unanswerable without one.
• Keep sar running. The sysstat package can archive metrics on a schedule, which means sar -d can show you what happened at 3am. Enable it before you need it.
• Separate hot write paths. Database data, database logs and application logs on separate volumes stops one from starving the others.
• Cap container logs. An uncapped JSON log driver quietly filling the root volume is a recurring cause of host-wide stalls.
• Alert on latency and pressure, not utilisation. Users feel w_await and PSI full. Nobody has ever felt a percentage of device busy time.

Frequently asked questions

What is a normal or acceptable I/O wait percentage?

There isn’t one, and any article that gives you a single number is guessing. Iowait scales with how idle your CPUs are, so the same workload produces very different figures on a busy server and a quiet one. Judge against your own baseline, and prefer PSI, which measures lost time directly rather than as a share of idle.

Can high I/O wait be caused by something other than the disk?

Regularly. Memory pressure driving swap and page cache eviction is the most common non-disk cause. Network filesystems, hypervisor contention on shared hosts, and cgroup or cloud volume throttling all produce the same symptom. The block device being genuinely slow is one possibility among several.

Why does iostat show 100% utilisation when the disk isn’t busy?

Because %util measures the fraction of time at least one request was in flight, which made sense for devices that handled one request at a time. NVMe drives and cloud volumes process many requests in parallel, so they can be continuously occupied while nowhere near capacity. Read aqu-sz and the await columns instead.

How do I find which process is causing I/O wait?

Start with pidstat -d 1, which needs no special kernel settings. List processes in uninterruptible sleep with ps -eo state,pid,comm and filter for state D to see who is actually blocked. iotop -o gives a live view but needs kernel.task_delayacct=1 on recent kernels for its percentage columns to populate.

Does adding more RAM reduce I/O wait?

Often, yes, and it’s underrated. More RAM means a larger page cache, fewer reads reaching the device, and less swapping. If vmstat shows sustained swap activity or memory pressure tracks I/O pressure, RAM is likely the cheapest fix available.

Should I change the I/O scheduler to fix high I/O wait?

Only after you know the device is the bottleneck. Check the current setting with cat /sys/block/sda/queue/scheduler. On NVMe, none is a common and sensible default because the hardware already does the queuing. On spinning disks, mq-deadline or bfq give better latency behaviour. It’s a tuning knob, not a diagnostic, and it will not help if you’re being throttled.

Why is my application slow when iostat says storage is fine?

Because iostat only sees the block layer. Latency in NFS, FUSE, overlay filesystems, journal serialisation or filesystem locking is entirely invisible to it. Run ext4slower or xfsslower alongside biolatency: if the filesystem is slow and the device is fast, the gap between them is your problem.

The one thing worth remembering

High I/O wait is a symptom that points in a general direction, not a diagnosis. It’s a slice of idle time, averaged across cores, measured at one specific layer of the stack. It goes up when nothing is wrong and stays flat when something is very wrong.

Read PSI to find out whether anyone is actually suffering. Read r_await, w_await and aqu-sz to find out whether the device is the constraint. Compare filesystem latency against block latency to find out whether you’re even looking at the right layer. Do those three things and the answer usually falls out in a few minutes, instead of after a week of arguing with your hosting provider about a volume that was fine all along.

Need someone to look at your storage stalls?

I work with teams on exactly this kind of problem, usually the ones that have already survived a round of “the host says the disk is fine”. Things I can help with:

• Diagnosing a live or recurring I/O stall and identifying which failure family it belongs to
• Setting up PSI, sar and per-cgroup I/O metrics in Prometheus and Grafana, with alerts that fire on latency rather than utilisation
• Separating cloud volume throttling from genuine device saturation, and sizing storage correctly instead of guessing upward
• Tuning writeback, dirty ratios and I/O schedulers for a specific workload, with before-and-after measurements
• Tracing container and Kubernetes I/O contention through cgroup v2 limits, overlay filesystems and log drivers
• Benchmarking with fio against your real access pattern so capacity decisions rest on something other than a dd run

If you’ve got an iostat capture, a PSI reading or just a graph that doesn’t make sense, send it over and I’ll tell you what I see in it.

The post Troubleshooting High I/O Wait Without Blaming the Disk appeared first on John Nessime.