No. And that question is the entire problem with this one.

This post is a reference for the PHP2Shell vulnerability: what actually broke inside PHP, which servers are exposed, how to patch each common setup, how to prove the fix is really running, and which quick fixes give you false comfort. Keep it open while you work through your servers.

What happened: the PHP2Shell vulnerability in plain terms

PHP2Shell is the nickname researchers gave to two flaws in the PHP engine itself, CVE-2026-109618 and CVE-2026-109619. PHP rates both 9.2 on CVSS v4.0, and both can let an unauthenticated attacker run code on a web server. The name is the researchers’, not the PHP project’s.

The dates matter here, because they decide if you were exposed and for how long:

• 20 October 2026: PHP announced an emergency release for two critical issues. Details leaked publicly the same day.
• 21 October 2026: fixed releases shipped: PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35, with a full advisory.
• 23 October 2026: PHP published an informational advisory for a third, much narrower issue, CVE-2026-115560, with a one-line mitigation and no emergency release.

The flaws were reported by Wiz Research, Calif and OpenAI’s Codex Security team, who used AI models to turn long-standing weaknesses into working exploits. The affected code goes back to PHP 5.4, so effectively every PHP version from the last decade is in scope.

The two bugs, without the jargon

CVE-2026-109618 (out-of-memory corruption). PHP enforces memory_limit by aborting a request the instant it crosses the limit. If that abort lands while PHP is still building internal request data, for example while parsing a multipart file upload, the data is left half-initialized. Request cleanup then treats it as valid and works with memory that can still hold attacker-controlled leftovers.

CVE-2026-109619 (stale global state). Some state owned by one request can survive after its memory has been freed, for example after a fatal error. The next request handled by the same worker can use that stale state before normal setup resets it.

Here is why this is worse than the usual plugin CVE. The bug lives in the runtime, not in your code. A site whose own code is flawless is still exposed, and updating WordPress, Laravel or Drupal changes nothing, because the PHP binary underneath comes from your OS packages, your host or your container image. The researchers also demonstrated reading memory from the PHP worker process, which can contain passwords, tokens and keys belonging to other people’s requests, without ever getting a shell.

Who is exposed, and why mod_php is the worst case

PHP’s advisory covers every installation that serves web requests: PHP-FPM, Apache with mod_php, php-cgi, FrankenPHP and LiteSpeed. The risk is not equal across them, though.

mod_php is the highest risk. Apache with mod_php runs request shutdown even when request startup already failed. That means simply running out of memory while PHP parses the request body is enough, and one unauthenticated request can trigger the attack before a single line of your application runs. PHP demonstrated this with default memory and input limits.

PHP-FPM is harder to hit, not safe. FPM shuts the worker down when a request fails early in setup, which blocks the cleanest path. Public reporting says one of the two bugs was still shown to reach code execution under FPM, through a path that depends on something the application does. Treat “we run FPM” as “we have slightly more time,” nothing more.

If you are not sure which mode you run, these two checks answer it on most Linux servers. The first lists Apache modules and shows a PHP module only if mod_php is loaded. The second lists running FPM services.

apachectl -M | grep -i php
systemctl list-units --type=service | grep -i fpm

How to fix PHP2Shell: it depends on where your PHP comes from

The fix is always the same in principle: run a patched PHP build. Getting there depends entirely on who builds your PHP. And there is a trap that runs in both directions. Linux distributions and long-term-support vendors received the patch early and backported it, so a patched package can carry an old-looking version like 8.3.6. Meanwhile, an unpatched process can keep running after the package updated. The version string answers neither question reliably.

Debian and Ubuntu packages

Debian, Ubuntu and Ondřej Surý’s widely used repository (packages.sury.org, also published as ppa:ondrej/php) all shipped fixed packages, including builds for some end-of-life branches. Update normally, then confirm the CVE is in the package changelog instead of reading the version number. Replace 8.3 with your branch.

sudo apt update && sudo apt upgrade
zgrep CVE-2026-109618 /usr/share/doc/php8.3-common/changelog.Debian.gz
sudo systemctl restart php8.3-fpm
sudo systemctl restart apache2

If zgrep prints a line, the fix is in the installed package. If it prints nothing, you are either on an older distro release that needs a paid extended-support channel, or on a repository that has not shipped the fix.

RHEL, AlmaLinux and Rocky Linux

Same idea with RPM. Update, then search the package changelog for the CVE. If you use Remi’s parallel packages, the package name follows its own pattern (for example php83-php-common), so query that name instead.

sudo dnf upgrade 'php*'
rpm -q --changelog php-common | grep CVE-2026-109618
sudo systemctl restart php-fpm httpd

One warning from PHP’s own advisory: the volunteer-run rpms.remirepo.net repository was not expected to have updates on release day, and PHP suggested switching to distribution packages or building from source if you depend on it. Check before assuming the update landed.

cPanel, CloudLinux and LiteSpeed

cPanel ships its own ea-php packages through EasyApache 4, and CloudLinux ships alt-php. CloudLinux backported the fix into supported and end-of-life branches, so most of its fixed builds keep version numbers below the php.net release. CloudLinux’s published procedure is to update both package families, then restart the services that run PHP so no worker keeps the old binary loaded.

yum clean all && yum update 'alt-php*' 'ea-php*' -y
/scripts/restartsrv_httpd

If your domains run PHP-FPM through cPanel, restart the FPM service as well. On LiteSpeed, restart the web server so every lsphp process respawns from the new build. Public analysis says LiteSpeed’s LSAPI behaves more like FPM than mod_php, which buys time but does not remove the need to patch.

DirectAdmin with CustomBuild

CustomBuild compiles PHP from source, so you get the fix when CustomBuild offers the patched release and you rebuild. Refresh the version list, confirm the fixed release appears for each PHP version you run, then build.

cd /usr/local/directadmin/custombuild
./build update
./build versions
./build update_versions

Read the ./build versions output before running the last command. If a branch you use still shows an unpatched release, apply the mitigations below for that branch until it updates. Compiling takes a while on small servers, so plan for it rather than starting it at peak traffic.

Docker and container images

A container never patches itself. If your image is built FROM an official php tag, rebuild with a fresh pull of the base image and redeploy. If your Dockerfile pins the base by digest, the digest has to change or nothing happens.

docker compose build --pull
docker compose up -d

Then run php -v inside the running container, not on the host, to confirm the version the app actually uses.

Managed hosting versus unmanaged VPS

On managed hosting, the host owns the PHP runtime, and several large hosts announced they had patched their fleets. Your job there is to get written confirmation, not to assume. On an unmanaged VPS from Contabo, Interserver or any similar provider, nobody patches PHP but you. That is the trade you accepted for the lower price and full root access, and this week is when it gets collected.

End-of-life PHP: 8.1 and older

The PHP project released no fixed builds below 8.2. Because of the severity, its security team published minimal source backports for PHP 7.4, 8.0 and 8.1 for people who build from source. Beyond that, your options are a distribution or vendor that backports (Ubuntu’s own PHP 8.1 on 22.04, the Surý repository, Freexian ELTS for older Debian, CloudLinux and TuxCare, HeroDevs), or upgrading. Note that PHP 8.2 itself reaches end of life on 31 December 2026, so if you are touching an 8.2 server now, plan the next upgrade at the same time.

Prove the patch is actually running

This is the failure mode that will bite most people. The package manager says “upgraded,” the ticket gets closed, and the old PHP code keeps serving traffic because the processes that loaded it were never restarted. Linux happily keeps running a binary whose file has been replaced on disk.

1. Confirm the fix is in the package with the changelog check above, not the version string.
2. Restart everything that runs PHP: every FPM pool, Apache (for mod_php), LiteSpeed, and any long-running PHP workers such as queue consumers managed by Supervisor or systemd.
3. Check the restart time is after the upgrade. systemctl status php8.3-fpm shows when the service started. If that is earlier than your package install, it is still running old code.
4. Ask the system what still needs a restart. On Debian and Ubuntu, sudo needrestart -r l lists services still using outdated files, without restarting anything.
5. Check the right binary. php -v on the command line reports the CLI build, which can differ from the FPM build serving your site. On Debian and Ubuntu, php-fpm8.3 -v asks the FPM binary directly.

Mitigations if you cannot patch today, and what they don’t do

PHP is blunt about this: only an update is a complete fix. The settings below reduce exposure to the specific attack paths it has confirmed. They do not make a server safe, and some of them have real costs. Use them to buy hours, not weeks.

Set them where applications cannot override them: php_admin_value or php_admin_flag in the FPM pool, or the Apache configuration for mod_php. Do not rely on ini_set() or an FPM .user.ini file, because those apply too late in the request. Here is the FPM pool version on a Debian-style path (adjust the file and branch to yours):

; /etc/php/8.3/fpm/pool.d/www.conf
php_admin_value[memory_limit] = -1
php_admin_value[post_max_size] = 8M
php_admin_value[upload_max_filesize] = 6M
php_admin_flag[session.upload_progress.enabled] = off

And the Apache equivalent for mod_php, inside the virtual host or a Directory block:

php_admin_value memory_limit -1
php_admin_value post_max_size 8M
php_admin_value upload_max_filesize 6M
php_admin_flag session.upload_progress.enabled off

Restart every worker afterwards. Now the reasoning behind each line, because each one protects a different path:

• memory_limit = -1 targets CVE-2026-109618. The confirmed exploit needs a predictable per-request limit to hit, so removing the limit removes the trigger. The cost: one request can now consume all available memory. A genuine allocation failure can also still reach the vulnerable code.
• Low post_max_size shrinks the request bodies an attacker can use. PHP suggests 8M where uploads are not needed and 32M where they are, with upload_max_filesize kept below it.
• session.upload_progress.enabled = off targets CVE-2026-109619. It blocks the confirmed session paths but not other variants. It does nothing at all for CVE-2026-109618.
• On PHP 8.5 only, fatal_error_backtraces = off blocks the confirmed fatal-backtrace path for CVE-2026-109619, and nothing else.

Because removing the memory limit opens a denial-of-service risk, PHP recommends a container or cgroup memory limit as the availability boundary, ideally one that kills only the affected worker. It specifically warns against RLIMIT_AS or RLIMIT_DATA, since those make allocations fail at the limit, which is exactly the condition the bug needs. Enforcing request-size, field-count and multipart-part limits at your reverse proxy helps too.

A web application firewall is a reasonable stopgap, not a fix. Cloudflare deployed rules before the patch shipped and reported attackers changing payloads to get around them. If your site already sits behind Cloudflare, those rules help. They do not replace the update.

The third issue: CVE-2026-115560

Two days after the release, PHP disclosed one more possible code-execution path. It needs an unusual setup: the application sets session.serialize_handler to php_binary or php_serialize in configuration, then switches it to php at runtime with ini_set() before calling session_start(), with session upload progress still enabled. The fixed releases above do not cover it. PHP’s advice is to keep session.upload_progress.enabled=0 until the next regular bugfix release. If you already set it for CVE-2026-109619, leave it in place and you are done.

After patching: assume someone tried

Amazon reported a honeypot hit against these flaws on 17 October, four days before disclosure, and exploitation attempts climbed fast once details were public. Most reports so far describe scanning and failed attempts rather than confirmed break-ins, but “nobody has reported it” is not evidence about your server. If you ran an unpatched, internet-facing PHP site during that window, do three things.

• Rotate secrets a PHP worker could have held in memory: database passwords, API keys, payment provider keys, SMTP credentials and, for WordPress, the salts in wp-config.php (which logs every user out, so warn people first).
• Read your logs for the patterns researchers flagged: unusual multipart requests followed by memory-limit errors or bursts of HTTP 500 responses, oddly large PHPSESSID cookies in responses, and PHP workers restarting with no explanation.
• Look for anything new and executable in your web roots and upload directories, especially PHP files that no deploy put there.

Troubleshooting

“php -v still shows the old version.” Either it is a distro backport (check the changelog for the CVE), or you are looking at the CLI binary while a different build serves the site. Check the FPM binary directly and the container, not the host.

“The site throws 500 errors after the update.” The fix changes how PHP handles running out of memory. Crossing memory_limit now raises a soft error at a safe point, and a second, higher threshold ends the process outright, without running shutdown handlers. A plugin or import job that used to squeak by can now fail differently. Check the error log, then review memory_limit and post_max_size against what the application genuinely needs.

“Cleanup or logging code stopped running on big jobs.” Same cause. If anything important lives in a shutdown function, it will not run when the hard limit is hit. Fix the memory usage rather than raising limits blindly.

“The package manager says there is nothing to update.” You are probably on a repository that has not shipped the fix, an end-of-life distro release, or an end-of-life PHP branch with no vendor backport. That is a migration problem, not a patching problem.

“I set the mitigation but the old value still shows.” Look for overrides in other pool files, per-directory configuration or application code, and make sure you used the admin variants and restarted the workers.

Common mistakes

• Updating WordPress, plugins or the framework and calling it done. None of that touches the PHP runtime.
• Trusting the version number in either direction, and declaring a backported package “vulnerable” or a stale process “patched.”
• Upgrading the package and never restarting FPM, Apache or long-running workers.
• Treating session.upload_progress.enabled=0 as the fix. It was circulated as a workaround before the release and only covers part of one CVE.
• Setting mitigations with ini_set(), .htaccess overrides that apps can change, or .user.ini, which take effect too late.
• Leaving memory_limit=-1 in place long after patching, with no cgroup boundary behind it.
• Forgetting the boxes nobody owns: the staging server, the old client site on PHP 7.4, the forgotten container in a side project.

Best practices

• Keep an inventory of every PHP install, its source (distro, panel, container, source build) and its branch. This week shows why.
• Verify fixes by CVE in the package changelog, and verify the running process by start time.
• Patch mod_php servers first, then public FPM sites, then internal tools.
• Once patched, restore a sane memory_limit, but keep session upload progress off until PHP’s next regular release covers CVE-2026-115560.
• Move servers off mod_php to PHP-FPM where you can. It would not have saved you here, but it narrowed the easiest path.
• Stay on a supported PHP branch, or pay for a vendor that backports. An unsupported branch turns every emergency into a migration.

FAQ

What is the PHP2Shell vulnerability?

It is the nickname for two memory-safety flaws in the PHP engine, CVE-2026-109618 and CVE-2026-109619, both rated 9.2 on CVSS v4.0. They can allow unauthenticated remote code execution on servers that run PHP for web requests.

Which PHP versions fix PHP2Shell?

PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35 from php.net. Distribution and vendor packages may carry the fix under different, older-looking version numbers, so check the package changelog for the CVE.

Is my WordPress site affected?

If it runs on an unpatched PHP build, yes, regardless of how up to date WordPress and its plugins are. The flaw is in PHP itself, so the fix comes from your server, host or container image.

Is disabling session.upload_progress enough?

No. It blocks the confirmed session paths for one of the two bugs and covers the narrow third issue, CVE-2026-115560. It does nothing for CVE-2026-109618. Only an update is a complete fix.

I’m on PHP 7.4 or 8.1. What now?

php.net has no release for you, but PHP published source backports for 7.4, 8.0 and 8.1, and several vendors and distributions shipped backported packages. Use one of those immediately, then plan the upgrade to a supported branch.

Do I need to restart anything after updating PHP?

Yes. Restart every PHP-FPM pool, Apache if you use mod_php, LiteSpeed, and long-running PHP workers. Until they restart, they keep running the vulnerable code already loaded in memory.

Should I rotate passwords and keys?

If an internet-facing server ran an unpatched build after details went public, it is a sensible precaution. Researchers showed worker memory could be read remotely, and that memory can hold credentials from other requests.

Conclusion

The PHP2Shell vulnerability lives below your application, so the only real fix is a patched PHP build that is actually running. Everything else in this post is either a way to get there or a way to buy time. If you remember one thing, make it this: verify the fix by CVE in the package, and verify the process by its restart time. The version string will mislead you in both directions.

Need help patching PHP2Shell across your servers?

If you have more PHP servers than certainty, I can help you close this out properly:

• Inventory every PHP install across your VPS, cPanel, DirectAdmin and container hosts, with branch and package source.
• Patch each one through the right channel and prove the fix by CVE and process restart time.
• Apply and later roll back the interim mitigations safely, with a memory boundary in place.
• Rebuild and redeploy PHP container images, including pinned-digest pipelines.
• Review logs for the published indicators and plan secret rotation without breaking production.
• Plan the move off end-of-life PHP branches and off mod_php.

Send me the output of php -v, your package changelog check, and how PHP runs on the server (FPM, mod_php or LiteSpeed), and I’ll tell you where you stand.

The post PHP2Shell Vulnerability: What Happened, How to Patch, and How to Prove It Worked appeared first on John Nessime.

They’re right. The database is fine. That’s exactly why this incident drags on for an hour instead of ten minutes.

Connection pool exhaustion on AWS is rarely a capacity problem. It’s a queueing problem, and the queue forms in one of three places: inside your application’s pool, at the database’s connection limit, or inside a proxy sitting between the two. Each layer has its own limit, its own error message and its own fix. Resizing the instance fixes none of them reliably, and sometimes makes one worse.

Below: how to tell which layer ran out, the SQL and metrics that prove it, and fixes that survive the next deploy. Examples use RDS and Aurora for PostgreSQL and MySQL, plus HikariCP, SQLAlchemy and node-postgres where their defaults matter.

First question: which layer ran out?

Before touching anything, figure out where requests are waiting. The error text usually tells you, if you read it literally instead of pattern-matching on “connection”.

• The application pool is empty. The error comes from your pool library, not the database. HikariCP says the connection is not available and the request timed out. SQLAlchemy raises a QueuePool limit ... reached timeout. The database never saw these requests at all.
• The database refused new connections. The error comes from the server. PostgreSQL returns a FATAL about too many clients or about the remaining connection slots being reserved (the exact wording varies by major version). MySQL returns error 1040, Too many connections.
• The proxy is holding the queue. With RDS Proxy in the path, clients connect fine but wait for a database connection to be borrowed, then time out after the proxy’s borrow timeout.

Then compare what the database is actually holding against its limit. On PostgreSQL:

SHOW max_connections;

SELECT count(*) AS client_connections
FROM pg_stat_activity
WHERE backend_type = 'client backend';

On MySQL, Max_used_connections is the high-water mark since the last restart, which is useful when the spike has already passed:

SHOW VARIABLES LIKE 'max_connections';
SHOW GLOBAL STATUS LIKE 'Threads_connected';
SHOW GLOBAL STATUS LIKE 'Max_used_connections';

If the app is throwing pool timeouts while the server is well under its limit, you’re in the first failure family. If the server is at the limit, it’s the second. That single comparison saves you from the most common wrong move, which is raising max_connections when the database was never the bottleneck.

Pool exhaustion with an idle database: the one that hides

This is the failure that fools people, because every database graph looks healthy while the application is effectively down.

A pool hands out a fixed number of connections per process. A request borrows one, and nobody else can use it until it’s returned. The pool doesn’t care whether the request is running SQL during that time. If your code opens a transaction, runs a query, then calls a payment API, uploads to S3 or sends an email before committing, the connection sits borrowed and idle for the whole round trip.

Now let that third-party API slow down. Each request holds its connection longer, the pool drains, new requests queue for a connection, and they time out. The database sees a handful of sessions doing nothing. CPU drops, because less SQL is running than usual. The dashboard gets greener as the outage gets worse.

So a slow dependency that has nothing to do with the database can exhaust your database pool, and database metrics will never show it.

Prove it with pg_stat_activity

Group sessions by state and by who owns them. The state column is the evidence; idle in transaction means a client opened a transaction and then went quiet without committing.

SELECT state,
       application_name,
       count(*) AS sessions,
       max(now() - xact_start) AS oldest_transaction
FROM pg_stat_activity
WHERE backend_type = 'client backend'
GROUP BY state, application_name
ORDER BY sessions DESC;

Then list the worst offenders. The query column shows the last statement the session ran, not what it’s doing now, which is exactly what you want here: it tells you where in the code the transaction was left open.

SELECT pid,
       application_name,
       client_addr,
       now() - xact_start AS transaction_age,
       left(query, 80) AS last_statement
FROM pg_stat_activity
WHERE state = 'idle in transaction'
ORDER BY xact_start;

On MySQL, open transactions live in information_schema.INNODB_TRX. The thread ID joins back to the process list so you can see the host and user:

SELECT trx_mysql_thread_id, trx_started, trx_state
FROM information_schema.INNODB_TRX
ORDER BY trx_started;

One thing makes all of this much faster: set application_name per service in the connection string. Without it, every session looks the same and you’re guessing from IP addresses that change with every task restart.

Leak or slow holder?

Two different bugs produce the same symptom. A leak is a code path that borrows a connection and never returns it, usually an exception path that skips the close. A slow holder returns the connection eventually but keeps it through non-database work.

The tell is recovery. Leaks never recover without a restart; the pool ratchets down one connection at a time until it’s empty. Slow holders recover when the slow dependency recovers. If restarting the service “fixes” the incident every few days, suspect a leak.

HikariCP has leakDetectionThreshold, which is off by default. Set it to a value longer than your slowest legitimate request, and it logs the stack trace of wherever a connection was borrowed and held past that threshold. That stack trace is usually the whole diagnosis. Other pools have their own hooks; if yours doesn’t, log borrow duration yourself around the checkout.

Fixes that actually hold

1. Stop holding connections across network calls. Do the external call first, then open the transaction, write, and commit. If you need the call’s result inside the transaction, restructure so the transaction is short. This is the real fix; everything else is a guardrail.
2. Put a server-side ceiling on idle transactions. PostgreSQL’s idle_in_transaction_session_timeout terminates a session that sits idle inside a transaction past the limit. The app will see an error, and that’s the point: a loud failure beats a silent pool drain. You can set it in the parameter group or scope it to the application role.
3. Fail fast on checkout. A web request that waits 30 seconds for a connection is worse than one that fails in two. Match the pool’s acquisition timeout to what the caller will tolerate.

ALTER ROLE app_user SET idle_in_transaction_session_timeout = '60s';

The role-level setting applies to new sessions for that role, so existing pooled connections keep the old value until they’re recycled. Pick a limit comfortably above your longest legitimate pause, or you’ll kill batch jobs that were behaving.

MySQL has no direct equivalent. wait_timeout closes idle sessions, but it’s a blunt setting that also affects pooled connections sitting legitimately idle, so tune it alongside your pool’s own idle and max-lifetime settings rather than instead of fixing the code.

Watch the node-postgres default specifically. Its pool connectionTimeoutMillis defaults to 0, which means no timeout: when the pool is empty, requests wait forever. You don’t get an error to alert on; you get hung requests and upstream timeouts that point everywhere except the pool.

Server slots run out while CPU idles

The second family is the one people expect: the database refuses connections. The surprise is that it happens at low CPU, and that the cause is arithmetic, not load.

Pool multiplication

Every pool size is per process. The number the database actually faces is the product of everything that opens a pool:

worst case = pool size
           x worker processes per container
           x containers at the autoscaler's maximum
           x 2 during a rolling deploy (old and new running together)
           + cron jobs, migrations, BI tools, admin sessions

Take an illustrative case: a pool of 10, four worker processes per ECS task, 30 tasks at the scaling maximum. That’s 1,200 potential connections before a deploy briefly doubles it. Nobody chose that number. It emerged from three settings owned by three different people, and it only shows up on the day traffic and a deploy coincide.

Use the autoscaler’s maximum, not today’s task count. Today’s count is what makes the math look safe.

The limit moves when you resize

On RDS and Aurora, the default max_connections in the parameter group is a formula based on the instance class’s memory, not a fixed number. So a rightsizing exercise that drops the instance a size to save money also quietly lowers the connection ceiling. The CPU graph said it was safe. The connection budget didn’t get a vote.

Don’t calculate the value by hand; the memory figure RDS uses excludes what its own processes consume. Run SHOW max_connections on the actual instance, and re-run it after every class change.

Why raising max_connections is the wrong reflex

You can override max_connections in a custom parameter group, and sometimes a modest bump is fine. But it treats the symptom. Each PostgreSQL connection is a separate backend process with its own memory, and sorts or hashes inside it can each use up to work_mem. Thousands of connections all trying to work at once contend for the same CPU cores, and throughput often drops as concurrency rises past what the hardware can run in parallel.

Also keep in mind that the last few slots are reserved for administrative roles. The application starts failing before the counter reaches the limit, and when it does, you may struggle to get your own session in to diagnose it. Keep headroom for humans.

Budget connections on purpose

1. Read the real max_connections from the instance and subtract a margin for admin sessions, monitoring agents and migrations.
2. List every consumer: each service, its worker count, its scaling maximum, and its batch jobs.
3. Divide the budget and set each pool’s maximum from that, including the deploy overlap.
4. Cap the autoscaler maximum at something the database can serve. An autoscaler that can add tasks faster than the database can accept connections just converts a traffic spike into a connection storm.
5. If the budget can’t cover the fleet, add a pooler in front instead of more slots.

Smaller pools feel scary, but a connection is only useful while SQL is running. A small pool with short transactions usually beats a large pool with long ones.

Zombie connections

Some slots belong to clients that are gone. A container killed without a clean shutdown, or a Lambda execution environment frozen between invocations, can leave server-side sessions open until something times them out. With Lambda, each concurrent environment holds its own connection, so cap concurrency and put a pooler in front; the serverless API work I do starts exactly there.

For long-lived services, set the pool’s idle timeout and max lifetime below any idle timeout on the network path, so connections are retired by the pool rather than cut silently in the middle.

When the proxy becomes the queue

RDS Proxy is the managed answer to pool multiplication. Many client connections share a smaller set of database connections, and the proxy hands a database connection to a client per transaction. When that works, the server-side limit stops being your problem.

It stops working in two ways. First, pinning: when a session changes state the proxy can’t safely share, such as setting a variable on PostgreSQL, the proxy ties that client to one database connection until the session ends. Enough pinned sessions and the proxy behaves like no proxy at all. Second, the borrow queue: when every database connection the proxy is allowed to open is busy, clients wait up to ConnectionBorrowTimeout (120 seconds by default) and then fail. Your application sees slow requests, not refused connections, and the database graphs look calm again.

The proxy’s own CloudWatch metrics tell you which it is. DatabaseConnectionsCurrentlySessionPinned shows pinning. DatabaseConnectionsCurrentlyBorrowed against DatabaseConnections shows how close the proxy is to its share of the database. And MaxConnectionsPercent on the target group caps that share as a percentage of max_connections, so a conservative setting there creates the borrow queue by itself.

The proxy is billed on the capacity of the database behind it (per vCPU-hour for provisioned instances, per ACU-hour for Aurora Serverless), so run your current rates through something like the AWS Cost Estimator before assuming it’s free relief. PgBouncer in transaction mode on a small EC2 instance is the self-managed alternative: cheaper to run, but it’s one more thing to patch, monitor and make highly available.

Metrics that warn you before connection pool exhaustion

Database CPU is the wrong alarm for this problem. By the time connection pressure shows up there, if it ever does, users already noticed. The signals that lead are on the client side:

• Threads waiting for a connection. HikariCP exposes pending threads, active and idle counts, and acquisition time through its Micrometer integration. Sustained pending threads above zero is the earliest reliable warning you get.
• Connection acquisition time. Healthy pools hand out connections almost instantly. A rising acquisition time is the queue forming, long before timeouts start.
• Server connections as a ratio of the limit. Alarm on CloudWatch DatabaseConnections relative to the instance’s real max_connections, not an absolute number that goes stale after a resize.
• Oldest open transaction. A scheduled query against pg_stat_activity exporting the age of the oldest xact_start catches slow holders and leaks early.

Where these land matters less than having them next to each other. Grafana with a Prometheus scrape of the application’s pool metrics works well, as do Datadog or New Relic APM, which can show pool wait time inside the request trace. Sentry is useful for grouping the pool-timeout exceptions so you see the first one, not the thousandth. The goal is one screen where app pool state and database connection counts share a time axis. If you’d rather not build that, it’s the kind of thing covered under AWS monitoring and alerting.

Troubleshooting by symptom

• Pool timeouts, database far below its limit, many idle-in-transaction sessions: slow holder or leak. Find the last statement in pg_stat_activity and trace the code path that follows it.
• Pool timeouts, few database sessions at all: the pool is too small for the worker count, or connections are borrowed and held without a transaction open. Check borrow duration, not the database.
• Refused connections only during deploys: the old and new versions overlap. Budget for the overlap or slow the rollout.
• Refused connections right after an instance class change: the formula-derived limit dropped. Check SHOW max_connections.
• Requests hang with no error at all: a pool with no acquisition timeout, such as node-postgres at its default. Set one.
• Everything was fixed by a restart, then came back days later: a leak. Turn on leak detection and wait for the stack trace.
• RDS Proxy in the path, slow requests, calm database: check pinned and borrowed connection metrics before anything else.

Common mistakes

• Scaling the instance up because “it’s the database”, when the database was idle the whole time.
• Raising max_connections to absorb pool multiplication, then losing throughput to contention.
• Making the pool bigger to fix pool timeouts caused by long transactions. It buys minutes and moves the failure to the server limit.
• Sizing pools from today’s task count instead of the autoscaler’s maximum.
• Adding RDS Proxy without checking pinning, and ending up with a paid queue instead of a pool.

Best practices

• Keep transactions short and never hold a connection across an external call.
• Write the connection budget down, with every consumer, and revisit it when scaling limits or instance classes change.
• Set an acquisition timeout that matches what callers will tolerate, and alert on pending threads, not on timeouts.
• Use idle_in_transaction_session_timeout on PostgreSQL as a safety net, scoped to application roles.

FAQ

What causes connection pool exhaustion when database CPU is low?

Usually connections held while no SQL runs: long transactions, external API calls inside a transaction, or leaked connections that were never returned. The pool is empty, but the database is doing very little.

Should I increase max_connections on RDS?

Only after confirming the server limit is actually the one being hit, and only modestly. If the cause is pool multiplication, budget the pools or add a pooler. More connections means more memory per backend and more contention, not more throughput.

How big should my connection pool be?

Smaller than most people set it. Start from the connection budget divided across every process at maximum scale, then load test. A pool large enough to cover every thread usually just moves the queue from the application to the database.

Does RDS Proxy fix pool exhaustion?

It fixes server-side slot exhaustion from many clients, as long as sessions aren’t pinned. It does nothing for an application pool drained by long transactions, because those transactions hold the proxy’s database connections just as firmly.

How do I find leaked connections in PostgreSQL?

Look for sessions in pg_stat_activity that stay idle or idle in transaction for far longer than any request should, grouped by application_name. Then turn on your pool’s leak detection to get the stack trace of where the connection was borrowed.

Does this happen on self-hosted databases too?

Exactly the same way. On PostgreSQL running on a VPS from providers like Hetzner, Contabo or InterServer, the pool math and idle transactions behave identically; the difference is that you run PgBouncer yourself instead of paying for RDS Proxy.

Conclusion

Connection pool exhaustion on AWS almost never means the database is too small. It means a queue formed somewhere, and the fix depends entirely on where. Read the error literally, compare the database’s connection count against its real limit, and within minutes you’ll know if you’re dealing with a drained application pool, a server limit reached by arithmetic, or a proxy holding the line.

If you remember one thing: a connection costs you for as long as it’s borrowed, not for as long as SQL runs. Shorten the borrow and most of these incidents stop happening.

Need help tracking down a connection problem on AWS?

This is work I do for teams running RDS, Aurora or self-managed PostgreSQL and MySQL. Typical engagements look like this:

• Working out which layer is exhausting during an incident, from pool logs, pg_stat_activity and CloudWatch.
• Building a written connection budget across services, workers, autoscaling limits and deploy overlap.
• Finding leaks and long-held connections in application code, with leak detection and transaction timeouts set sensibly.
• Evaluating RDS Proxy or PgBouncer for your workload, including pinning and cost.
• Dashboards and alerts that put application pool state and database connections on the same screen.
• Checking what an instance resize will do to your connection limit before you make it.

Send me the pool timeout from your logs and the output of the pg_stat_activity query above, and I’ll tell you which layer ran out.

The post Plenty of CPU, Zero Connections: Debugging Connection Pool Exhaustion on AWS appeared first on John Nessime.

The rule was fine. The query was fine. The evaluation ran on schedule and the state transition happened exactly when it should have. What failed was everything after that, and none of it made a sound.

That is the failure mode worth designing around. Rule evaluation and notification delivery are two different systems, and a red rule only proves the first one works. Grafana alert notifications run against SMTP servers, Slack’s API and Telegram’s bot endpoint, all of which fail in their own quiet ways. A rejected login, a revoked webhook, a message over a length limit: each produces a log line and nothing else. The alert list still shows the rule firing, so everything looks healthy from the one place you’d think to check.

This post covers the practical setup for the three channels most small and mid-sized teams actually use: email, Slack and Telegram. It covers what breaks in each of them, how to route alerts so the right ones interrupt people and the rest don’t, how to provision the whole thing as files instead of clicking through the UI, and how to detect the case where delivery has stopped entirely.

Where a firing alert actually goes

Before touching any integration, it helps to know how many hops sit between “the rule fired” and “my phone buzzed”.

• The rule evaluates on its own schedule and produces alert instances, each carrying labels.
• Those instances go to an Alertmanager. Grafana ships its own and can also use an external Prometheus Alertmanager.
• The Alertmanager applies mute timings, groups instances, and walks the notification policy tree for a match.
• The matched policy names a contact point, which holds one or more integrations.
• Each integration makes its own outbound call to a third-party service.

Steps one and two are visible in the Grafana UI. Steps three through five are where things go wrong, and they are visible only in logs. That asymmetry is the whole problem.

One thing worth knowing early: a contact point can hold several integrations at once, so a single contact point named critical can fire email, Slack and Telegram together. Building one contact point per channel instead makes routing much harder than it needs to be.

Email: the channel everyone assumes works

Email is the one people configure first and verify least. The catch is that the email contact point holds only the recipient addresses. The actual SMTP credentials live in grafana.ini, so a perfectly valid contact point can sit there doing nothing because the server underneath was never configured.

[smtp]
enabled = true
host = smtp.example.com:587
user = alerts@example.com
# Wrap the password in triple quotes if it contains # or ;
password = """your-smtp-password"""
from_address = alerts@example.com
from_name = Grafana Alerts
skip_verify = false
# Default is OpportunisticStartTLS
startTLS_policy = MandatoryStartTLS

Several details in that block matter more than they look.

• The port is part of host. There is no separate port setting. Port 587 with STARTTLS is the normal choice; 465 is implicit TLS and behaves differently, which is usually why one fails cleanly and the other hangs.
• startTLS_policy decides how hard Grafana insists on encryption. The default is opportunistic: it upgrades the connection if the server offers it and quietly continues in the clear if not. On a relay you control, mandatory turns a silent downgrade into a visible failure.
• from_address has to be something the relay accepts. Managed relays reject mail from unverified domains after the connection succeeds, which is why the Grafana side looks fine.
• Leave skip_verify false. It exists for internal relays with self-signed certificates. Turning it on to clear an error removes the check that was telling you the certificate is wrong.

Changes to this file need a Grafana restart. Nothing in the UI reloads SMTP settings.

Deliverability is a separate problem from delivery

Sending directly from a VPS is the fastest way to get alerts filed as spam. A fresh IP range from a provider like Contabo or InterServer has no sending reputation, and a mail server that emits one message a week during incidents never builds one.

Point Grafana at a transactional relay instead. Amazon SES, Postmark, Mailgun and Brevo all do the same job at this volume. Whichever you pick, publish SPF and DKIM records for the sending domain and set a DMARC policy, then send a real alert to a Gmail address and a Microsoft 365 address and check the spam folder. That is the only test that tells you anything.

Slack: webhook or bot token

Grafana supports two ways of posting to Slack, and the choice has consequences beyond setup time.

Incoming webhooks are the quick path. Slack generates a URL under hooks.slack.com/services/ that is bound to a single channel, and Grafana posts to it. There’s no OAuth flow and no scopes to reason about. The trade-off is that the URL is the credential and it maps to exactly one channel, so a team with six alert channels ends up managing six secrets with no central view of them.

A Slack app with a bot token takes longer. You create the app, add the chat:write.public scope so it can post to public channels without being invited to each one, then copy the bot user OAuth token, which starts with xoxb-. In Grafana you supply that token plus the channel ID as the recipient. One credential covers every channel, permissions are visible in one place, and revoking it kills all posting at once.

Webhooks win when you have one or two channels and want to be done in five minutes. The bot token wins as soon as channels start multiplying or someone asks what the integration is permitted to do. I reach for the webhook on small setups and regret it around the fourth channel.

Two constraints either way. A Slack integration targets one channel, so several channels means several integrations. And notification templates control the message title and body text, not its visual structure, so rich block layouts aren’t on the table.

Telegram: cheap paging, one nasty edge

Telegram is genuinely useful for out-of-hours alerts because it pushes to a phone without anyone paying for a paging product. The setup is two values: a bot API token and a chat ID.

1. Message BotFather and send /newbot. The username has to end in bot or _bot.
2. Copy the HTTP API token it returns. That’s the bot API token Grafana wants.
3. Create a group for alerts and add the bot to it. A bot cannot message a chat it has never seen.
4. Send a slash command in the group, such as /start@your_bot_name. An explicit command is the reliable way to make the group visible to the bot.
5. Fetch the chat ID from the Telegram API.

curl -s "https://api.telegram.org/bot<YOUR_BOT_TOKEN>/getUpdates" | jq '.result[].message.chat'

The id field in that output is what goes in Grafana’s Chat ID field. Group chat IDs are negative. Copy the minus sign; leaving it off is the single most common reason a Telegram contact point tests as failing.

Now the edge case, and it’s the reason Telegram belongs in a post about invisible failures. Telegram caps messages at 4096 UTF-8 characters. If you’re using a parse mode other than none, truncating at that boundary can leave the markup unbalanced, and Telegram rejects the whole message as malformed. You don’t get a shortened alert. You get nothing.

This is exactly backwards, because the message most likely to blow the limit is a large grouped alert during a real outage. The channel works perfectly for months of single alerts, then goes quiet the first time forty instances fire together. Three ways to deal with it, in the order I’d try them:

• Write a deliberately short notification template for Telegram: alert name, severity, instance count and a dashboard link. Treat it as a pager, not a report.
• Narrow group_by on the policy that feeds Telegram so a single notification covers fewer instances.
• Set the parse mode to none, which makes truncation harmless at the cost of formatting.

Routing: split by severity, not by tool

The most common design mistake is a policy tree organised around channels: a Slack policy, an email policy, a Telegram policy. That gives you no way to express the thing you actually care about, which is that some alerts should wake someone up and most should not.

Put severity on the alert rule as a label and route on that instead. Contact points then become bundles of channels chosen to match urgency.

• severity=critical goes to a contact point holding Telegram and Slack. Something is broken and someone needs to look now.
• severity=warning goes to Slack only, in a channel people read during working hours.
• Everything else falls through to the default policy and lands in email, where it accumulates without interrupting anyone.

Four timing settings control how noisy this is, and each one does something specific:

• group_by decides what counts as the same notification. Grouping by alertname and a service label means one cascading failure produces one message instead of forty.
• group_wait (default 30 seconds) is how long to hold the first notification for a new group, so related instances that fire a few seconds apart arrive together.
• group_interval (default 5 minutes) is the minimum gap before a group that has already notified sends again because new instances joined it.
• repeat_interval (default 4 hours) is how often an unresolved alert re-notifies with nothing new to say.

The temptation is to drop repeat_interval to fifteen minutes so nothing gets forgotten. Resist it. Teams that do this stop reading the channel within a month, which converts a routing problem into a much worse cultural one. Longer intervals plus a real escalation path beats short intervals every time.

Grafana also offers a simpler path: pick a contact point directly on the alert rule form and skip label matching entirely. Reasonable when a rule has one obvious destination. The trade-off is that routing knowledge moves into individual rules, so changing where a class of alerts goes means editing every rule instead of one policy.

Provision it as files, not clicks

Contact points built through the UI are invisible to review, easy to delete by accident, and impossible to recreate after a bad restore. If Grafana alert notifications matter enough to build, they matter enough to keep in Git. Grafana reads alerting configuration from files in the provisioning/alerting directory under its provisioning path at startup.

Before writing any YAML by hand, build one contact point in the UI, test it, then use the export option on the contact points list to dump it as YAML. That gives you the exact setting keys for that integration type instead of the ones you half-remember from a blog post. Setting names are not consistent between integration types, and a guessed key produces a config that loads without error and sends nothing.

apiVersion: 1

contactPoints:
  - orgId: 1
    name: critical-oncall
    receivers:
      # Telegram receiver goes here. Export it from the UI first:
      # the setting key names differ from what you would guess.
      - uid: critical-slack
        type: slack
        settings:
          url: https://hooks.slack.com/services/REPLACE/WITH/YOURS
        disableResolveMessage: false

  - orgId: 1
    name: warnings-slack
    receivers:
      - uid: warnings-slack
        type: slack
        settings:
          url: https://hooks.slack.com/services/REPLACE/WITH/YOURS

  - orgId: 1
    name: email-digest
    receivers:
      - uid: email-digest
        type: email
        settings:
          addresses: ops@example.com
          singleEmail: false

And the policy tree that uses them:

apiVersion: 1

policies:
  - orgId: 1
    receiver: email-digest
    group_by: [grafana_folder, alertname]
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - receiver: critical-oncall
        matchers:
          - severity = critical
        group_wait: 10s
        repeat_interval: 1h
      - receiver: warnings-slack
        matchers:
          - severity = warning
        repeat_interval: 12h

Two things to know about provisioned resources. By default they become read-only in the UI, which is the point but surprises people the first time. And a contact point that a policy references cannot be deleted until the policy stops referencing it, so removals have to happen in the right order.

Keep tokens out of the files themselves. Grafana supports environment variable substitution in provisioning files, which means the YAML can live in Git and the secrets can come from your existing secret store.

Detecting the silence

Everything above still leaves the original problem unsolved. If the Slack app gets revoked or the SMTP password rotates, notifications stop and Grafana does not tell you.

The fix is a heartbeat alert, and it’s worth the twenty minutes. Create a rule whose condition is always true, give it a low severity label, route it to each contact point with a short repeat_interval, and point the resulting notification at an external dead-man’s-switch service. Healthchecks.io, Cronitor and Better Stack all offer this, and Healthchecks.io can be self-hosted if you’d rather not add a dependency. The service expects a signal on a schedule and alerts you when it stops arriving.

The logic is inverted on purpose. You aren’t monitoring infrastructure here, you’re monitoring the path that tells you about your infrastructure, and it has to be watched by something that isn’t itself.

As a second layer, Grafana exposes its own internal metrics on a Prometheus-compatible endpoint, including counters for notification attempts and failures. If you’re already scraping Grafana, alert on that failure counter increasing. It won’t catch a credential that works but delivers to a dead mailbox, which is why the heartbeat comes first.

Troubleshooting: the rule fires, nothing arrives

Work down this list in order. Each step rules out one hop, and stopping early is how people spend an afternoon on the wrong layer.

1. Test the contact point in isolation. The Test button sends a notification without touching rules or policies. If it fails, the problem is credentials or network and nothing else matters yet.
2. Watch the Grafana logs during the test. On a systemd install, run journalctl -u grafana-server -f while you click Test. SMTP, Slack and Telegram delivery errors all surface here, usually specific enough to fix directly.
3. Compare the instance’s labels against your matchers. A policy matching severity = critical never sees severity = Critical. Matchers are exact and case-sensitive. Read the firing instance’s real labels, not the ones you think the rule sets.
4. Look for a policy earlier in the tree that swallowed it. Evaluation stops at the first match unless the policy continues. A broad route near the top silently absorbs everything below it.
5. Check active silences and mute timings. A silence created during last month’s maintenance with a generous matcher is the classic. So is a mute timing in the wrong timezone.
6. Check whether you’re inside repeat_interval. A red alert now does not mean a notification is due now. This looks identical to a broken integration and isn’t.
7. For email, read the relay’s own logs. Grafana handing the message off is not delivery. Bounces and spam filing happen after Grafana’s involvement ends.

Common mistakes

• Configuring the email contact point and never touching grafana.ini. The contact point is only a recipient list. Without the SMTP block, it has nothing to send with.
• Routing by channel instead of severity. It feels tidy and it makes urgency impossible to express.
• Dropping the minus sign from a Telegram group chat ID. Group IDs are negative and the field takes the value literally.
• Sending verbose templates to Telegram. The 4096-character limit turns your biggest incident into your quietest one.
• Testing with the Test button and calling it done. Test bypasses the policy tree entirely. It proves the credential works, not that a real alert would ever reach it.
• Leaving resolved notifications on for every channel. Resolution messages in a pager channel double the volume and train people to ignore it.
• Storing webhook URLs and bot tokens in the UI only. They exist in one place, are invisible to review, and disappear with the database.

Best practices for Grafana alert notifications

• Set a consistent severity label on every rule from day one. Retrofitting labels across dozens of rules is miserable work.
• Use one contact point per urgency level, holding as many integrations as that level needs.
• Put a dashboard or runbook link in an annotation your templates render. An alert that says what broke but not where to look costs the responder five minutes every time.
• Write channel-specific templates: short for Telegram, richer for email, in between for Slack.
• Provision contact points and policies from files, with secrets injected from the environment, and run a heartbeat through every channel.
• Re-test every integration after credential rotation, Slack workspace changes or a Grafana upgrade. Those are the three events that quietly break delivery.

FAQ

Can one contact point send to email, Slack and Telegram at the same time?

Yes. A contact point holds multiple integrations and every one of them fires when that contact point is selected. This is the right way to build an escalation tier: one contact point per severity, with the channel mix chosen to match how urgently that severity needs attention.

Why does the Test button work but real alerts never arrive?

Test sends straight to the contact point and skips the policy tree. If it works and real alerts don’t, the break is in routing: matchers that don’t match, a broader policy earlier in the tree absorbing the alert, an active silence, or a mute timing. Open the firing instance, read its real labels, and compare them character by character against your matchers.

Do I need Grafana Cloud for any of this?

No. Contact points, notification policies, templates and all three integrations ship in the open source build. Grafana Cloud handles hosting and adds on-call tooling on top, which matters if you need rotations and escalation chains. For a team routing alerts to three channels, a self-hosted instance on a modest VPS does everything described here.

Should I use a Slack webhook or a bot token?

Webhook for one or two channels where speed of setup matters. Bot token once you have several channels, want one revocable credential, or need to see in one place what the integration is allowed to do. Both deliver identically; the difference is entirely in credential management.

Why did my Telegram alerts stop during a big outage?

Almost certainly the 4096-character limit. A large grouped notification exceeded it, truncation broke the message markup, and Telegram rejected the send outright. Shorten the template for that channel, group more narrowly, or drop the parse mode. It’s the failure mode most worth guarding against, because it only shows up when you need the channel most.

Can I keep using the Prometheus Alertmanager I already run?

Yes. Grafana can send to an external Alertmanager instead of its built-in one, and the contact point selector at the top of the alerting pages lets you choose which one you’re configuring. If you already have a mature Alertmanager routing config, keeping it and pointing Grafana at it is usually less work than migrating the tree.

The one thing to remember

A firing rule proves evaluation works. It proves nothing about whether anyone was told. Grafana alert notifications cross three or four systems after the rule turns red, and every one of them fails quietly by default.

So build the channels carefully, route by severity rather than by tool, then run a heartbeat through every path and let something outside Grafana watch it. The alternative is finding out during the incident, which is the one time you have no attention to spare.

Need alerting that reaches people?

Most of the Grafana setups I get asked to look at have good dashboards and alerting nobody trusts. Work I take on here:

• Fixing email delivery end to end: SMTP config, relay selection, SPF, DKIM and DMARC so alerts stop landing in spam.
• Setting up Slack and Telegram contact points properly, including bot apps, scoped tokens and channel-appropriate templates.
• Designing a policy tree around severity labels, with grouping and timing tuned so people keep reading the channel.
• Converting click-built alerting into provisioned YAML under version control, with secrets kept out of Git.
• Adding heartbeat monitoring so a broken notification path announces itself instead of waiting for an outage.
• Auditing alert rules for missing labels, missing runbook links, and rules that fire constantly and are therefore ignored.

Send me your notification policy export and the output of journalctl -u grafana-server from a period when an alert should have gone out, and I’ll tell you which hop is dropping it.

The post Grafana Alert Notifications That Actually Arrive: Email, Slack and Telegram appeared first on John Nessime.

That is the failure mode nobody catches during a review, and it is why most AWS accounts have dozens of alarms and about three anyone trusts. The rest either fire constantly and get muted, or never fire at all and get mistaken for coverage.

This post is about building useful CloudWatch alarms: which signals deserve a page, which settings decide whether an alarm fires correctly, how to collapse alarm storms without going blind, and how to prove the notification path works before you need it.

The test an alarm has to pass before it exists

Before the threshold, before the metric, one question: if this fires at 3am, what does the person on call do?

If the honest answer is “look at it and go back to bed”, it is not an alarm. It is a dashboard panel. That single filter removes most of the CPU, memory and disk alarms people inherit. High CPU is not an incident. High CPU while latency climbs and the queue backs up is an incident. Every alarm that fires without requiring action trains the on-call to ignore the channel, and that training is permanent.

• Page on symptoms a customer can feel: error rate, latency at a high percentile, queue age, failed jobs, expiring credentials.
• Ticket on causes that bite next week: disk trending toward full, approaching a service quota, a renewal that is not happening.
• Graph only on everything else. Resource utilisation is context during an incident, not the incident.

Encode that split in the alarm’s action, not in your head. A page goes to an SNS topic wired to PagerDuty or Opsgenie. A ticket goes to a different topic that opens a Jira issue or a Systems Manager OpsItem. Same engine, different consequences.

The alarm that goes quiet: missing data is the invisible failure

Every metric alarm has a TreatMissingData setting, and if you never touched it, it is missing. That default means the alarm ignores gaps entirely when deciding state. It keeps whatever state it had, or sits in INSUFFICIENT_DATA if there is nothing at all. For a metric that always reports, that is a silent hole: the instance dies, the metric stops, and the alarm does not go red. It just stops having an opinion.

• breaching for metrics that report continuously. If the data stops, something is wrong. Heartbeats, health checks, CPU on a box that should always be up.
• notBreaching for metrics that only exist when something bad happens. Error counts, throttled requests, dead letter queue depth. No data means no errors.
• ignore keeps the current state through gaps, so an alarm stays red until genuinely resolved rather than flapping to OK on a hole in the data.
• missing is the default, and defensible only when you decided it was right.

There is a second-order effect that surprises people. When an alarm evaluates, CloudWatch pulls a wider range of data points than the evaluation periods you configured. AWS calls that the evaluation range. If some points in that range are missing but enough real ones exist, CloudWatch evaluates against the real points and your missing-data setting never applies to the gaps.

The consequence: an alarm set to eight breaching points out of ten can fire on one breaching point followed by nine gaps, because the older breach is still inside the evaluation range and everything newer is absent. Teams hit this with load balancer 5XX alarms constantly, and it looks exactly like an unexplained false positive.

Setting notBreaching on sparse error metrics is usually the fix. Filling gaps with metric math is the other, and it is more explicit:

# Treat absent error data points as zero rather than as gaps.
# m1 is the raw error metric; e1 is what the alarm evaluates.
e1 = FILL(m1, 0)

One exception: alarms on metrics in the AWS/DynamoDB namespace always ignore missing data regardless of what you set. If you need a DynamoDB alarm that reacts to silence, build the fill into metric math.

Then find the ones already stuck. This is the fastest audit you can run on an inherited account:

aws cloudwatch describe-alarms 
  --state-value INSUFFICIENT_DATA 
  --query 'MetricAlarms[].[AlarmName,MetricName,StateReason]' 
  --output table

Anything on that list older than a deploy cycle is either watching something that no longer exists, or watching something that stopped reporting. Both deserve thirty seconds.

The alarm that cries wolf: periods, data points and windows

Three settings decide whether an alarm is jumpy: Period (how long each data point covers), EvaluationPeriods (how many recent points to look at), and DatapointsToAlarm (how many of those must breach).

M out of N beats consecutive breaches

Most people set evaluation periods to 3 and leave data points to alarm unset, which makes it 3 as well. That demands three consecutive breaching points. Real degradation is rarely that tidy; it flickers, one clean point resets your patience, and the alarm never fires.

Set them separately. Three out of five catches intermittent problems while still filtering single-point spikes. The breaching points do not need to be consecutive, only inside the last N.

aws cloudwatch put-metric-alarm 
  --alarm-name "api-5xx-elevated" 
  --namespace "AWS/ApplicationELB" 
  --metric-name HTTPCode_ELB_5XX_Count 
  --dimensions Name=LoadBalancer,Value=app/my-alb/0123456789abcdef 
  --statistic Sum 
  --period 60 
  --evaluation-periods 5 
  --datapoints-to-alarm 3 
  --threshold 25 
  --comparison-operator GreaterThanThreshold 
  --treat-missing-data notBreaching 
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:oncall-page

Read that as: over the last five minutes, if three or more individual minutes each saw more than 25 backend 5XX responses, page. Minutes with no data count as fine, because a load balancer with no errors publishes nothing.

Percentiles need a sample-count decision

Average latency hides everything interesting, so alarm on p95 or p99 using the extended statistic rather than --statistic Average.

The catch is low traffic. A p99 computed from four requests is noise, and by default CloudWatch still evaluates it. The EvaluateLowSampleCountPercentile setting controls that: set it to ignore and the alarm holds its current state when the sample count is too small to mean anything. On an overnight window or a quiet endpoint, that one setting removes a whole class of 4am pages.

Sliding versus wall clock windows

By default the evaluation window slides: it advances a minute at a time and is not aligned to the clock. That is what you want for latency and errors, because you want to know now.

CloudWatch also supports a wall clock window aligned to real boundaries in a time zone you pick: top of the hour, midnight, start of the calendar week. That is the right choice for anything measured per calendar period, like a daily batch job or a monthly threshold, because the alarm considers a completed period rather than a rolling one. Note that the wider evaluation range described earlier applies only to sliding windows, so wall clock alarms are easier to reason about but have no safety net for late-arriving data.

Alarming on the wrong number

A large share of useless alarms are correctly configured alarms watching a meaningless quantity.

Rates, not counts

An alarm on “more than 50 errors” is wrong twice. During a traffic peak, 50 errors out of half a million requests is nothing. At 3am, 50 errors out of 60 requests is a total outage that never crosses the threshold.

# m1 = Errors (Sum), m2 = Invocations (Sum)
# Guard the zero case so the alarm gets a data point at idle
# instead of a gap it has to interpret.
e1 = IF(m2 > 0, m1 / m2 * 100, 0)

Only the final expression should return data to the alarm; the underlying metrics are inputs. A rate alarm alone is still incomplete, though, because at very low volume one error is 100 percent. Pair it with a minimum-volume condition in a composite alarm, or fold the request count into the expression so tiny denominators cannot trip it.

Quotas and saturation, before they bite

Service quotas are the outage nobody sees coming, because everything is healthy right up until it is not. For services that publish usage metrics, the SERVICE_QUOTA metric math function pulls your current limit from Service Quotas, so you can alarm on percentage consumed instead of a number you hardcoded and forgot:

# m1 is a usage metric under the Usage namespace.
# Alarm on consumption as a share of the live quota.
e1 = m1 / SERVICE_QUOTA(m1) * 100

This is the definitive ticket-not-page alarm. Nobody needs waking for concurrency at 80 percent, but somebody needs to file a limit increase before the deploy that takes it to 100. Same category: queue age rather than queue depth. A deep queue draining fast is fine; a shallow queue whose oldest message is twenty minutes old means your consumers are stuck.

Cutting the storm: composite alarms, suppression and mute rules

One bad deploy in an Auto Scaling group with per-instance alarms produces forty notifications describing one event. A composite alarm evaluates a boolean rule over other alarms. The metric alarms stay exactly as they are and keep their own state, but stop being wired to your paging topic. The composite becomes the only thing that pages.

aws cloudwatch put-composite-alarm 
  --alarm-name "checkout-degraded" 
  --alarm-rule 'ALARM("checkout-latency-p99") AND ALARM("checkout-error-rate")' 
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:oncall-page

Latency alone might be a slow dependency. Errors alone might be a bad client. Both together is a real problem, and that is what earns a page.

The second half is suppression. A composite alarm accepts a suppressor alarm: while that alarm is in ALARM, the composite still evaluates but takes no actions. Point it at a deployment-in-progress alarm and you stop paging for known-bad windows.

aws cloudwatch put-composite-alarm 
  --alarm-name "checkout-degraded" 
  --alarm-rule 'ALARM("checkout-latency-p99") AND ALARM("checkout-error-rate")' 
  --actions-suppressor "deployment-in-progress" 
  --actions-suppressor-wait-period 60 
  --actions-suppressor-extension-period 60 
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:oncall-page

Those two periods are the part people skip and then get burned by. The wait period gives the suppressor time to reach ALARM before the composite decides to fire; the extension period gives the composite time to settle back to OK after the suppressor clears. AWS suggests 60 seconds for each, on the reasoning that metric alarms evaluate once a minute. Leave them at zero and you get paged in the gap between the deploy starting and the deploy alarm noticing.

CloudWatch has since added alarm mute rules, which handle the scheduled case directly. A mute rule targets specific alarms by name and mutes their actions during a one-time or recurring window. The alarms keep evaluating and keep changing state; only the actions are muted. That removes the EventBridge and Lambda contraption teams used to build for maintenance windows. Two things to keep straight: mute rules take precedence over composite action suppression when both apply, and a mute rule targets alarms by name, so a rename in your Terraform or OpenTofu module silently orphans the rule.

Suppression is for known-bad windows you chose. If you are muting an alarm because it is annoying rather than because you scheduled the disruption, the alarm is wrong and muting it just hides that.

Test the path, not the threshold

An alarm you have never seen fire is a hypothesis. The threshold is the easy part. The fragile part is everything downstream: SNS subscriptions that were never confirmed, a Lambda whose role lost a permission, a chat webhook rotated three months ago, a topic policy that quietly rejects CloudWatch.

You can force a state transition without touching the metric:

aws cloudwatch set-alarm-state 
  --alarm-name "api-5xx-elevated" 
  --state-value ALARM 
  --state-reason "Routing test, ticket OPS-1234"

This drives the real actions, so the page really goes out. Tell the on-call first. The forced state is temporary; the next real evaluation overwrites it. Do this for every new alarm before you call it done, and again after any change to the notification stack. On a mixed estate where some workloads sit on a VPS at a provider like Contabo or InterServer alongside your AWS footprint, test each path separately, because a shared SNS topic creates the illusion that one confirmed subscription covers everything.

Then confirm the transition and the action both happened:

aws cloudwatch describe-alarm-history 
  --alarm-name "api-5xx-elevated" 
  --history-item-type Action 
  --max-records 10

What useful CloudWatch alarms cost you

Alarms bill per alarm metric per month, so the mental model that matters is this: a standard metric alarm counts as one, and a metric math alarm counts as one per metric referenced in the expression. An error-rate alarm built from errors and invocations bills as two, not one. Anomaly detection alarms carry their own multiplier because the band is evaluated alongside the metric. High-resolution alarms, with periods of 10, 20 or 30 seconds, are charged at a higher rate than standard ones, which makes them worth it for a handful of latency-critical paths and wasteful everywhere else.

Check current rates on the CloudWatch pricing page rather than trusting a number in a blog post, including this one. And note the bigger cost is usually not the alarm at all. It is the custom metrics feeding it, especially high-cardinality ones published per request or per container.

Troubleshooting an alarm that misbehaves

1. Stuck in INSUFFICIENT_DATA. Either the metric is not being published, or the alarm’s dimensions do not match the metric’s dimensions exactly. Dimension mismatch is the common one and it is silent: an alarm on a dimension set that never existed looks identical to an alarm on a dead resource.
2. Fired but nobody heard. Check describe-alarm-history filtered to Action items. Transition present but action missing points at the SNS topic policy or the subscription. Both present points downstream of SNS.
3. Fires with no visible breach on the graph. Almost always missing data plus the wider evaluation range. The state reason names how many points were breaching and how many were unknown.
4. Flapping between OK and ALARM. The period is too short for the metric’s natural variance, or data points to alarm equals evaluation periods. Widen the window before raising the threshold; raising the threshold loses sensitivity you may want later.
5. Composite alarm never fires. Confirm every child alarm named in the rule exists and is spelled correctly, then confirm no suppressor or mute rule is active. The console shows a mute indicator on affected alarms.

Common mistakes

• Leaving TreatMissingData at the default because you did not know it existed.
• Picking 80 percent CPU because it sounds like a threshold, on a service that normally runs at 75.
• Alarming per instance in an Auto Scaling group instead of on the aggregate a customer experiences.
• Wiring OK actions to the same paging channel, so every recovery is also an interruption.
• Alarm descriptions that restate the alarm name. The description is the only context the responder gets at 3am, so put the runbook link there.
• Anomaly detection on a metric with no stable pattern. The band is only as good as the history it learned from.

Best practices for useful CloudWatch alarms

• Set TreatMissingData explicitly on every alarm, and write down why in the description.
• Separate DatapointsToAlarm from EvaluationPeriods. M out of N is almost always better than N consecutive.
• Observe a metric across a full weekly cycle before fixing its threshold. Weekend traffic is a different distribution.
• Define alarms in code, in whatever you already use: Terraform, OpenTofu, CDK, CloudFormation. Console-created alarms drift and disappear.
• Use anomaly detection for the signal and composite alarms for the logic. An anomalous shape plus real user impact is a page; an anomalous shape alone is a graph.
• Review what fired last quarter. Any alarm that fired and was closed without action is a candidate for deletion. Send it to a Grafana panel instead.

Frequently asked questions

How many CloudWatch alarms should a service have?

Fewer than you think. A handful of symptom alarms that page, plus a set of cause alarms that open tickets, covers most services. The number that matters is not how many exist but how many fired last quarter and led to somebody doing something.

Should I use anomaly detection or static thresholds?

Static thresholds where a real limit exists: a disk that must not fill, a latency budget you promised. Anomaly detection where the metric has a genuine daily or weekly shape and no fixed correct value, like request volume. Anomaly detection needs history before it is useful and struggles when a deploy changes the pattern, so it is a poor fit for anything new.

Why did my alarm fire when the graph looks fine?

Usually missing data. CloudWatch pulls a wider evaluation range than the periods you configured, so an older breaching point can still count while newer points are simply absent. Read the state reason, which spells out how many points were breaching and how many were unknown, then reconsider the missing-data setting.

Can an alarm watch a metric in another AWS account?

Cross-account observability lets a monitoring account view and alarm on metrics from linked source accounts, which is the usual pattern for an AWS Organizations estate. The alternative is running alarms in each account and aggregating at the SNS or paging layer. That is more alarms to manage but keeps the blast radius per account smaller.

How do I stop alarms firing during a deployment?

Two options. An alarm mute rule on a schedule, if your deploys are predictable. Or a composite alarm with a deployment-in-progress alarm as its actions suppressor, if they are not. The suppressor approach suits continuous delivery better because it reacts to the actual deploy rather than a calendar guess.

Do CloudWatch alarms work on logs?

Indirectly. A metric filter on a log group turns matching log lines into a numeric metric, and you alarm on that metric like any other. Metric filters only emit data points when they match, so notBreaching is almost always the right missing-data setting for them.

Is CloudWatch enough, or do I need Datadog or Grafana?

For a single-account AWS workload, CloudWatch alarms plus SNS handle alerting fine, and staying inside AWS avoids another integration to maintain. Third-party platforms earn their keep when you need correlation across non-AWS systems, richer on-call routing, or long metric retention with fast queries. The honest test is whether you are already spending engineering time rebuilding those features on top of CloudWatch.

The one thing to keep

Useful CloudWatch alarms are not the ones with the cleverest thresholds. They are the ones where somebody decided, on purpose, what silence means.

Every alarm in your account is making a claim about the absence of data, whether you configured that claim or inherited it. Go through them, set TreatMissingData deliberately, split data points from evaluation periods, delete the ones nobody acts on, and force a state transition on the ones you keep so you know the page lands. Six alarms that fire correctly beat sixty that nobody reads.

Need a second pair of eyes on your alerting?

I work with teams on AWS monitoring that people actually trust. Typically that looks like:

• Auditing existing CloudWatch alarms and separating the ones that page, the ones that should open tickets, and the ones to delete.
• Fixing missing-data and evaluation settings so alarms stop firing on gaps and stop going silent on dead metrics.
• Building composite alarms, suppression and mute rules so one incident produces one page instead of forty.
• Writing metric math for error rates, queue age and quota headroom instead of raw counts.
• Codifying the whole alarm set in Terraform or OpenTofu so it survives the next person.
• Wiring CloudWatch into Grafana dashboards and testing every notification path end to end.

If you want a concrete starting point, send me the output of describe-alarms for one account and I will tell you which alarms are load-bearing and which are decoration.

The post CloudWatch Alarms That Are Actually Useful (and the Ones to Delete) appeared first on John Nessime.