That is the failure mode worth understanding before you touch anything. A Salesforce AWS PrivateLink setup does not fail loudly. It fails by working. The callout succeeds, the payload arrives, the dashboard is green, and the packets went over the public internet anyway because one DNS record was wrong. Nothing in either console tells you.

This post covers what actually breaks when you wire Salesforce to AWS over private connectivity: the DNS trap above, the throughput ceiling nobody budgets for, the TLS and port constraints, why your sandbox cannot rehearse the change, and how to prove after the fact that traffic really is staying on the AWS network.

What Private Connect actually gives you

Salesforce Private Connect is the product name. AWS PrivateLink is the underlying mechanism. Salesforce runs a managed transit VPC in each supported AWS region, and you attach to it: either your VPC creates an interface endpoint pointing at their endpoint service, or their transit VPC creates an endpoint pointing at yours.

Two things people consistently get wrong here.

First, the direction naming is from the org’s point of view, not yours. Inbound means traffic entering Salesforce, so your AWS workload calling the Salesforce REST or Bulk API. Outbound means Salesforce calling out to something you host. If you are an AWS engineer, “inbound” is the one that feels like outbound to you. Half the confused meetings I have seen on this trace back to that single word.

Second, Private Connect only builds the pipe. It does not move data. You still need Apex callouts, External Services, Platform Events, MuleSoft, Amazon AppFlow, or whatever else was already doing the work. If your integration is broken today, a private path will carry the broken requests privately.

Private connectivity is a network property, not an application feature. Nothing above layer 4 changes. Plan the rollout as a routing change, not an integration project.

Failure family one: DNS that quietly resolves to the public internet

This is the one that produced the ticket at the top, and it is by far the most common way a Salesforce AWS PrivateLink deployment ends up decorative.

For the inbound case, your application calls a hostname like yourorg.my.salesforce.com. That hostname resolves publicly. Creating the interface endpoint does not change that. Unless something inside your VPC overrides the answer, the resolver returns Salesforce’s public addresses and your traffic takes the normal internet path. The endpoint sits there costing you money and carrying nothing.

The override is a Route 53 private hosted zone, associated with the VPC that holds the endpoint, containing a record that points the org hostname at the endpoint. Salesforce’s own walkthrough names the zone my.salesforce.com and puts the org-specific record inside it.

Three things have to line up or the override silently does nothing:

• The zone is associated with the right VPC. A private hosted zone answers only for VPCs it is attached to. Query it from anywhere else and the request falls through to public recursion.
• The VPC has DNS support and DNS hostnames enabled. Both attributes must be on, or the VPC resolver will not serve the zone at all.
• Callers actually use the VPC resolver. Workloads pinned to an external resolver, containers with a hardcoded nameserver line, or on-premises clients arriving over Direct Connect will bypass the zone entirely unless you front it with a Route 53 Resolver inbound endpoint.

There is a sharp edge in the other direction too. If a private hosted zone matches the queried name but contains no record of the right type, Route 53 returns NXDOMAIN rather than falling through to public DNS. So a half-finished zone does not degrade gracefully, it takes the hostname down. Creating the zone and the record in separate change windows is how teams turn a routing improvement into an outage.

Verify from inside the VPC, not from your laptop:

# Run this on an instance in the VPC associated with the private zone.
dig +short yourorg.my.salesforce.com

# You want RFC 1918 addresses from your endpoint subnets.
# Public addresses mean the private zone is not being consulted.

# Confirm which resolver answered:
dig yourorg.my.salesforce.com | grep -i "SERVER:"

Failure family two: the throughput ceiling nobody reads

Private Connect is rate limited, and the default entitlement is small enough to catch people out badly.

Salesforce documents an initial license entitlement of 225 MB of data per hour, measured separately for inbound and outbound, with an add-on ceiling far higher. The allowance does not roll over. It resets on the hour.

Now consider what a nightly Bulk API extract looks like. A few gigabytes pushed in a twenty minute window is entirely normal and completely incompatible with a 225 MB hourly bucket. The two directions behave differently when you hit the wall, which matters more than the number itself:

• Outbound is enforced technically. Exceed it and outbound traffic stops until the counter resets at the top of the next hour. Your Apex callouts start failing. This is a production incident.
• Inbound is enforced contractually. Salesforce documents that you are responsible for monitoring and enforcing the inbound limit yourself. Nothing throttles you at the network layer, but you are outside your agreement, and standard enterprise API limits still apply on top.

The outbound allowance is visible through the Limits API as PrivateConnectOutboundCalloutHourlyLimitMB, which is what makes it monitorable. Poll it, put it on a dashboard next to your other API limit metrics in Grafana or CloudWatch, and alert well before exhaustion rather than after the callouts fail.

If your volume genuinely exceeds the entitlement, the honest options are buying more, reshaping the workload to spread across hours, or accepting that bulk movement stays on a different path and Private Connect covers only your latency-sensitive transactional traffic. That last one is a legitimate architecture, not a failure.

Failure family three: TLS, SNI, and the port you cannot use

Private Connect inbound supports port 443 only. Port 8443 is explicitly not supported because Salesforce uses it for mutual TLS. If your integration depends on client certificate authentication on that port, this design does not cover it, and finding out during cutover is expensive.

The subtler problem is SNI. Salesforce documents that Hyperforce requires callers to include the Server Name Indication extension in the TLS ClientHello so the right certificate can be served, and that clients calling a hostname outside the *.my.salesforce.com or *.sandbox.my.salesforce.com patterns will hit HTTPS errors. Browsers always send SNI. Older HTTP libraries, embedded clients, and some middleware stacks do not.

The reason this bites specifically during a PrivateLink migration is that you are changing what the name resolves to while keeping the name identical. A client that was quietly relying on the public path’s behavior now hits a different frontend. Test the handshake explicitly:

# From inside the VPC, forcing SNI to the org hostname.
openssl s_client -connect yourorg.my.salesforce.com:443 
  -servername yourorg.my.salesforce.com </dev/null

# Compare with SNI omitted. If this one fails and the above
# succeeds, your client library is the problem, not the network.
openssl s_client -connect yourorg.my.salesforce.com:443 </dev/null

On the outbound side the mirror image applies. Salesforce is the client calling your endpoint service, and it will validate your certificate against the hostname it was told to use. If your Network Load Balancer terminates TLS with a certificate whose subject alternative names do not include the private DNS name you registered, the callout fails on hostname verification. Issue the certificate for a name you control and that matches what the named credential targets.

Failure family four: you cannot rehearse this properly

This one deserves its own section because it changes how you plan the work.

Private connections are not copied from production into sandboxes. You recreate them. And the ability to recreate them depends on sandbox type:

• Full and Partial Copy sandboxes: you can create connections and provision them. This is where genuine rehearsal happens.
• Developer and Developer Pro sandboxes: you can create connections but cannot provision them.
• Scratch orgs and Developer orgs: create only, no provisioning.

If your organization only has Developer sandboxes, there is no lower environment where the private path can be exercised end to end. That is a procurement conversation, not an engineering one, and it needs to happen at design time. Budget for a Full or Partial Copy sandbox or accept that production is your first real test.

Related trap: connections are packageable while unprovisioned, and unprovisioned connections do not consume a license. That is useful for shipping configuration through a deployment pipeline, but it also means a successful deploy proves nothing about connectivity.

Failure family five: license arithmetic and connection identity

Each Private Connect license covers one provisioned connection in each direction. Each connection is a one-to-one mapping between an org ID and a VPC endpoint ID.

Read that mapping carefully, because it is where multi-account AWS estates get expensive. Separate VPCs mean separate endpoints mean separate connections mean separate licenses. Separate orgs mean the same again. A design with three application VPCs across two orgs is not one connection, it is six.

The architectural answer is usually a shared services or egress VPC that owns the endpoint, with the application VPCs reaching it through a Transit Gateway. You pay for one endpoint and route to it. This is the same centralized-endpoint pattern you would already use for interface endpoints to AWS services, and the reasoning is identical: endpoints are billed per attachment, so consolidate them.

Four inbound connections consume four licenses. If you hold four licenses, you have used all of them and have none left for outbound. Count both directions before the renewal conversation.

Failure family six: the AWS side of an outbound connection

For outbound, you are the service provider. Salesforce’s transit VPC creates the endpoint into your account, which means you have to build a proper endpoint service.

The sequence, in the order that avoids rework:

1. Stand up a Network Load Balancer in front of your service, across at least two Availability Zones.
2. Create the VPC endpoint service backed by that NLB.
3. Copy the IAM role ARN from the Private Connect setup page in Salesforce for your region, and add it to the allowed principals on the endpoint service. Without this, Salesforce cannot connect and the failure looks like a generic timeout.
4. Register a private DNS name on the endpoint service and complete domain ownership verification.
5. Create the outbound connection in Salesforce using the endpoint service name, then sync it to pick up the AWS-side status.
6. Associate the connection with a named credential. Callouts only take the private path when they go through a named credential bound to that connection.

Step four is where people lose an afternoon. Domain verification requires a TXT record in a public zone, because AWS queries the public name servers registered at your domain’s registrar. Put the record in a private hosted zone and verification sits in pending forever with no useful error.

# Register the private DNS name on your endpoint service.
aws ec2 modify-vpc-endpoint-service-configuration 
  --service-id vpce-svc-EXAMPLE 
  --private-dns-name "sf-integration.example.com"

# Read back the TXT record name and value AWS expects.
aws ec2 describe-vpc-endpoint-service-configurations 
  --service-ids vpce-svc-EXAMPLE 
  --query 'ServiceConfigurations[0].PrivateDnsNameConfiguration'

# After publishing the TXT record in your PUBLIC zone:
aws ec2 start-vpc-endpoint-service-private-dns-verification 
  --service-id vpce-svc-EXAMPLE

One detail worth knowing: if verification status later changes away from verified, new connection requests are refused while existing connections keep working. So a lapsed TXT record does not break you immediately. It breaks you the next time something reconnects, which is usually during an unrelated incident.

What this costs, mechanically

Rates change, so model the mechanism rather than memorizing figures from a blog post.

• Interface endpoint hours. Billed per endpoint, per Availability Zone, per hour, whether or not traffic flows. A three-AZ deployment is three billable attachments.
• Data processing. Per GB through the endpoint, on a tiered scale that drops as regional volume rises.
• Cross-AZ transfer through interface endpoints. AWS removed this charge, so multi-AZ endpoint placement is a resilience decision rather than a cost trade-off.
• Cross-region. Still charged, in both directions, plus a per-hour fee on the endpoint service for each remote region with attached endpoints. Match your org region to your VPC region if you can.
• Provider-side infrastructure. For outbound, the NLB and its targets are your bill, not PrivateLink’s.
• The Salesforce license. Separate from all of the above, priced per connection.

Compare honestly against what you are replacing. If the alternative is a NAT gateway carrying the same traffic, NAT has both an hourly charge and a per-GB processing charge of its own, and the comparison is often closer than people assume. If the alternative is a site-to-site VPN or a commercial network overlay, the calculus changes again. PrivateLink wins on the security posture, not automatically on the invoice.

Troubleshooting: proving the traffic is actually private

Do not trust a green status field. Prove it with evidence from the data path.

1. Resolve the hostname from inside the VPC. Private addresses from your endpoint subnets, or nothing else matters.
2. Check the endpoint state. aws ec2 describe-vpc-endpoints should show the endpoint available, with an ENI in each subnet you expect.
3. Read VPC flow logs. Filter for your application’s source addresses and confirm the destination is the endpoint ENI, not a NAT gateway or internet gateway address. This is the actual proof.
4. Check the security group on the endpoint. Interface endpoints carry security groups. If 443 is not permitted from your application subnets, connections hang rather than refuse, which reads as a Salesforce problem in your logs.
5. For outbound, sync the connection in Salesforce before believing its status. The displayed state is the last polled AWS state, not live.
6. Test the callout through the named credential. A direct callout to a hardcoded URL will not use the private path even when everything else is correct.

Two symptom patterns worth recognizing. Connections that hang and time out point at security groups or NLB target health. Connections that fail fast with a TLS error point at certificate naming or SNI. They are diagnosed in completely different places.

Common mistakes

• Treating the setup wizard’s success message as proof that traffic is private. It proves the control plane is configured.
• Creating the private hosted zone in one change and the record in another, producing NXDOMAIN for the org hostname in between.
• Publishing the domain verification TXT record in a private zone instead of the public one.
• Forgetting the Salesforce IAM role in the allowed principals list on the endpoint service, then debugging it as a network problem.
• Sizing the connection for peak batch throughput without checking the hourly data entitlement.
• Assuming sessions locked to their originating IP address will keep working. That behavior is documented as unsupported with Private Connect, along with asynchronous callouts using Apex continuations.
• Building one endpoint per application VPC and discovering the license count at renewal.
• Leaving callouts on hardcoded endpoints instead of named credentials, so outbound traffic never enters the private path.

Best practices for a Salesforce AWS PrivateLink rollout

• Inventory every integration touching the org first, with direction, protocol, hostname, and peak volume per hour. The inventory determines the design.
• Centralize endpoints in a shared services VPC and route from application VPCs through Transit Gateway rather than duplicating endpoints and licenses.
• Put the endpoint in the same region as the org. Cross-region private connectivity works but gives up most of the latency benefit and adds transfer charges.
• Enable VPC flow logs before cutover so you have a clean before-and-after picture. Ship them somewhere you can query, whether that is CloudWatch Logs Insights, Athena over S3, or a platform like Grafana Cloud or Datadog.
• Alert on the outbound hourly limit as a percentage of entitlement, not on callout failures. By the time callouts fail you are already down.
• Terraform the AWS side. The endpoint, the endpoint service, the allowed principals, the private hosted zone, and the records all belong in code. The Salesforce side is click-configured, so document it in the same repository as a runbook.
• Keep a documented rollback: the change that reverts you to public routing is removing or disassociating the private hosted zone, and it should be a one-step action you have practiced.
• Review the configuration on a schedule. Private routing is invisible in normal operation and easy to break with an unrelated DNS or VPC change.

Frequently asked questions

Does Private Connect encrypt my traffic?

Your traffic is already encrypted in transit by TLS. What private connectivity changes is the path, not the cipher. The value is that packets stay on the AWS network rather than traversing the public internet, which removes an entire class of exposure and simplifies a lot of compliance conversations. It is not a substitute for TLS, and it does not authenticate anything.

Can I use Private Connect to access the Salesforce user interface privately?

No. This is API-path connectivity for integrations. Browser access to the org is a separate concern, and if that is what you need, you are looking at conditional access policies, login IP ranges, or a corporate network solution rather than PrivateLink.

What Salesforce features actually route through an outbound connection?

Salesforce documents outbound support for Apex callouts, External Services, Platform Events, Change Data Capture, CRM Analytics, the OData 4.01 adapter and custom adapter for Salesforce Connect, the Salesforce Connect SQL adapter for Amazon Athena, and Agent Actions backed by External Services. Inbound covers the supported public APIs. Anything not on the current list should be verified against the documentation before you design around it, because that list grows.

Do I need Hyperforce for this?

Salesforce states that Private Connect works whether the org sits in first-party data centers or on Hyperforce. That said, Hyperforce brings its own requirements, particularly the SNI behavior described above, so confirm where your org actually runs before you start debugging TLS.

How do I connect on-premises systems, not just AWS workloads?

Bring the on-premises network into AWS first with Direct Connect or a site-to-site VPN, then use the interface endpoint from there. The extra work is DNS: on-premises resolvers need a Route 53 Resolver inbound endpoint to see the private hosted zone, otherwise they resolve the org hostname publicly and you are back to the first failure family.

Can I use one connection for several Salesforce orgs?

No. A connection is a one-to-one mapping between an org ID and a VPC endpoint ID, so each org needs its own provisioned connection and its own license, even if they all point at the same VPC.

Is a VPN a reasonable alternative?

For Salesforce specifically, no. Salesforce does not terminate a customer VPN into the org, so PrivateLink is the mechanism on offer. A VPN or a commercial overlay such as NordLayer solves a different problem, which is getting your own users and sites onto your own network. Where those tools genuinely overlap is the on-premises leg, and even there Direct Connect is usually the better answer for sustained integration traffic.

The one thing to remember

A Salesforce AWS PrivateLink deployment that is misconfigured looks exactly like one that works. Same green statuses, same successful callouts, same clean application logs. The only difference is which path the packets took, and the only place that difference is visible is your flow logs.

So build the verification step into the project rather than the postmortem. Resolve the hostname from inside the VPC, read the flow logs, confirm the destination is the endpoint ENI, and keep the check running afterwards. Everything else in this post is detail. That one habit is what separates private connectivity from a line item that says private.

Need help wiring Salesforce to AWS privately?

This is the kind of work where a second pair of eyes on the DNS and the flow logs saves a security review later. I help teams with:

• Designing the endpoint topology, including centralized endpoints behind Transit Gateway so you are not buying a connection per VPC.
• Building and verifying the Route 53 private hosted zone override, including Resolver inbound endpoints for on-premises callers.
• Standing up the endpoint service side for outbound connections: NLB, allowed principals, private DNS name registration, and domain verification.
• Proving traffic is genuinely private with VPC flow log analysis, before and after cutover.
• Instrumenting the hourly data entitlement so you find out about throttling from a dashboard rather than from failed callouts.
• Codifying the AWS side in Terraform or OpenTofu, with a documented rollback to public routing.

If you are mid-way through one of these and something is not behaving, send me the dig output from inside the VPC, a flow log sample, or your endpoint service configuration and I will tell you what I see.

The post Salesforce AWS PrivateLink: The Failures Nobody Warns You About appeared first on John Nessime.

No. And that question is the entire problem with this one.

This post is a reference for the PHP2Shell vulnerability: what actually broke inside PHP, which servers are exposed, how to patch each common setup, how to prove the fix is really running, and which quick fixes give you false comfort. Keep it open while you work through your servers.

What happened: the PHP2Shell vulnerability in plain terms

PHP2Shell is the nickname researchers gave to two flaws in the PHP engine itself, CVE-2026-109618 and CVE-2026-109619. PHP rates both 9.2 on CVSS v4.0, and both can let an unauthenticated attacker run code on a web server. The name is the researchers’, not the PHP project’s.

The dates matter here, because they decide if you were exposed and for how long:

• 20 October 2026: PHP announced an emergency release for two critical issues. Details leaked publicly the same day.
• 21 October 2026: fixed releases shipped: PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35, with a full advisory.
• 23 October 2026: PHP published an informational advisory for a third, much narrower issue, CVE-2026-115560, with a one-line mitigation and no emergency release.

The flaws were reported by Wiz Research, Calif and OpenAI’s Codex Security team, who used AI models to turn long-standing weaknesses into working exploits. The affected code goes back to PHP 5.4, so effectively every PHP version from the last decade is in scope.

The two bugs, without the jargon

CVE-2026-109618 (out-of-memory corruption). PHP enforces memory_limit by aborting a request the instant it crosses the limit. If that abort lands while PHP is still building internal request data, for example while parsing a multipart file upload, the data is left half-initialized. Request cleanup then treats it as valid and works with memory that can still hold attacker-controlled leftovers.

CVE-2026-109619 (stale global state). Some state owned by one request can survive after its memory has been freed, for example after a fatal error. The next request handled by the same worker can use that stale state before normal setup resets it.

Here is why this is worse than the usual plugin CVE. The bug lives in the runtime, not in your code. A site whose own code is flawless is still exposed, and updating WordPress, Laravel or Drupal changes nothing, because the PHP binary underneath comes from your OS packages, your host or your container image. The researchers also demonstrated reading memory from the PHP worker process, which can contain passwords, tokens and keys belonging to other people’s requests, without ever getting a shell.

Who is exposed, and why mod_php is the worst case

PHP’s advisory covers every installation that serves web requests: PHP-FPM, Apache with mod_php, php-cgi, FrankenPHP and LiteSpeed. The risk is not equal across them, though.

mod_php is the highest risk. Apache with mod_php runs request shutdown even when request startup already failed. That means simply running out of memory while PHP parses the request body is enough, and one unauthenticated request can trigger the attack before a single line of your application runs. PHP demonstrated this with default memory and input limits.

PHP-FPM is harder to hit, not safe. FPM shuts the worker down when a request fails early in setup, which blocks the cleanest path. Public reporting says one of the two bugs was still shown to reach code execution under FPM, through a path that depends on something the application does. Treat “we run FPM” as “we have slightly more time,” nothing more.

If you are not sure which mode you run, these two checks answer it on most Linux servers. The first lists Apache modules and shows a PHP module only if mod_php is loaded. The second lists running FPM services.

apachectl -M | grep -i php
systemctl list-units --type=service | grep -i fpm

How to fix PHP2Shell: it depends on where your PHP comes from

The fix is always the same in principle: run a patched PHP build. Getting there depends entirely on who builds your PHP. And there is a trap that runs in both directions. Linux distributions and long-term-support vendors received the patch early and backported it, so a patched package can carry an old-looking version like 8.3.6. Meanwhile, an unpatched process can keep running after the package updated. The version string answers neither question reliably.

Debian and Ubuntu packages

Debian, Ubuntu and Ondřej Surý’s widely used repository (packages.sury.org, also published as ppa:ondrej/php) all shipped fixed packages, including builds for some end-of-life branches. Update normally, then confirm the CVE is in the package changelog instead of reading the version number. Replace 8.3 with your branch.

sudo apt update && sudo apt upgrade
zgrep CVE-2026-109618 /usr/share/doc/php8.3-common/changelog.Debian.gz
sudo systemctl restart php8.3-fpm
sudo systemctl restart apache2

If zgrep prints a line, the fix is in the installed package. If it prints nothing, you are either on an older distro release that needs a paid extended-support channel, or on a repository that has not shipped the fix.

RHEL, AlmaLinux and Rocky Linux

Same idea with RPM. Update, then search the package changelog for the CVE. If you use Remi’s parallel packages, the package name follows its own pattern (for example php83-php-common), so query that name instead.

sudo dnf upgrade 'php*'
rpm -q --changelog php-common | grep CVE-2026-109618
sudo systemctl restart php-fpm httpd

One warning from PHP’s own advisory: the volunteer-run rpms.remirepo.net repository was not expected to have updates on release day, and PHP suggested switching to distribution packages or building from source if you depend on it. Check before assuming the update landed.

cPanel, CloudLinux and LiteSpeed

cPanel ships its own ea-php packages through EasyApache 4, and CloudLinux ships alt-php. CloudLinux backported the fix into supported and end-of-life branches, so most of its fixed builds keep version numbers below the php.net release. CloudLinux’s published procedure is to update both package families, then restart the services that run PHP so no worker keeps the old binary loaded.

yum clean all && yum update 'alt-php*' 'ea-php*' -y
/scripts/restartsrv_httpd

If your domains run PHP-FPM through cPanel, restart the FPM service as well. On LiteSpeed, restart the web server so every lsphp process respawns from the new build. Public analysis says LiteSpeed’s LSAPI behaves more like FPM than mod_php, which buys time but does not remove the need to patch.

DirectAdmin with CustomBuild

CustomBuild compiles PHP from source, so you get the fix when CustomBuild offers the patched release and you rebuild. Refresh the version list, confirm the fixed release appears for each PHP version you run, then build.

cd /usr/local/directadmin/custombuild
./build update
./build versions
./build update_versions

Read the ./build versions output before running the last command. If a branch you use still shows an unpatched release, apply the mitigations below for that branch until it updates. Compiling takes a while on small servers, so plan for it rather than starting it at peak traffic.

Docker and container images

A container never patches itself. If your image is built FROM an official php tag, rebuild with a fresh pull of the base image and redeploy. If your Dockerfile pins the base by digest, the digest has to change or nothing happens.

docker compose build --pull
docker compose up -d

Then run php -v inside the running container, not on the host, to confirm the version the app actually uses.

Managed hosting versus unmanaged VPS

On managed hosting, the host owns the PHP runtime, and several large hosts announced they had patched their fleets. Your job there is to get written confirmation, not to assume. On an unmanaged VPS from Contabo, Interserver or any similar provider, nobody patches PHP but you. That is the trade you accepted for the lower price and full root access, and this week is when it gets collected.

End-of-life PHP: 8.1 and older

The PHP project released no fixed builds below 8.2. Because of the severity, its security team published minimal source backports for PHP 7.4, 8.0 and 8.1 for people who build from source. Beyond that, your options are a distribution or vendor that backports (Ubuntu’s own PHP 8.1 on 22.04, the Surý repository, Freexian ELTS for older Debian, CloudLinux and TuxCare, HeroDevs), or upgrading. Note that PHP 8.2 itself reaches end of life on 31 December 2026, so if you are touching an 8.2 server now, plan the next upgrade at the same time.

Prove the patch is actually running

This is the failure mode that will bite most people. The package manager says “upgraded,” the ticket gets closed, and the old PHP code keeps serving traffic because the processes that loaded it were never restarted. Linux happily keeps running a binary whose file has been replaced on disk.

1. Confirm the fix is in the package with the changelog check above, not the version string.
2. Restart everything that runs PHP: every FPM pool, Apache (for mod_php), LiteSpeed, and any long-running PHP workers such as queue consumers managed by Supervisor or systemd.
3. Check the restart time is after the upgrade. systemctl status php8.3-fpm shows when the service started. If that is earlier than your package install, it is still running old code.
4. Ask the system what still needs a restart. On Debian and Ubuntu, sudo needrestart -r l lists services still using outdated files, without restarting anything.
5. Check the right binary. php -v on the command line reports the CLI build, which can differ from the FPM build serving your site. On Debian and Ubuntu, php-fpm8.3 -v asks the FPM binary directly.

Mitigations if you cannot patch today, and what they don’t do

PHP is blunt about this: only an update is a complete fix. The settings below reduce exposure to the specific attack paths it has confirmed. They do not make a server safe, and some of them have real costs. Use them to buy hours, not weeks.

Set them where applications cannot override them: php_admin_value or php_admin_flag in the FPM pool, or the Apache configuration for mod_php. Do not rely on ini_set() or an FPM .user.ini file, because those apply too late in the request. Here is the FPM pool version on a Debian-style path (adjust the file and branch to yours):

; /etc/php/8.3/fpm/pool.d/www.conf
php_admin_value[memory_limit] = -1
php_admin_value[post_max_size] = 8M
php_admin_value[upload_max_filesize] = 6M
php_admin_flag[session.upload_progress.enabled] = off

And the Apache equivalent for mod_php, inside the virtual host or a Directory block:

php_admin_value memory_limit -1
php_admin_value post_max_size 8M
php_admin_value upload_max_filesize 6M
php_admin_flag session.upload_progress.enabled off

Restart every worker afterwards. Now the reasoning behind each line, because each one protects a different path:

• memory_limit = -1 targets CVE-2026-109618. The confirmed exploit needs a predictable per-request limit to hit, so removing the limit removes the trigger. The cost: one request can now consume all available memory. A genuine allocation failure can also still reach the vulnerable code.
• Low post_max_size shrinks the request bodies an attacker can use. PHP suggests 8M where uploads are not needed and 32M where they are, with upload_max_filesize kept below it.
• session.upload_progress.enabled = off targets CVE-2026-109619. It blocks the confirmed session paths but not other variants. It does nothing at all for CVE-2026-109618.
• On PHP 8.5 only, fatal_error_backtraces = off blocks the confirmed fatal-backtrace path for CVE-2026-109619, and nothing else.

Because removing the memory limit opens a denial-of-service risk, PHP recommends a container or cgroup memory limit as the availability boundary, ideally one that kills only the affected worker. It specifically warns against RLIMIT_AS or RLIMIT_DATA, since those make allocations fail at the limit, which is exactly the condition the bug needs. Enforcing request-size, field-count and multipart-part limits at your reverse proxy helps too.

A web application firewall is a reasonable stopgap, not a fix. Cloudflare deployed rules before the patch shipped and reported attackers changing payloads to get around them. If your site already sits behind Cloudflare, those rules help. They do not replace the update.

The third issue: CVE-2026-115560

Two days after the release, PHP disclosed one more possible code-execution path. It needs an unusual setup: the application sets session.serialize_handler to php_binary or php_serialize in configuration, then switches it to php at runtime with ini_set() before calling session_start(), with session upload progress still enabled. The fixed releases above do not cover it. PHP’s advice is to keep session.upload_progress.enabled=0 until the next regular bugfix release. If you already set it for CVE-2026-109619, leave it in place and you are done.

After patching: assume someone tried

Amazon reported a honeypot hit against these flaws on 17 October, four days before disclosure, and exploitation attempts climbed fast once details were public. Most reports so far describe scanning and failed attempts rather than confirmed break-ins, but “nobody has reported it” is not evidence about your server. If you ran an unpatched, internet-facing PHP site during that window, do three things.

• Rotate secrets a PHP worker could have held in memory: database passwords, API keys, payment provider keys, SMTP credentials and, for WordPress, the salts in wp-config.php (which logs every user out, so warn people first).
• Read your logs for the patterns researchers flagged: unusual multipart requests followed by memory-limit errors or bursts of HTTP 500 responses, oddly large PHPSESSID cookies in responses, and PHP workers restarting with no explanation.
• Look for anything new and executable in your web roots and upload directories, especially PHP files that no deploy put there.

Troubleshooting

“php -v still shows the old version.” Either it is a distro backport (check the changelog for the CVE), or you are looking at the CLI binary while a different build serves the site. Check the FPM binary directly and the container, not the host.

“The site throws 500 errors after the update.” The fix changes how PHP handles running out of memory. Crossing memory_limit now raises a soft error at a safe point, and a second, higher threshold ends the process outright, without running shutdown handlers. A plugin or import job that used to squeak by can now fail differently. Check the error log, then review memory_limit and post_max_size against what the application genuinely needs.

“Cleanup or logging code stopped running on big jobs.” Same cause. If anything important lives in a shutdown function, it will not run when the hard limit is hit. Fix the memory usage rather than raising limits blindly.

“The package manager says there is nothing to update.” You are probably on a repository that has not shipped the fix, an end-of-life distro release, or an end-of-life PHP branch with no vendor backport. That is a migration problem, not a patching problem.

“I set the mitigation but the old value still shows.” Look for overrides in other pool files, per-directory configuration or application code, and make sure you used the admin variants and restarted the workers.

Common mistakes

• Updating WordPress, plugins or the framework and calling it done. None of that touches the PHP runtime.
• Trusting the version number in either direction, and declaring a backported package “vulnerable” or a stale process “patched.”
• Upgrading the package and never restarting FPM, Apache or long-running workers.
• Treating session.upload_progress.enabled=0 as the fix. It was circulated as a workaround before the release and only covers part of one CVE.
• Setting mitigations with ini_set(), .htaccess overrides that apps can change, or .user.ini, which take effect too late.
• Leaving memory_limit=-1 in place long after patching, with no cgroup boundary behind it.
• Forgetting the boxes nobody owns: the staging server, the old client site on PHP 7.4, the forgotten container in a side project.

Best practices

• Keep an inventory of every PHP install, its source (distro, panel, container, source build) and its branch. This week shows why.
• Verify fixes by CVE in the package changelog, and verify the running process by start time.
• Patch mod_php servers first, then public FPM sites, then internal tools.
• Once patched, restore a sane memory_limit, but keep session upload progress off until PHP’s next regular release covers CVE-2026-115560.
• Move servers off mod_php to PHP-FPM where you can. It would not have saved you here, but it narrowed the easiest path.
• Stay on a supported PHP branch, or pay for a vendor that backports. An unsupported branch turns every emergency into a migration.

FAQ

What is the PHP2Shell vulnerability?

It is the nickname for two memory-safety flaws in the PHP engine, CVE-2026-109618 and CVE-2026-109619, both rated 9.2 on CVSS v4.0. They can allow unauthenticated remote code execution on servers that run PHP for web requests.

Which PHP versions fix PHP2Shell?

PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35 from php.net. Distribution and vendor packages may carry the fix under different, older-looking version numbers, so check the package changelog for the CVE.

Is my WordPress site affected?

If it runs on an unpatched PHP build, yes, regardless of how up to date WordPress and its plugins are. The flaw is in PHP itself, so the fix comes from your server, host or container image.

Is disabling session.upload_progress enough?

No. It blocks the confirmed session paths for one of the two bugs and covers the narrow third issue, CVE-2026-115560. It does nothing for CVE-2026-109618. Only an update is a complete fix.

I’m on PHP 7.4 or 8.1. What now?

php.net has no release for you, but PHP published source backports for 7.4, 8.0 and 8.1, and several vendors and distributions shipped backported packages. Use one of those immediately, then plan the upgrade to a supported branch.

Do I need to restart anything after updating PHP?

Yes. Restart every PHP-FPM pool, Apache if you use mod_php, LiteSpeed, and long-running PHP workers. Until they restart, they keep running the vulnerable code already loaded in memory.

Should I rotate passwords and keys?

If an internet-facing server ran an unpatched build after details went public, it is a sensible precaution. Researchers showed worker memory could be read remotely, and that memory can hold credentials from other requests.

Conclusion

The PHP2Shell vulnerability lives below your application, so the only real fix is a patched PHP build that is actually running. Everything else in this post is either a way to get there or a way to buy time. If you remember one thing, make it this: verify the fix by CVE in the package, and verify the process by its restart time. The version string will mislead you in both directions.

Need help patching PHP2Shell across your servers?

If you have more PHP servers than certainty, I can help you close this out properly:

• Inventory every PHP install across your VPS, cPanel, DirectAdmin and container hosts, with branch and package source.
• Patch each one through the right channel and prove the fix by CVE and process restart time.
• Apply and later roll back the interim mitigations safely, with a memory boundary in place.
• Rebuild and redeploy PHP container images, including pinned-digest pipelines.
• Review logs for the published indicators and plan secret rotation without breaking production.
• Plan the move off end-of-life PHP branches and off mod_php.

Send me the output of php -v, your package changelog check, and how PHP runs on the server (FPM, mod_php or LiteSpeed), and I’ll tell you where you stand.

The post PHP2Shell Vulnerability: What Happened, How to Patch, and How to Prove It Worked appeared first on John Nessime.

Then you open the ModSecurity audit log and find eleven rule matches on a single request, none of them obviously the culprit.

Dealing with ModSecurity false positives on WordPress is not hard once you understand how the OWASP Core Rule Set decides to block. It is very hard if you go looking for “the rule that blocked me,” because under the default configuration that rule does not exist. This post covers reading the log, choosing the right exclusion mechanism, the WordPress failure families that generate most of the noise, and the tuning knobs that quietly gut your coverage.

The invisible part: no single rule blocked the request

The Core Rule Set runs in anomaly scoring mode by default. Individual detection rules do not deny anything. They match, they log, and they add to a running score with setvar. At the end of the request phase, a separate blocking evaluation rule compares the accumulated score against a threshold and returns the 403.

Scores come from rule severity. A CRITICAL match adds 5, ERROR adds 4, WARNING adds 3, NOTICE adds 2. The inbound threshold ships at 5 and the outbound threshold at 4. So one critical match is enough on its own, and so are two notices plus a warning.

That is the failure mode nobody warns you about. The log line that produced your 403 lives in REQUEST-949-BLOCKING-EVALUATION.conf and it says the inbound anomaly score was exceeded. It does not name what matched. Take that ID and disable it and you have not fixed a false positive, you have turned off blocking for the whole site while leaving every detection rule running and logging. The site will look fixed. It is not protected.

First rule of tuning, then: the blocking rule is a symptom. Read backwards from it.

Read the log before you touch a rule

Every alert carries a unique_id. That value ties the error log line to the full audit log record for the same transaction, and it is the only reliable way to see every rule that fired on one request instead of a scattering of lines from different visitors.

# Paths vary by build. Check SecAuditLog in modsecurity.conf for yours.
tail -n 200 /var/log/httpd/error_log | grep -i modsecurity

# Then pull the whole transaction using the unique_id from that output
grep -F 'Zm9vYmFyCg' /var/log/httpd/modsec_audit.log

From each match, four fields matter and the rest is noise:

• The rule ID, as [id "941320"]. This is what you exclude.
• The matched variable, at the end of the pattern-match message as something like at ARGS:content. This is what you exclude it from.
• The URI, which tells you how narrowly you can scope the exclusion.
• The paranoia level tag, as [tag "paranoia-level/2"]. If every alert is tagged level 2 or higher, the real fix may be a paranoia level change rather than an exclusion.

One warning most guides skip: audit logs of false positives contain the payload that triggered them. If a user’s password happens to look like a shell command, it lands in your log in plaintext. Treat these files as sensitive, rotate them, and redact before pasting anything into a public bug report.

Pick the right exclusion mechanism

ModSecurity gives you four ways to write an exclusion, differing on two axes: applied once at startup or evaluated per request, and removing the whole rule or just one variable from it.

• SecRuleRemoveById and SecRuleRemoveByTag delete a rule or a whole tagged category at configure time. Blunt, cheap, global.
• SecRuleUpdateTargetById and SecRuleUpdateTargetByTag strip one variable from a rule at configure time, leaving the rule live for everything else.
• ctl:ruleRemoveById and ctl:ruleRemoveByTag do the same removal, but only for transactions matching a condition you write.
• ctl:ruleRemoveTargetById and ctl:ruleRemoveTargetByTag strip one variable, conditionally. This is the surgical option and the one to reach for first.

Runtime exclusions cost a little, since each is an extra rule evaluated on every request. That overhead is small next to the regex work the rule set already does, and the precision is worth it.

Placement decides whether an exclusion works at all

This is where most people lose an afternoon. The two types go in opposite places, for mechanical reasons rather than arbitrary ones.

• Configure-time exclusions go after the CRS include. They remove rules, and a rule has to exist before you can remove it.
• Runtime exclusions go before the CRS include. They change how rules behave for this transaction, and changing a rule after it has already run does nothing.

The Core Rule Set ships two files for exactly this, both with an .example suffix you drop before they load. REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf holds runtime exclusions, and RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf holds configure-time ones. Use them. Local files that survive a rule set update are the entire point.

Five failure families behind most ModSecurity false positives

Almost everything you will hit on WordPress falls into one of these. Recognizing the family tells you the shape of the fix before you read a single log line.

1. Post content that legitimately contains markup

WordPress lets editors put HTML in posts. It maintains its own allowed-tag list, sanitizes server side, and that mechanism has been reviewed hard over the years. The Core Rule Set knows none of this. It sees an <h1> or an inline style attribute in a POST parameter and matches an XSS rule.

This produces the “it saved yesterday” ticket, because it is content-dependent. Plain paragraphs pass. A pasted table with attributes does not. The fix is a target exclusion scoped to the save endpoint and the specific parameter, never a global rule removal:

# Runtime exclusion. Goes in REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf
# Replace 941320 and the parameter with what your audit log actually shows.
SecRule REQUEST_URI "@beginsWith /wp-admin/post.php" 
    "id:1000,
    phase:1,
    pass,
    t:none,
    nolog,
    ctl:ruleRemoveTargetById=941320;ARGS:content"

Pick your own IDs from a range CRS will never use. Reusing an existing ID is a startup error rather than a silent one, which is the good kind of mistake.

2. REST and admin-ajax calls that fail with no visible error

This family is nastier, because there is no error page. The browser hits /wp-json/ or /wp-admin/admin-ajax.php, gets a 403, and the JavaScript either swallows it or shows a generic “updating failed” toast. Autosave stops. An editor panel stays empty. A form plugin silently drops submissions. Users report this as flakiness rather than a security block, so it sits unfixed for months. When someone says the editor is “just slow sometimes,” open the browser network tab and look for 403s before going anywhere near PHP.

Scoping differs here. REST routes are structured, so use @rx against REQUEST_FILENAME to hit a route family rather than one path. For admin-ajax.php, the useful discriminator is usually the action parameter, since that one endpoint multiplexes dozens of unrelated plugin handlers.

3. Media uploads that break at a size boundary

Multipart uploads have two independent failure paths that look identical from the browser. The first is a genuine rule match on the multipart structure, or on binary content containing a byte sequence some rule dislikes. That shows up with a rule ID and you tune it normally.

The second is an engine limit, not a rule. SecRequestBodyLimit caps the total body ModSecurity will buffer, SecRequestBodyNoFilesLimit caps the non-file portion, and SecRequestBodyLimitAction decides whether an oversized body is rejected outright or inspected partially. If uploads fail above a consistent size and no CRS rule ID appears anywhere in the log, stop tuning rules. That is a limit, and the fix is in engine config.

Raising them costs memory per concurrent request, and partial inspection leaves part of the body unscanned. Decide which trade you are making rather than bumping the number until the ticket closes.

4. Cookies with dynamic names

WordPress authentication cookies carry a hash in the name, so a literal exclusion is impossible. Configure-time directives accept a regular expression in the target if you wrap it in forward slashes:

# Configure-time exclusion. Goes in RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf
SecRuleUpdateTargetById 942440 "!REQUEST_COOKIES:/^wordpress_logged_in_.*/"

The catch is significant: the ctl action supports no regular expressions at all. If you need a pattern match on a variable name, you are forced into the configure-time form, which applies site-wide rather than to one path. That single constraint drives more design decisions than people expect.

5. Chained rules that refuse to be tuned

Some rules are written as a chain of conditions. Target exclusions only apply to the first rule in the chain, so if the variable you want to strip is referenced further down, nothing you write has any effect and you will burn an hour convincing yourself the config file is not loading.

When a target exclusion visibly does nothing on a rule you can watch matching, check whether it is chained. If it is, your only option is removing the whole rule under a narrow condition. Accept the wider blast radius, or change the request instead.

Install the WordPress plugin before writing anything custom

The Core Rule Set maintains a WordPress rule exclusions package. In CRS 3 it was built in and switched on with a variable in crs-setup.conf. In CRS 4 it moved out into a separate plugin published in the CRS plugin registry, and it is not installed by default.

That catches people upgrading. Exclusions that used to work vanish, WordPress starts throwing 403s again, and the setup variable they remember is gone from the config file. The reasoning behind the split is sound: fewer rules loaded means less code that can carry a bug, and one of the old exclusion packages did carry a bypass.

Plugins load through wildcard includes in a specific order. Get the order wrong and the plugin silently does nothing:

Include /path/to/crs/crs-setup.conf

Include /path/to/crs/plugins/*-config.conf
Include /path/to/crs/plugins/*-before.conf

Include /path/to/crs/rules/*.conf

Include /path/to/crs/plugins/*-after.conf

Two limits before you lean on it. The plugin covers vanilla WordPress only, so false positives from third-party plugins, page builders and form plugins are still yours to tune. And the comment field is deliberately left out of the exclusions, because comment forms have a long history as a real attack surface. If your comment form throws false positives, that is a decision the maintainers made on purpose, and overriding it is a genuine risk trade rather than a gap in their package.

On a server hosting more than one application, scope the plugin to the WordPress vhost using WebAppID or a Host header condition. WordPress exclusions applied to your unrelated API are a hole you did not mean to open.

The knobs that look like a fix and are not

When exclusions get tedious, two shortcuts start looking attractive. Both cost more than they appear to.

Raising the anomaly threshold. Going from 5 to 20 makes most false positives vanish, and it also means any attack category that scores low can never trigger a block. Detections that only ever produce a single critical match become unblockable. The CRS documentation is direct here: keep the threshold at the default and change the paranoia level instead. Raising it is a temporary tuning aid, not a configuration.

Dropping the paranoia level. Legitimate, but often applied backwards. Level 1 is designed to produce few false positives. Levels 2 and up add progressively more aggressive rules and progressively more tuning work. If your alerts are all tagged level 2, dropping to level 1 is the correct fix and it is honest about what you gave up.

The better tool for moving up is the split between blocking and detection paranoia levels. Set tx.blocking_paranoia_level to what you actually enforce and tx.detection_paranoia_level higher. Rules from the higher level execute and log, but only rules up to the blocking level count toward the score. You get to see what level 2 would have caught, in production, on real traffic, without anyone getting blocked. Then you tune, then you promote. The performance cost lands as soon as you raise the detection level, though, since those rules run either way.

Two smaller settings help here. tx.sampling_percentage sends only a fraction of requests into the rule set, a gentler introduction for a busy site than blocking everything on day one. And tx.reporting_level can log requests that would have been blocked at a higher paranoia level, which is exactly the signal you want while climbing.

Troubleshooting

The exclusion has no effect. Check placement first. A runtime exclusion loaded after the rules, or a configure-time exclusion loaded before them, fails silently. Then check for a chained rule. Then confirm the file is being included at all by breaking it deliberately with a syntax error and seeing whether the server refuses to start.

It works on one site and not another. Different vhosts load different config. On control panels, per-domain overrides usually sit in a separate file from the global rule set.

It worked and then stopped. Something updated the rule set. If you edited CRS rule files directly, your edits are gone, which is the whole argument against editing them.

Nothing appears in the log at all. Confirm the rule engine is on, and that auditing is enabled with enough log parts to include the request body. An engine in detection only mode logs without blocking, which is confusing if you did not know you were in it.

You are on shared hosting and cannot edit any of this. Some hosts expose a per-domain exclusion box, some let you disable rule IDs, some give you nothing. That is a real ceiling. Providers that hand you root, such as InterServer or Contabo, let you own the rule set and its update cadence, which on a site with genuine tuning needs beats the panel convenience you trade away.

Common mistakes

• Disabling the blocking evaluation rule because it showed up in the log. It reports the score, it detects nothing, and removing it disables blocking site-wide.
• Editing CRS rule files directly. Every update reverts you, and you now maintain a fork by hand.
• Excluding by message rather than by ID. Messages carry dynamic data and special characters, so the match is fragile.
• Removing an entire attack tag because two rules misfired. SecRuleRemoveByTag attack-sqli is defensible when nothing behind the WAF touches SQL. On WordPress it is not.
• Excluding /wp-admin/ wholesale, or stacking multiple rule set vendors. Admin is where a compromised session does the most damage, and overlapping rule sets make attribution guesswork.
• Leaving known false positives in place. Real attacks get lost in the noise, and the logs accumulate user data you never intended to collect.

Best practices

• Start in detection only. Collect real traffic, tune what you find, then enable blocking. Enforcing first is how you learn about broken checkout at two in the morning.
• Install the WordPress exclusions plugin before writing custom rules. Do not hand-write what upstream already maintains.
• Write the narrowest exclusion that resolves the alert: one rule, one variable, one path. Comment it with the rule ID, the endpoint, and why you judged it safe. Six months later that comment is the only thing between you and a blind revert.
• Keep exclusion files in version control and deploy them with the rest of your config. Test rule set updates on staging, since new signatures land regularly.
• Ship audit logs somewhere queryable. Grepping a growing text file stops scaling fast, and a platform like Grafana Cloud turns “which rule fires most” into a query instead of an afternoon.
• If an edge WAF such as Cloudflare sits in front of the origin, decide deliberately which layer owns which rule. Two WAFs scanning the same body doubles your false positive surface for no extra coverage.

Frequently asked questions

How do I find which ModSecurity rule blocked my WordPress request?

Take the unique_id from the 403 entry in the error log and grep the audit log for it. That gives you every rule that fired on the transaction. The blocking evaluation rule is what returned the 403, but the detection rules listed alongside it are what you tune.

Should I disable ModSecurity for WordPress instead of tuning it?

It is a defensible choice if the alternative is blocking mode with unresolved false positives, since a WAF everyone has learned to route around is worse than none. But detection only mode gives you the logs with none of the breakage, and that is almost always the better middle ground while you work through the tuning.

Is the WordPress rule exclusions plugin enough on its own?

For a stock install on a standard theme, usually. It covers vanilla WordPress and both block-based and classic themes. It does not cover third-party plugins, page builders, or theme frameworks, and those generate most of the false positives on a real site.

Why do false positives come back after a rule set update?

Either you edited rule files directly and the update overwrote them, or the update shipped new signatures your existing exclusions do not cover. The first is avoidable by keeping every change in the exclusion files. The second is the cost of running a maintained rule set, and it is why updates belong on staging first.

Does ModSecurity slow WordPress down?

It adds per-request work, mostly regular expression evaluation against request data, and the cost scales with paranoia level and body size. The settings that matter are the paranoia level, the request body limits, and whether response body inspection is on. Measure on your own traffic rather than trusting someone else’s benchmark.

Is ModSecurity still maintained?

The engine moved to the OWASP Foundation after Trustwave stepped back from commercial support, and it still receives fixes, though feature work is slower than it once was. The Core Rule Set is a separate project on its own cadence and is actively developed. Coraza is a compatible engine running the same rule language, worth evaluating for new deployments.

Wrapping up

The thing to hold on to about ModSecurity false positives on WordPress is that the rule which returned your 403 is almost never the rule you need to change. Anomaly scoring separates detection from blocking on purpose, and reading the log backwards from the blocking decision to the detection rules that fed it is the whole skill.

Everything else follows. Install the maintained WordPress exclusions instead of reinventing them. Write the narrowest exclusion you can, in the file that survives updates. Move the paranoia level, not the threshold.

Need help tuning your WAF?

Most of the WAF work I take on is untangling exactly this: a rule set that is technically installed and practically ignored, because nobody had time to work through the false positives. I can help with:

• Triaging your audit logs and mapping every recurring false positive to the rule, variable and endpoint that produced it
• Writing scoped exclusion sets for WooCommerce, form plugins, page builders and custom REST routes that upstream packages do not cover
• Moving a site off direct rule file edits and onto exclusion files that survive rule set updates
• Planning a CRS 3 to CRS 4 migration, including the exclusion package to plugin transition and the paranoia level variable renames
• Running a staged rollout from detection only to blocking, with the reporting and sampling settings that make it safe
• Shipping ModSecurity audit logs into a queryable log platform so rule tuning becomes a dashboard instead of a grep

If you have a blocked request you cannot explain, send me the audit log entry and your exclusion file and I will tell you what is firing and why.

The post Your WordPress 403 Isn’t the Rule You Think: Tuning ModSecurity False Positives appeared first on John Nessime.

The user pool was set to MFA: Required. Somebody had screenshotted that console page for the security questionnaire. The screenshot was accurate. It just didn’t mean what everyone assumed it meant.

That gap is the whole problem with Cognito MFA for healthcare apps. The service gives you a setting that reads like a guarantee, and then quietly carves out entire classes of users, sessions, and sign-in paths that the setting never touches. Nothing errors. Nothing logs a warning. Your compliance evidence looks fine right up until someone actually tests it.

This post covers what the MFA switch in Amazon Cognito actually controls, the five failure families that show up in real healthcare deployments, how to wire a user pool so the enforcement matches the claim, and what to check when a clinician calls the help desk because they can’t get in.

What HIPAA actually requires from MFA right now

Worth getting this straight before touching any configuration, because a lot of vendor content is confidently wrong about it.

The HIPAA Security Rule as it stands today does not name multi-factor authentication. It requires person or entity authentication, and it leaves the mechanism to your risk analysis. MFA is how most people satisfy that in practice, but the rule does not spell it out.

What changes the picture is the Notice of Proposed Rulemaking that HHS published in the Federal Register on January 6, 2025. It would make MFA an explicit requirement for access to systems handling ePHI, with narrow exceptions, and it would remove the “addressable” escape hatch that currently lets organizations document a control as infeasible. This one matters enough to name the date, because timelines have moved: the comment period closed in March 2025, the rule is still proposed, and the Unified Agenda has pushed anticipated final action out to 2027. That estimate is not binding and has already slipped once.

Practical read: build for the proposed rule now, but don’t tell a client the mandate is already law. It isn’t, and auditors know the difference.

On the AWS side, Amazon Cognito is on the HIPAA Eligible Services list and is covered by the AWS Business Associate Agreement. Eligibility is a starting line, not a finish line. You still have to accept the BAA in each AWS account that touches PHI, and you still have to configure the service correctly. Nothing about creating a user pool makes it compliant.

Failure family one: half your users never see the MFA prompt

This is the one that produced the pen test finding, and it’s the most common serious gap I run into.

Cognito user pool MFA applies to local users. Those are the accounts that live in the user pool directory and authenticate against it. For federated users, the ones arriving through SAML or OIDC from a hospital identity provider, Okta, Entra ID, or an in-house ADFS, Cognito hands the entire authentication decision to the upstream IdP. It does not add a second factor. It cannot. It never sees the password.

So a portal with patients as local users and staff federated from a hospital directory has exactly the split you’d fear: patients get MFA, clinicians don’t, unless the hospital IdP is enforcing it independently. And the hospital IdP is usually somebody else’s system, run by a team you don’t have a Slack channel with.

There’s a second version of this. The app client, not the user pool, controls which authentication flows are permitted. If a legacy mobile client is still allowed to use a flow you thought you’d retired, that path stays open regardless of what the pool-level MFA setting says.

What to do about it:

• Inventory every app client attached to the pool. Not the ones you remember creating. Run list-user-pool-clients and account for all of them.
• For each federated IdP, get written confirmation of what the upstream enforces. A trust relationship is not a control. Put the answer in your risk analysis where an auditor will find it.
• Treat federation as a boundary you inherit risk across, and say so out loud in design reviews. This is the single most useful sentence you can contribute to a healthcare identity project.

Failure family two: the recovery attribute collision

This one is genuinely subtle, and it doesn’t bite during development. It bites during rollout, at scale, to the people least able to tolerate it.

Cognito refuses to let a user’s MFA delivery method be the same channel as their account recovery method. The logic is sound: if a password reset code and a second factor both land in the same inbox, an attacker with that inbox has everything, and the second factor is decoration.

The consequence is where it gets sharp. If MFA is required, a user has only an email address and no phone number, email is your account recovery channel, and they have not registered a TOTP authenticator, then that user cannot complete sign-in at all. Cognito returns an error. They are not locked out temporarily. There is no path forward from the login screen.

Now think about who that describes in a healthcare deployment. Part-time nursing staff enrolled from an HR export that had work email but no mobile number. Older patients who gave an email at registration and nothing else. Contractors. Locums. It is never a small number, and they all arrive at the help desk in the same week.

The same rule cuts the other way on password reset. A user whose preferred MFA is email cannot receive a reset code by email. If they have no phone number, self-service recovery is gone, and someone with administrative credentials has to intervene for every single one.

Three ways out, in the order I’d reach for them:

• Make TOTP the default factor. An authenticator app doesn’t consume either the email or phone attribute, so the collision never occurs. This is the one I reach for first in staff-facing apps.
• Require both email and phone_number as sign-up attributes. When a user has both, Cognito automatically routes reset codes to whichever channel isn’t their MFA factor. Clean, but it forces a phone number on people who may not want to give one.
• Configure a recovery fallback. Set account recovery to prefer email but fall back to SMS, so users who aren’t eligible for email recovery still have a route.

Whatever you choose, run a query against your existing user records for accounts missing a phone number before you flip MFA to required. Finding those users in a report is cheap. Finding them through a help desk queue is not.

Failure family three: required MFA and adaptive authentication are mutually exclusive

This trips up almost every security-conscious design I review, because both features are things a healthcare client will ask for by name.

Cognito’s threat protection includes adaptive authentication: risk scoring on each sign-in, with automated responses like requiring MFA or blocking outright. To use those automated responses in full-function mode, MFA must be set to optional in the user pool. That is a documented constraint, not a bug. The mechanism only makes sense if there’s a state where MFA is not already being demanded.

So you cannot check both boxes. Either MFA is required for everyone and adaptive responses have nothing to escalate to, or MFA is optional and risk scoring can escalate, which means low-risk sign-ins may proceed on a password alone.

For anything touching ePHI, I’d take required MFA and give up the adaptive escalation. A control that always applies beats a control that usually applies, and “usually” is a hard thing to defend in an audit.

You don’t have to give up the visibility, though. Threat protection runs in audit-only mode alongside required MFA. You still get risk signals, device fingerprinting, and compromised credential detection surfaced in CloudWatch and in the user event logs, without the enforcement conflict. That’s a genuinely good trade: enforcement from the required setting, detection from threat protection.

Two things to know before you budget for it. Threat protection lives in the Plus feature tier, and audit-only mode is billed the same as full-function. There’s no free observation period. Export those events somewhere you’ll actually look at them, whether that’s a CloudWatch Logs destination feeding your SIEM, or a platform like Datadog or Grafana Cloud if that’s where your on-call already lives.

Failure family four: the session outlives the shift

MFA protects the moment of sign-in. It does nothing about the eight hours after it. In healthcare that gap is unusually expensive, because the workstation is shared, the browser stays open, and the next person to touch the keyboard is a different human being.

Cognito’s defaults are built for consumer apps. Access and ID tokens last one hour. Refresh tokens last thirty days. Neither default is wrong in general and both are wrong for a clinical workstation.

The ranges you have to work with:

• Access token and ID token: five minutes to one day, set per app client.
• Refresh token: sixty minutes to ten years, set per app client. It cannot be shorter than the access token lifetime.

Now the part that catches people, and the reason I put this failure family on the list. If you use managed login, Cognito sets a browser session cookie valid for one hour. Setting your access token to fifteen minutes does not shorten the re-authentication window. During that hour the user’s session refreshes without presenting credentials again, because the cookie is still good. Teams configure short tokens, write “15-minute session timeout” in the security documentation, and the observable behavior is an hour.

The second trap is revocation. Revoking a token, or a global sign-out, invalidates refresh tokens. It does not invalidate an access token that has already been issued. That token stays valid until it expires, and any API accepting it will keep accepting it. So the practical blast radius of a compromised session is your access token TTL, not the moment you clicked “sign out this user.”

That’s the argument for short access tokens: it is the only lever that shortens the window between “we revoked access” and “access actually stopped.”

# Tighten an app client used by clinical staff.
# Access tokens live 15 minutes; refresh tokens expire at the end of a shift.
aws cognito-idp update-user-pool-client 
  --user-pool-id us-east-1_EXAMPLE 
  --client-id EXAMPLECLIENTID 
  --access-token-validity 15 
  --id-token-validity 15 
  --refresh-token-validity 9 
  --token-validity-units AccessToken=minutes,IdToken=minutes,RefreshToken=hours 
  --enable-token-revocation

Refresh token rotation is worth turning on where your client can support it. Each refresh returns a new refresh token and invalidates the old one, so a stolen refresh token is usable once at most. Two caveats that bite in practice: rotation does not extend the original lifetime, so a nine-hour refresh token still dies nine hours after sign-in no matter how often it rotated, and app clients with rotation enabled cannot use the older REFRESH_TOKEN_AUTH flow. That second one has broken more than one mobile release. Rotation requires the Essentials tier or higher.

None of this replaces workstation-level controls. Screen lock policy, OS session timeouts, and endpoint hardening tooling like the utilities from O&O Software do work your identity layer cannot do from the outside. If staff reach the admin console remotely, a managed VPN such as NordLayer or Surfshark reduces network exposure, but be honest in your documentation about what that is. It is not an authentication control, and treating it as one is exactly the reasoning auditors push back on.

Failure family five: the factor you picked is phishable

Cognito offers SMS, email, and TOTP as second factors, plus passkeys as a first factor under specific conditions. They are not equivalent, and the console presents them as though they were.

SMS is the weakest. SIM swap and SS7 interception are real, and an SMS code is trivially relayed through a phishing proxy. It has one genuine advantage worth naming: enrollment friction is near zero, and for elderly or low-technology-comfort patient populations that is not a small thing. An SMS factor that gets used beats a TOTP factor that gets abandoned. Just don’t put clinical staff behind it. There’s also an operational trap: if your account is in the Amazon SNS SMS sandbox in the relevant region, you must verify each destination number before delivery works, which turns a smooth rollout into a stalled one.

Email OTP is roughly SMS-equivalent in phishing resistance and worse if the mailbox is on the same laptop as the session you’re protecting. It needs the Essentials tier or higher and a configured Amazon SES sending identity. Useful as a fallback, weak as a primary.

TOTP is the sensible default for staff. It’s phishable, the code can be relayed in real time by an attacker-in-the-middle, but it’s immune to SIM swap and it doesn’t collide with your account recovery channel. Any standard authenticator works. If your organization already uses a password manager with built-in TOTP, such as 1Password or Bitwarden, enrollment gets meaningfully easier, at the cost of both factors living in one vault. That’s a trade-off to make deliberately rather than by accident.

Passkeys are the only phishing-resistant option here, and their integration has a shape you need to understand. A passkey cannot be a second factor on top of a password in Cognito. It can satisfy the MFA requirement on its own, as a first factor, when you set FactorConfiguration to MULTI_FACTOR_WITH_USER_VERIFICATION in the pool’s WebAuthn configuration. The user verification step, biometric or device PIN, is what makes it count as multi-factor.

That’s the path I’d push for privileged accounts: administrators, anyone with bulk record export, anyone who can change the pool configuration itself. Platform authenticators cover most staff; hardware keys like YubiKey are worth the cost for the handful of accounts whose compromise ends the conversation.

One hard constraint to plan around: you cannot set MFA to required in a pool that allows email or SMS one-time passwords as a first factor. Passwordless OTP sign-in and required MFA are incompatible. Passkeys with user verification are the documented exception.

Wiring up Cognito MFA for healthcare apps

An order of operations that avoids locking people out. Do this in a non-production pool first, with synthetic users only.

1. Accept the BAA in the AWS account through AWS Artifact, before any real user data lands in the pool. The BAA is per-account and is not inherited by new accounts.
2. Audit your user records for accounts missing phone_number or email. This is the population that will fail after you enable required MFA.
3. Decide the recovery configuration and set it explicitly, with a fallback channel, so the MFA factor and the recovery channel can never be the same for any user.
4. Enable MFA methods with the pool set to optional first. Let users enroll voluntarily. Track enrollment coverage as a number you report weekly.
5. Flip to required only when coverage is high and the stragglers have a support path. Managed login will prompt unenrolled users to set up a factor at next sign-in, but only when MFA is required.
6. Tighten token lifetimes per app client, then verify the observable behavior rather than the configuration. If you use managed login, remember the one-hour cookie.
7. Turn on threat protection in audit-only mode and stream the events out to wherever you review security signal.

The core enforcement setting, once you’re ready:

# Require MFA pool-wide, TOTP as the available method.
aws cognito-idp set-user-pool-mfa-config 
  --user-pool-id us-east-1_EXAMPLE 
  --mfa-configuration ON 
  --software-token-mfa-configuration Enabled=true

# Set a specific user's preferred factor administratively.
aws cognito-idp admin-set-user-mfa-preference 
  --user-pool-id us-east-1_EXAMPLE 
  --username clinician@example.org 
  --software-token-mfa-settings Enabled=true,PreferredMfa=true

Note that admin-set-user-mfa-preference does not reset an existing TOTP registration. To issue a new authenticator secret for a user who lost their phone, you go through AssociateSoftwareToken and VerifySoftwareToken. Build that runbook before you need it, because you will need it, and improvising an identity-proofing procedure at 2pm on a Tuesday is how bad help desk habits start.

A staging pool holding only synthetic patients doesn’t need to live inside the BAA account, and there’s no reason a test environment should carry production’s cost structure. A plain VPS from InterServer or Contabo is fine for the app tier in front of a throwaway pool, provided you are certain no real PHI can reach it. That certainty is the whole condition, and it’s worth writing down rather than assuming.

Troubleshooting

A user gets an error at sign-in and never reaches an MFA prompt

Almost always the recovery collision. Check whether the account has both email and phone_number, and compare the populated attribute against the pool’s account recovery setting. If the only available attribute is also the recovery channel, and there’s no TOTP registration, the user has no eligible factor. Register TOTP for them or add the missing attribute.

Sign-in succeeds without MFA even though the pool says required

Work backward from the user, not from the pool. Is this a federated user? Then Cognito is not in the authentication path and pool MFA does not apply. Is it their first sign-in after account creation? Cognito treats the initial verification message as the second factor on that first sign-in and prompts for factor registration afterward. Is a different app client involved with different permitted flows?

Password reset stopped working after enabling MFA

Expected behavior, not a regression. The reset code cannot go to the same channel as the user’s MFA factor. Users with only one contact attribute lose self-service recovery. Add the second attribute or configure a fallback recovery mechanism.

SMS codes never arrive

Check the Amazon SNS SMS sandbox status in the region holding your SNS resources. In the sandbox, destination numbers must be verified in SNS before delivery. Also confirm the IAM role Cognito assumes for SNS is present and correctly scoped.

Repeated MFA failures lock a user out

After five unsuccessful MFA code attempts, Cognito starts an exponential timeout lockout. It resolves on its own. Most real cases are clock drift on the user’s phone breaking TOTP validation, so check that before assuming an attack.

Tokens keep working after you revoked them

Correct and by design. Revocation kills refresh tokens; already-issued access tokens survive to their expiry. If that window is unacceptable for your risk profile, shorten the access token TTL. There is no other lever short of checking revocation on every API call.

Common mistakes

• Treating the pool-level MFA setting as coverage for federated users. It is not, and this is the gap most likely to survive to production.
• Storing PHI in Cognito custom attributes. Diagnosis codes, MRNs, and care team assignments do not belong in a directory record that gets copied into ID token claims, which then land in browser storage, proxy logs, and error reports. Keep the identity layer to identity.
• Enabling required MFA in one deploy without an enrollment ramp. You will manufacture a support incident.
• Assuming a short access token gives a short session when managed login is in play. The one-hour cookie governs re-authentication.
• Believing revocation is immediate.
• Leaving threat protection off because required MFA is on. Audit mode and required MFA coexist, and the detection signal is separately valuable.
• Enabling refresh token rotation without checking whether your client uses REFRESH_TOKEN_AUTH. That combination is not permitted and it breaks at runtime.

Best practices

• Separate app clients for patient-facing and staff-facing applications, with independently tuned token lifetimes. Staff sessions should not inherit consumer defaults.
• TOTP as the baseline factor for anyone with access to more than their own record.
• Passkeys with user verification for administrators and bulk-export roles.
• Both email and phone_number collected at enrollment where the population allows it, purely to keep recovery working.
• Access token TTL short enough that revocation means something. Fifteen minutes is a reasonable starting point for clinical apps.
• Threat protection in audit mode, with events exported and actually reviewed.
• Manage the pool in Terraform or OpenTofu so MFA settings, recovery configuration, and token lifetimes are reviewable in a diff instead of discovered in the console during an audit.
• Test the MFA path in CI against a non-production pool. Configuration drift in an identity provider is silent by nature.

FAQ

Is Amazon Cognito HIPAA compliant?

No service is HIPAA compliant by itself. Cognito is HIPAA eligible and covered under the AWS Business Associate Agreement, which means you may process PHI with it once you have accepted the BAA in that account and configured it appropriately. Compliance is a property of your whole system and your documentation, not of the service.

Does HIPAA require MFA today?

Not explicitly. The current Security Rule requires person or entity authentication and leaves the method to your risk analysis. The proposed 2025 update would make MFA an express requirement, but it remains proposed and the timeline for final action has moved more than once.

Can I use required MFA and adaptive authentication together?

Not in full-function mode. Adaptive authentication’s automated responses require MFA to be optional in the user pool. You can run threat protection in audit-only mode alongside required MFA, which keeps the detection signal without the conflict.

Which MFA factor should healthcare apps use?

TOTP as the baseline for staff, passkeys with user verification for privileged accounts. SMS is defensible for patient populations where enrollment friction genuinely determines whether MFA gets used at all, but it should not protect clinical access.

Does Cognito MFA apply to users signing in through our hospital SSO?

No. For federated users arriving via SAML or OIDC, Cognito delegates authentication entirely to the upstream identity provider and does not add a second factor. Enforcement has to happen at the IdP, and you should get that in writing.

How short should session timeouts be for a clinical application?

Set access tokens short, commonly around fifteen minutes, and refresh tokens to roughly a shift length rather than the thirty-day default. Then verify what users actually experience, because managed login’s one-hour browser cookie can keep a session alive past your token settings.

Can I store patient identifiers in Cognito user attributes?

You can, and I would avoid it. Custom attributes flow into ID token claims, which end up in browser storage, application logs, and error tracking. Keep an opaque internal identifier in the token and resolve clinical data from your application database behind its own authorization check.

Conclusion

If you remember one thing about Cognito MFA for healthcare apps, make it this: the MFA setting is a claim about local users at the moment of sign-in, and nothing else. It says nothing about your federated clinicians, nothing about the session that continues for the next eight hours, and nothing about whether the factor you chose survives a phishing proxy.

Verify enforcement by observing behavior, per user population, per app client. Not by screenshotting the console page. That screenshot is what got flagged in the pen test at the top of this post, and it was completely accurate.

Need help with Cognito and healthcare authentication?

I work with teams building and hardening authentication on AWS, particularly where a compliance obligation makes “it works” insufficient. Things I can help with:

• Auditing an existing user pool for the gaps in this post: federated coverage, recovery collisions, token lifetimes, app client sprawl
• Designing an MFA rollout that won’t generate a help desk queue, including the pre-flight query against your user records and the enrollment ramp
• Implementing passkeys with user verification for privileged accounts, alongside TOTP for general staff
• Moving user pool configuration into Terraform or OpenTofu so identity settings are reviewable and drift is visible
• Wiring threat protection events into your existing monitoring and building the alerts that are worth waking someone for
• Writing the technical evidence that maps your configuration to your risk analysis, in language an auditor accepts

If you have a user pool that isn’t behaving the way the console suggests it should, send me the output of describe-user-pool and describe-user-pool-client with the identifiers stripped, and I’ll tell you what I see.

The post Cognito MFA for Healthcare Apps: The Settings That Look Compliant but Aren’t appeared first on John Nessime.

The fix looked obvious. Drop in a limit_req_zone, apply it, reload, done. Except the reload changed nothing at all, and the reason took an embarrassingly long time to find.

This post covers Nginx rate limiting and bot mitigation the way it actually behaves in front of a real site: how the limiting modules key requests, what burst and nodelay really do to your traffic shape, why a limit that looks correct in the config can be doing nothing (or throttling everyone at once), and how to layer bot handling so you are not playing whack-a-mole with user agent strings forever.

The failure that hides: your limit is keyed on the wrong address

This is the one that bites hardest, because the config parses, nginx -t passes, and the error log stays quiet. Nothing tells you it is wrong.

If anything sits in front of Nginx (Cloudflare, another CDN, a load balancer, or a local Varnish tier), then $remote_addr is the address of that proxy, not the visitor. Key a rate limit on it and you get one of two outcomes, both bad:

• Every visitor arriving through a given edge node shares a single bucket. One aggressive client burns the budget and legitimate users start collecting 503s.
• Or your rate is generous enough that the shared bucket never fills, in which case the limit is decorative and the abusive client sails straight through.

The real IP module fixes this, but only if you trust the right sources. It rewrites $remote_addr and $binary_remote_addr from a header, and only for connections coming from addresses you have explicitly declared trustworthy.

# http context. One entry per CDN range you actually receive traffic from.
set_real_ip_from 203.0.113.0/24;
set_real_ip_from 2001:db8::/32;

real_ip_header CF-Connecting-IP;   # or X-Forwarded-For for a generic proxy
real_ip_recursive on;

Two things matter here more than the syntax.

First, never widen set_real_ip_from to 0.0.0.0/0. That tells Nginx to believe a client-supplied header from anyone, which means an attacker can pick a new “client IP” on every request and your per-IP limit becomes unlimited. It also poisons your logs, your deny rules and anything downstream that trusts the forwarded address.

Second, after the real IP module has run, $binary_remote_addr holds the visitor address and $realip_remote_addr holds the original connecting address (the proxy). Key your limits on $binary_remote_addr. Keying on $realip_remote_addr puts you right back in the shared-bucket failure, silently.

The proxy ranges change over time, so treat the trust list as a file that gets regenerated on a schedule and reloaded, not something you paste in once. On a self-managed VPS from a provider like Contabo or InterServer, a small cron job that fetches the current ranges, rewrites an include file, runs nginx -t, and only reloads on success is about twenty lines of shell and saves you from a very confusing afternoon later.

What limit_req actually does, and what burst really means

Nginx rate limiting for requests uses a leaky bucket. The zone stores, per key, the current number of excessive requests. The rate defines how fast that excess drains. Two directives do the work: limit_req_zone declares the bucket in the http context, limit_req applies it in http, server or location.

http {
    limit_req_zone $binary_remote_addr zone=general:10m rate=10r/s;
    limit_req_zone $binary_remote_addr zone=login:10m   rate=20r/m;

    limit_req_status 429;
    limit_req_log_level warn;
}

The rate is expressed in requests per second (r/s) or requests per minute (r/m). If you want less than one request per second, use r/m. There is no native per-hour or per-day unit in the stock module, which surprises people writing password-reset limits.

burst without nodelay is a queue, not a rejection

This is the second-most common misunderstanding after the keying problem. By default, excessive requests are delayed so that they come out at the configured rate. They are only rejected once the number waiting exceeds burst.

location /search/ {
    limit_req zone=general burst=20;
}

At 10r/s, a client that fires 21 requests at once gets one served immediately, nineteen held and released over roughly two seconds, and the last one rejected. Those nineteen held requests are still open connections on your server. You have not reduced concurrency, you have converted a spike into a slow trickle and paid for it in worker slots and file descriptors.

That is genuinely useful for smoothing traffic to a fragile upstream. It is a bad idea for abuse control, where holding the connection is exactly what the attacker wants.

nodelay, delay, and picking between them

nodelay changes the behaviour to: serve everything in the burst window immediately, reject everything past it. The bucket still drains at the configured rate, so a client that sustains the burst rate will hit the wall and stay there.

location /wp-login.php {
    limit_req zone=login burst=5 nodelay;
    limit_req_status 429;
    # ...fastcgi_pass or proxy_pass here
}

The delay parameter is the middle ground: it sets the point within the burst at which requests start being delayed rather than served immediately. burst=20 delay=10 means the first ten excess requests go straight through, the next ten get queued, and anything beyond is rejected.

How I decide: if a human clicking around your site could plausibly generate the spike, use nodelay or a delay that covers it. A single page load pulling twelve uncached assets through Nginx is one “request” to the user and twelve to the limiter. If the endpoint is machine-facing and expensive (search, export, login, an API write path), tighten the rate, keep the burst small, and use nodelay so rejection is fast and cheap.

Also set limit_req_status 429. The default is 503, which tells clients and monitoring that your server is broken rather than that they are being throttled. 429 is the correct semantic, and well-behaved clients back off on it.

Sizing the zone, and the number most guides get wrong

You will see “10m holds about 160,000 IP addresses” repeated everywhere. That is the 32-bit figure. Nginx documents the stored state for limit_req as 64 bytes on 32-bit platforms and 128 bytes on 64-bit platforms, so one megabyte holds roughly 16 thousand states on 32-bit and roughly 8 thousand on 64-bit. On the 64-bit server you are almost certainly running, a 10m zone is closer to 80,000 states.

This matters because of what happens when the zone fills. Nginx evicts the least recently used state. If it still cannot allocate, the request is terminated with your configured error status. A zone that is constantly thrashing under a distributed crawl will start returning errors for reasons that have nothing to do with the rate you configured. If you are limiting a high-cardinality key, give the zone room.

limit_conn solves a different problem, and you probably need both

limit_req caps how fast requests arrive. It does nothing about a client that opens sixty connections and reads each response one byte at a time. That is a concurrency problem, and it belongs to limit_conn.

http {
    limit_conn_zone $binary_remote_addr zone=perip:10m;
    limit_conn_status 429;
}

server {
    limit_conn perip 20;

    location /downloads/ {
        limit_conn perip 4;
        limit_rate_after 2m;
        limit_rate 512k;
    }
}

Three behaviours worth knowing before you pick a number.

• Not every connection counts. Nginx only counts a connection once the full request header has been read and the request is being processed. A client stalling mid-header is not counted here, which is why client_header_timeout and client_body_timeout still matter.
• Under HTTP/2 and HTTP/3, each concurrent request counts as a separate connection. A limit of 4 that felt generous on HTTP/1.1 can break a modern browser loading a page with many assets. This is a very common self-inflicted outage.
• The limit_conn state is smaller than the limit_req state, so the same zone size holds roughly twice as many entries.

The pairing is the point: limit_req protects against volume, limit_conn protects against occupancy. Rate limiting alone leaves you open to slow-read exhaustion, and connection limiting alone does nothing about a client making one fast request after another on a keepalive connection.

Choosing keys: IP is a starting point, not the answer

The key can be any variable or combination of variables. That flexibility is where most of the value is, because per-IP limits have a real accuracy problem in both directions.

On one side, carrier-grade NAT, corporate egress, university networks and shared VPN exit nodes (NordVPN, Surfshark and every other provider works this way) put many genuine users behind one address. A tight per-IP limit will hit them. On the other side, a distributed scraper renting a few hundred cheap addresses never touches a per-IP limit at all, because no single address is doing much.

So layer keys instead of tuning one harder:

limit_req_zone $binary_remote_addr zone=perip:20m    rate=10r/s;
limit_req_zone $server_name        zone=persite:1m   rate=200r/s;

Applied together, the per-IP zone catches the loud individual and the per-site zone caps total damage regardless of how many addresses are involved. When several limit_req directives match, all of them are evaluated and the most restrictive result wins. If any one of them rejects, the request is rejected.

One inheritance rule catches people out: limit_req directives are inherited from the enclosing level only if the current level defines none. Add a single limit_req inside a location and you have dropped every limit set at server level for that location. If you want both, restate both.

Allowlisting with an empty key

Requests whose key evaluates to an empty string are not accounted at all. That is the clean way to exempt monitoring probes, your office, or a payment webhook source without maintaining a parallel set of locations.

geo $limit {
    default        1;
    10.0.0.0/8     0;
    192.0.2.10/32  0;
}

map $limit $limit_key {
    0 "";                    # empty key, never limited
    1 $binary_remote_addr;
}

limit_req_zone $limit_key zone=perip:20m rate=10r/s;

Nginx evaluates map variables only when they are used, so declaring a large map costs nothing on requests that never reference it.

Roll it out in dry run, not in production

Both modules have a dry run mode. Excessive requests are counted in the shared memory zone and logged exactly as they would be, but nothing is actually rejected or delayed. Combined with the status variables, this gives you a real answer to “would this limit have hurt anyone” before you find out from a customer.

log_format ratelimit '$remote_addr $status "$request" '
                     'req=$limit_req_status conn=$limit_conn_status '
                     'ua="$http_user_agent"';

server {
    limit_req_dry_run  on;
    limit_conn_dry_run on;

    access_log /var/log/nginx/ratelimit.log ratelimit;
}

$limit_req_status reports PASSED, DELAYED, REJECTED, DELAYED_DRY_RUN or REJECTED_DRY_RUN. $limit_conn_status reports PASSED, REJECTED or REJECTED_DRY_RUN. A numbered rollout that has never embarrassed me:

1. Add the zones and the limit_req directives with dry run on, plus the log format above.
2. Run for a full traffic cycle. A week if you have weekly patterns, longer if you have a monthly billing spike.
3. Count the would-be rejections by user agent and by address. If real browsers appear, your rate or burst is wrong, not your users.
4. Turn dry run off on the least critical location first. Watch 429 rates alongside your normal error budget.
5. Only then apply it to login, checkout, or anything that costs you money when it breaks.

Shipping the rate limit log into Loki and graphing it in Grafana makes step three take minutes instead of an evening with awk. Any log platform with decent label cardinality handling works; the point is having rejection counts broken down by zone and user agent, over time, next to your latency graphs.

Bot mitigation in layers, cheapest first

Rate limiting treats all clients the same. Bot mitigation is about deciding that some clients deserve different treatment before the limiter ever sees them. The useful mental model is a set of layers ordered by cost and by how easy each is to defeat.

Layer one: robots.txt, and being honest about what it does

robots.txt is a request, not a control. Crawlers that respect it will honour a Disallow on your search endpoint, your faceted filter URLs and your infinite calendar pages, and that alone removes a surprising amount of load. Crawlers that do not respect it will ignore the file completely, and some will read it specifically to find out where the interesting URLs are.

Use it to shape well-behaved traffic. Never use it as a security boundary. And be careful about blanket disallows on search engines you actually want: deindexing yourself is a much more expensive mistake than a bit of crawler load.

Layer two: user agent classification with map

A map on $http_user_agent is cheap, readable, and easy to revise. It is also trivially spoofable, so treat it as traffic shaping rather than enforcement.

map $http_user_agent $ua_class {
    default          human;
    ""               empty;
    "~*bot|crawler|spider|scrape"  bot;
}

map $ua_class $bot_limit_key {
    bot   $binary_remote_addr;
    empty $binary_remote_addr;
    human "";
}

limit_req_zone $bot_limit_key zone=bots:10m rate=30r/m;

That gives declared bots a much tighter budget than browsers while leaving human traffic on the general limit. An empty user agent is worth its own class: almost nothing legitimate sends one, and it is a reliable signal for cheap scraping tools.

Two cautions. Broad substring matching on bot will catch strings you did not intend, so review a sample of your own logs before deploying the pattern. And blocking rather than throttling is a bigger decision than it looks: if you return 403 to a crawler you actually want, you find out weeks later in your search traffic, not in your error log.

Layer three: verifying bots that claim a name

The interesting case is a request claiming to be a major search crawler. Impersonating one is the standard way to bypass a user agent based allowlist, so a claim needs verification.

The method the search vendors themselves document is a forward-confirmed reverse DNS check: reverse-resolve the client address, confirm the resulting hostname belongs to the vendor’s domain, then forward-resolve that hostname and confirm it returns the original address. An attacker cannot fake this without controlling the vendor’s DNS. Several vendors also publish their crawler IP ranges as machine-readable lists, which avoids the DNS round trip entirely.

Here is the honest limitation: stock Nginx has no directive that does this. There is no built-in reverse DNS verification in the core modules. Your realistic options are a third-party module, doing the verification out of band and feeding the result into a geo block or an included allowlist file, or pushing the decision up to a CDN or WAF layer that already does it. Anyone showing you a pure stock-Nginx config that “verifies Googlebot” is verifying a string, not an identity.

This is also the point where an edge provider earns its keep. Cloudflare, Fastly and similar edge platforms maintain verified bot lists and fingerprinting that no single origin server can reproduce, and they drop the traffic before it costs you a worker. If your bot problem is large and distributed, the edge is the right layer and Nginx is your backstop, not your front line.

Layer four: escalation to the firewall

A client that collects hundreds of 429s and keeps going is not going to stop. Serving it a rejection still costs you a TCP handshake, a TLS negotiation and a log line. At that point the cheapest response is to stop answering at all.

Fail2ban is the usual tool: a filter that matches your rate limit rejections in the Nginx error or access log, an action that inserts a firewall drop, and a ban time long enough to be inconvenient. Two things to get right. Match on a pattern that only fires for real limit rejections, or you will ban yourself during a deploy. And if you are behind a CDN, your firewall sees the CDN address, not the client, so the ban has to be applied at the edge instead. Banning a Cloudflare range at your origin firewall takes your whole site offline, and it is a mistake people make exactly once.

Return code 444 is worth knowing here. It is Nginx-specific and closes the connection without any response, which is cheaper than generating an error page for a client that will never read it.

Troubleshooting Nginx rate limiting

The limit does nothing. Check the key first. Log $binary_remote_addr alongside $realip_remote_addr and confirm they differ in the way you expect. Then check inheritance: a limit_req in the matched location discards the server level ones. Confirm which location actually matched by adding $request_uri and a marker header while testing.

Real users are getting 429 or 503. Turn dry run back on and read the log rather than guessing. Usually it is one of three things: a burst too small for a normal page load’s asset count, a limit_conn value set for HTTP/1.1 assumptions on an HTTP/2 server, or a shared-egress network where many people genuinely share one address.

Nothing appears in the error log. The default log level for rate limit refusals is error, and delays are logged one level below whatever you set. If you set limit_req_log_level warn, delays go to info, which your log level may be filtering out entirely. Grep for the limiting messages directly:

grep 'limiting requests' /var/log/nginx/error.log | tail -50
grep 'limiting connections' /var/log/nginx/error.log | tail -50

Intermittent errors under heavy load with no matching rate. Suspect zone exhaustion. Enlarge the zone and see whether the pattern changes. High-cardinality keys and distributed crawls are the usual cause.

Health checks and uptime probes are being throttled. Add them to the allowlist map. Monitoring hitting a rate limit produces alerts about an outage that is not happening, which is worse than no monitoring.

Common mistakes

• Trusting forwarded headers from every source, which turns a per-IP limit into no limit at all.
• Leaving limit_req_status at the default 503, so throttling looks like an outage in your own monitoring.
• Using burst without nodelay on an abuse path, and quietly filling worker slots with held connections.
• Setting a limit_conn value based on HTTP/1.1 intuition on a server serving HTTP/2 or HTTP/3.
• Adding one limit_req in a location and silently dropping every server-level limit for that path.
• Blocking user agent substrings without checking a sample of real logs first.
• Applying identical limits to reads and writes. A cacheable GET and a POST that writes to the database do not deserve the same budget.
• Banning CDN address ranges at the origin firewall because the logs show them as the client.

Best practices worth the effort

• Verify the key before tuning the rate. Every other decision depends on it being correct.
• Always start in dry run and measure a full traffic cycle before enforcing.
• Use separate zones per traffic class. Search, login, API, static and general traffic have different shapes and should not share a bucket.
• Pair limit_req with limit_conn. The first caps arrival rate, the second caps occupancy, and neither substitutes for the other.
• Layer a per-server zone above your per-IP zone so distributed traffic still hits a ceiling.
• Allowlist with an empty key rather than duplicating location blocks.
• Export $limit_req_status to your log platform and graph rejections next to latency and upstream errors.
• Keep CDN trust ranges in a generated include file that is refreshed on a schedule and validated with nginx -t before reload.
• Throttle before you block. Blocking failures are invisible until they show up in your search traffic.

Frequently asked questions

Does Nginx rate limiting stop a DDoS attack?

It helps with application-layer floods that reach your server, because rejecting a request is far cheaper than processing it. It does nothing about volumetric attacks that saturate your network link, since that traffic never gets to Nginx to be limited. For that you need capacity in front of the origin: a CDN or scrubbing service. Treat rate limiting as protection against abuse and accidental floods, not as DDoS mitigation.

What rate should I actually set?

There is no universal number, and any guide that gives you one has not seen your traffic. Measure instead: run dry run mode, look at your real per-IP request distribution, and set the rate above the busiest legitimate client with headroom. Then set a separate, much tighter zone for the specific endpoints that are expensive or attacked.

Should I use 429 or 503 for rejected requests?

429 in nearly every case. It communicates that the client should slow down, well-behaved clients back off on it, and it keeps your 5xx metrics meaningful. The default is 503 purely for historical reasons.

Can Nginx rate limit per hour or per day?

Not with the stock module. It accepts requests per second and requests per minute only, so the slowest native rate is one request per minute expressed as 1r/m. Longer windows need a third-party module, a shared store such as Redis via a scripting module, or enforcement in the application itself. For something like a password reset limit, the application is usually the better place anyway, because it can key on the account rather than the address.

Do rate limit zones work across multiple Nginx servers?

Not in the open source build. Each instance keeps its own shared memory zone, so a client spread across four load-balanced servers effectively gets four times the budget. The sync parameter on limit_req_zone synchronises state across a cluster, but it is part of the commercial subscription. Without it, either divide your intended rate by your instance count, ensure your balancer pins clients consistently, or move the limit to a shared store.

Will blocking AI crawlers hurt my SEO?

Blocking training crawlers is separate from blocking search indexing crawlers, and the distinction is worth checking per vendor before you write a rule, because some vendors run several crawlers with different purposes under related names. The risk is collateral damage: a broad user agent pattern that also matches a search crawler will remove you from results, and you will not notice quickly. Verify your pattern against real log samples, and prefer throttling over outright blocking while you are unsure.

Where should rate limiting live: the CDN, Nginx, or the application?

All three, doing different jobs. The CDN handles volume and known-bad reputation before it costs you anything. Nginx handles per-client shaping and protects your workers from anything that gets through. The application handles limits that need identity or business context, like per-account quotas, which Nginx cannot see. Skipping the Nginx layer because you have a CDN leaves you exposed the moment someone finds your origin address.

The one thing worth remembering

Nginx rate limiting is not hard to configure. It is easy to configure into a state where it silently does nothing, or silently punishes the wrong people, and both failures look identical from the outside: a config that parses, a clean nginx -t, and no complaints until there are.

So get the key right, prove it with a log line rather than assuming it, and run every new limit in dry run until the data tells you it is safe. Layer bot handling by cost, throttle before you block, and be honest about which layer can actually verify an identity. Everything else is tuning, and tuning is easy once the foundation is not lying to you.

Need a second pair of eyes on your Nginx edge?

Most of the work here is diagnostic rather than clever. Things I regularly help with:

• Auditing an existing limit_req and limit_conn setup to find out whether it is actually limiting anything, including the keying and inheritance traps above.
• Fixing real client IP handling behind Cloudflare or another CDN, with a generated trust list that refreshes safely instead of drifting.
• Designing per-endpoint zones for WordPress, WooCommerce or API origins so login, search and write paths get budgets that match their real cost.
• Running a measured dry run rollout and reporting back what would have been rejected, broken down by user agent and network, before anything is enforced.
• Building bot classification and escalation: user agent maps, allowlists, Fail2ban integration, and knowing which decisions belong at the edge instead.
• Wiring rate limit telemetry into Grafana and Loki so throttling is something you can see on a dashboard rather than discover from a ticket.

If you would like a look at yours, send me the relevant server block, a sample of the access log, and whatever is currently going wrong. That is usually enough to say something useful straight away.

The post Nginx Rate Limiting and Bot Mitigation Without Blocking Real Users appeared first on John Nessime.

The answer on screen was good. Fluent, specific, correctly hedged, three sources listed underneath it. Then a partner in the room asked a simple question: which matter is source three from? Nobody could answer it in the room, and when we went and looked, it was from a matter that half the people watching the demo were formally walled off from.

Nothing had crashed. No alarm fired. The retrieval layer had done exactly what a similarity search does, which is return the nearest vectors, and the nearest vectors did not care about the ethical wall. That is the thing about building a legal document intelligence platform on AWS: the failures that matter almost never look like failures. They look like a confident answer with a citation attached.

This post is about the parts of that build that are hard, organised around the five boundaries that actually have to hold. The extraction pipeline is the easy half. I will cover it, but quickly, because the AWS documentation is good and the failure modes are visible. The other four boundaries fail quietly, and that is where the engineering goes.

The shape of the thing

Before the boundaries, the skeleton. A legal document intelligence platform on AWS almost always ends up looking like this, whether you plan it or arrive at it:

• Documents land in Amazon S3, one prefix per matter, versioning on.
• An S3 event triggers AWS Lambda, which starts an asynchronous Amazon Textract job.
• Textract output lands back in S3 as JSON. Something normalises it into text plus page and bounding-box coordinates.
• Amazon Comprehend or a foundation model classifies the document and pulls entities: parties, dates, governing law, clause types.
• Chunks get embedded and written to a vector store, usually Amazon OpenSearch Serverless behind Amazon Bedrock Knowledge Bases.
• An application layer, typically API Gateway plus Lambda, takes a question, retrieves, and calls a model on Amazon Bedrock.

AWS Step Functions is worth reaching for once you have more than three stages, because Textract jobs are long-running and Lambda timeouts are not a retry strategy. That is the whole architecture. You can stand it up in a fortnight. Then you spend six months on everything below.

Boundary one: isolation, and why metadata filtering is not optional

This is the one from the demo, and it is the one that ends engagements.

A vector index has no native concept of a matter, a client, or an ethical wall. If you pool every document into one index and rely on the prompt to keep things separate, you have built a system where a well-phrased question can pull privileged material across a wall. Prompt instructions are not an access control. They are a suggestion that usually works.

There are two shapes that do work, and the choice between them is a real trade-off rather than a best practice.

Pooled index with server-side filtering

Every chunk carries a matter_id and a client_id as metadata. Every retrieval call attaches a filter. With an S3 data source in Bedrock Knowledge Bases, the metadata comes from a sidecar JSON file that sits next to the document and carries a metadataAttributes object.

{
  "metadataAttributes": {
    "matter_id": "M-4417",
    "client_id": "C-108",
    "doc_type": "engagement_letter",
    "privileged": true
  }
}

The retrieval call then narrows the search before the model ever sees a chunk:

"retrievalConfiguration": {
  "vectorSearchConfiguration": {
    "filter": {
      "equals": { "key": "matter_id", "value": "M-4417" }
    }
  }
}

Two rules make this safe, and both are the kind of thing that gets skipped under deadline. First, the filter value is derived on the server from the authenticated caller’s claims, never accepted from the client. If the browser can send you a matter_id, the browser can send you a different one. Second, the filter is applied by code that no feature request can bypass. A helper that builds every retrieval request, and a code review rule that no other path may call the retrieve API directly.

The trap here is ordering. You cannot filter on an attribute the index does not have. Add matter_id after ingestion and every chunk already in the index is invisible to that filter, which means it either returns for everyone or for no one, depending on how your filter is written. Design the metadata schema before the first ingestion run, not after the first demo.

Separate collection per client

The heavier option. A separate OpenSearch Serverless collection per client, which buys you a separate AWS KMS key per client, separate index settings, and a failure mode where a bug in the filtering logic cannot reach across clients at all.

It costs you sprawl. Every collection is a resource to provision, monitor, patch policy on, and eventually delete. Ingestion jobs run per data source per knowledge base, so freshness guarantees fragment. For a firm with twelve institutional clients this is fine. For a platform onboarding a hundred small clients it becomes the main operational burden of the product.

My default is pooled with strict server-side filtering, and per-client collections only where the client’s own contract demands a dedicated encryption key. If you cannot say out loud which of those two you are running, you are running neither properly.

Boundary two: extraction, and the confidence score everyone ignores

Textract returns a confidence score between 0 and 100 with each prediction. AWS is explicit in its own best-practice guidance that applications sensitive to detection errors should enforce a minimum threshold and route anything below it for human scrutiny, and that the right threshold depends entirely on how the output gets used.

Legal documents sit at the harsh end of that. A misread date on an archival scan is a curiosity. A misread date on a limitation period is a problem with a name on it.

Practical notes from this layer:

• Use the asynchronous operations for anything multipage. StartDocumentAnalysis and GetDocumentAnalysis handle long PDFs and TIFFs; the synchronous AnalyzeDocument path is for single pages and will fight you on a 400-page bundle.
• Textract’s Queries feature earns its place on structured instruments. Instead of extracting everything and grepping, you ask the document a direct question and get a scoped answer with its own confidence.
• Keep the page number and bounding box for every chunk. When a lawyer asks where a sentence came from, “page 14, second column” is an answer. “It’s in the corpus somewhere” is not, and the platform loses credibility the first time you say it.
• Amazon Augmented AI wires low-confidence predictions into a human review queue with a private workforce, so the reviewers are your people rather than an anonymous pool. For privileged material that distinction is the entire point.

What I would skip early: building a custom entity recognition model. Comprehend supports custom entity recognition and it is genuinely useful, but you need labelled data you do not have yet, and a foundation model with a decent prompt gets you far enough to find out which entities the users actually care about. Train the custom model once the requirement has stopped moving.

Boundary three: retention, where deletion is a distributed problem

Here is the failure that will not show up in any test suite you write.

A client asks for a document to be removed. Someone deletes the S3 object. The ticket closes. The document is still fully searchable, because its chunks are sitting in the vector index and will stay there until the data source is re-synced. Meanwhile the Textract JSON output is in a second bucket, the extracted text may be in a database, and if model invocation logging is on, whole passages of the document are sitting in CloudWatch Logs or another S3 bucket entirely.

One delete, at least four places holding a copy. Write the deletion path as a real workflow with an assertion at the end, not as a single API call:

1. Delete the source object and, if versioning is on, its versions.
2. Delete the derived artefacts: Textract JSON, normalised text, any thumbnails or page images.
3. Trigger an ingestion job so the knowledge base drops the orphaned chunks.
4. Confirm removal by querying the index for the document identifier and expecting nothing back.
5. Deal with the logs, which means either a retention policy short enough to make the problem expire or a deliberate decision, written down, that logs are out of scope.

Step four is the one people leave out, and it is the only step that actually proves anything.

The opposite problem: things that must not be deleted

Legal work has the reverse requirement too. When a matter goes into litigation hold, the documents need to survive an administrator with delete permissions and a bad afternoon.

S3 Object Lock is the mechanism, and it has two independent controls that people routinely conflate. A retention period protects an object version until a fixed date, in either governance mode, which privileged users can override, or compliance mode, which nobody can override and where the period cannot be shortened. A legal hold has no date at all. It stays until someone with s3:PutObjectLegalHold explicitly removes it. The two can be active at once, and while either is active the object version cannot be deleted or overwritten.

# Place an indefinite hold on a specific object version
aws s3api put-object-legal-hold 
  --bucket matters-archive 
  --key M-4417/exhibit-c.pdf 
  --version-id 3sL7f2Qz9pXvB1kR 
  --legal-hold Status=ON

Two things bite here. Object Lock requires versioning and turns it on automatically, and once enabled on a bucket you cannot turn it off or suspend versioning again. And compliance mode is genuinely permanent: if you set a seven-year retention when you meant seven days, that is the answer, for everyone, including the account root. Test in governance mode first. The bypass path exists and is deliberately awkward, requiring both the s3:BypassGovernanceRetention permission and an explicit x-amz-bypass-governance-retention:true header on the request, which is exactly the level of friction you want on that operation.

Design the hold model before you design the deletion model, because holds win. A deletion request that collides with an active hold is a legal question, not an engineering one, and the platform’s job is to surface the collision clearly rather than resolve it silently.

Boundary four: disclosure, or what leaves the account

This is the boundary the client’s general counsel will ask about, usually in writing, usually before signature. Three separate controls, and they are not interchangeable.

The AI services opt-out policy

AWS Organizations has a governance policy type that opts your accounts out of having content processed by certain AI services stored and used for service improvement. The list includes Amazon Textract and Amazon Comprehend, which are precisely the two doing the reading in this architecture. The policy type has to be enabled at the organisation root before you can attach anything:

aws organizations enable-policy-type 
  --root-id r-example 
  --policy-type AISERVICES_OPT_OUT_POLICY

Read the AWS documentation on this one carefully rather than taking my summary as gospel, because there is a caveat in it that matters: the services may still need to store your data operationally even when you have opted out of it being used for improvement. Opting out is not the same as the data never existing outside your account. Say that plainly in the client conversation. It is a much better position than being asked about it later.

Bedrock’s data position, and the log that undoes it

Amazon Bedrock’s published position is that inputs and outputs are not shared with third-party model providers and are not used to train the base models, and that fine-tuning operates on a private copy. That is a strong starting point for privileged content and it is the main reason Bedrock rather than a direct provider API shows up in these builds.

Then there is model invocation logging. It captures full prompt and response payloads to CloudWatch Logs or S3, and you want it on, because without it you cannot reconstruct what the system told someone. But understand what you have just built: a log that contains privileged document text, at a per-region setting, in a destination that probably has looser access controls than the document store it came from. Encrypt it with the same KMS key discipline, restrict it harder than you think you need to, and set a retention period on purpose.

Worth separating in your head: CloudTrail records that an API call happened and who made it. Model invocation logging records what was in it. You need both, for different questions, and only one of them contains client confidences.

Network path

Interface VPC endpoints via AWS PrivateLink keep traffic to Bedrock, Textract and the rest on the AWS network rather than out through an internet gateway or NAT. Add a gateway endpoint for S3 while you are there.

Endpoint policies are the part that gets forgotten. An endpoint without a policy is a private path to the whole service, including into accounts that are not yours. Scope it to the operations and resources you actually use, and pair it with an S3 bucket policy that rejects requests not arriving through your endpoint. The first protects what leaves; the second protects what can be reached.

The parts that are not on the architecture diagram

Two of these have caught people out badly, and neither is an AWS control.

The first is the development environment. You do not want real client documents on a laptop or in a scratch account, so build the parsing and chunking code against synthetic documents on a small separate box. A cheap VPS from a provider like Contabo or InterServer is fine for this, and the separation is worth more than the convenience of iterating in the production account.

The second is the reviewers. If your human review loop involves contractors working remotely, their network path and their disks are part of your boundary whether or not you drew them. A managed VPN such as NordVPN or Surfshark handles the network side, and when a matter closes and local copies have to go, a dedicated erasure tool from something like O&O Software does what dragging a folder to the bin does not. Unglamorous, and it is the layer that gets audited.

Boundary five: evidence, because someone will ask

At some point the question stops being “does it work” and becomes “who saw what, and when”. If you cannot answer that from stored records, the platform is not defensible regardless of how good the answers are.

• Turn on CloudTrail data events for the document buckets. They are off by default, they are billed separately from management events, and they are the only way to see individual object-level reads.
• Log the retrieval, not just the generation. Store which chunks came back and which filter was applied. When somebody asks whether the wall held on a specific query, this record is the answer and there is no reconstructing it later.
• Carry the end user’s identity through to the audit record. A Lambda execution role in the logs tells you the platform did something. It does not tell you who asked.
• Alarm on the filter, not just on errors. A retrieval that ran without a matter filter should page someone, even though it returned HTTP 200 and a perfectly good answer.

That last one is the closest thing to a single takeaway in this post. The dangerous events in this architecture are successful ones.

Troubleshooting the things that will go wrong

Retrieval returns nothing after you add a filter

Nearly always a metadata problem rather than a query problem. Either the sidecar metadata file was missing at ingestion, so the chunks carry no attribute to match, or the attribute was added after the chunks were indexed. Check whether the attribute exists on a known chunk before you touch the query. If it does not, you need a re-sync, not a different filter expression.

Textract returns text but the layout is scrambled

Common on two-column pleadings and on documents with headers and footers on every page. Raw text detection reads in a reading order that is not always yours. Use the layout and table features rather than plain detection, and hold onto the geometry so you can reassemble columns yourself if the default order is wrong for that document class.

The model cites a document that does not exist

Usually the citation is being generated rather than passed through. If document names are being written by the model instead of read from the retrieval response, it will invent plausible ones. Build citations from the retrieval result metadata in your application code and never from the generated text. Bedrock Guardrails helps with contextual grounding, but the structural fix is not asking the model to produce the reference in the first place.

Ingestion is slow and nobody knows where

Instrument per stage before you optimise anything. In most of these pipelines the time is in Textract for large scanned bundles, and the fix is parallelism at the document level rather than tuning the extraction itself. Step Functions with a distributed map over documents gets you there without a queue you have to babysit.

Costs climb faster than volume

Look at re-processing first. A pipeline that re-extracts a document every time anything downstream changes will quietly multiply your Textract bill against a static corpus. Key the extraction cache on the object’s content hash, not its path, and make re-ingestion an explicit action. The billing mechanisms differ per service, so read the current pricing pages rather than trusting a number from a blog post.

Common mistakes

• Treating the system prompt as an access control. It is a formatting instruction that happens to look like a rule.
• Taking the tenant identifier from the request body. If the client can name the matter, the client can name someone else’s.
• Designing the metadata schema after the first ingestion run. Retrofitting a filterable attribute means re-indexing the whole corpus.
• Assuming a deleted S3 object is gone from the platform. The chunks, the extraction output and the invocation logs all outlive it.
• Setting compliance-mode retention without testing in governance mode. There is no support ticket that fixes a seven-year mistake.
• Enabling model invocation logging without treating the log as privileged. You have just made a second copy of the documents in a less protected place.
• Building VPC endpoints and leaving the default policy on them. A private path with no policy is still a path.

Best practices worth the effort

• One retrieval helper, server-side filter injection, and a review rule that no other code path calls the retrieve API directly.
• Metadata schema agreed and frozen before the first production ingestion. Include the fields you might filter on later, even if unused today.
• Page-level provenance on every chunk, surfaced in the interface. It builds trust faster than any accuracy improvement.
• A confidence threshold chosen per document class, with a human review path for everything under it.
• Deletion implemented as a verified workflow across every store, ending in an assertion that the content is actually unretrievable.
• Object Lock legal holds for matters under litigation hold, tested in governance mode before anything runs in compliance mode.
• Customer-managed KMS keys across S3, the vector store and the logs, so key access is a control you can actually revoke.
• An alarm on unfiltered retrievals, because the failure mode is a success response.

Frequently asked questions

Is a legal document intelligence platform on AWS safe for privileged material?

The services support it. Bedrock’s stated position is that inputs and outputs are not shared with model providers or used to train base models, PrivateLink keeps traffic off the public internet, and KMS covers encryption at rest. What determines safety is your own configuration: tenant isolation, log handling and deletion behaviour. The platform is exactly as confidential as its weakest copy of the text.

Should I use Bedrock Knowledge Bases or build retrieval myself?

Start with Knowledge Bases. It handles chunking, embedding, sync and metadata filtering, and those are weeks of work with no differentiation in them. Build your own when you need retrieval behaviour it does not expose, such as unusual re-ranking or per-tenant chunking strategies. Going custom on day one usually means reimplementing the managed service badly.

How do I stop one client’s documents surfacing in another client’s answers?

Tag every chunk with a tenant identifier at ingestion, and inject the matching filter server-side on every retrieval from the authenticated caller’s identity. Never accept the identifier from the client. If the contract requires per-client encryption keys, use a separate collection per client instead, accepting the extra operational load.

Does deleting a document from S3 remove it from the search index?

No. The chunks stay in the vector store until an ingestion job runs and reconciles the data source. Until then the document is deleted and still fully searchable, which is the worst combination available. Always finish a deletion by querying the index and confirming nothing comes back.

What is the difference between an S3 legal hold and a retention period?

A retention period runs until a fixed date and comes in governance mode, which privileged users can override, or compliance mode, which nobody can. A legal hold has no end date and stays until someone explicitly removes it. They are independent, they can both be active on the same object version, and while either is active the object cannot be deleted or overwritten.

Do I need human review, or is the model accurate enough?

Accuracy is the wrong frame. The question is what happens to a wrong answer downstream. Where output feeds a decision with consequences, you want a confidence threshold and a review queue, and Amazon Augmented AI with a private workforce gives you that without building the review tooling yourself. Where output is a search aid a person will verify anyway, review is friction you do not need.

Can I run this in a single AWS account?

Technically yes, and for a pilot it is reasonable. It gets uncomfortable once you have production documents and a development environment in the same place, because an IAM mistake has nowhere to stop. Separate accounts under Organizations also gives you the policy layer, including the AI services opt-out policy, which only exists at the organisation level.

The one thing to take away

A legal document intelligence platform on AWS is not hard to build. Textract reads, Bedrock reasons, OpenSearch retrieves, and the tutorial version works on the first afternoon.

What is hard is that its failures are silent and well-formed. The cross-matter citation, the deleted document that still answers questions, the privileged passage sitting in a log bucket, the retrieval that ran without a filter and returned a beautiful paragraph. None of these throw an error. All of them are the kind of thing that ends a client relationship.

So build the boundaries first and the features second, and make sure every one of them is something you can prove from a stored record rather than something you believe about the code. If you can only take one habit from this: alarm on the successful requests that should not have been possible.

Need a second pair of eyes on your document pipeline?

I work on AWS document processing and retrieval systems where the confidentiality requirements are real. Typical things I get called in for:

• Reviewing tenant isolation in an existing RAG setup and finding the paths where the filter can be bypassed.
• Designing the metadata and chunking schema before ingestion, so filtering works without a re-index later.
• Building Textract pipelines with confidence thresholds, human review routing and page-level provenance.
• Implementing verified deletion across S3, derived artefacts, the vector index and logs, with a proof step at the end.
• Setting up retention and legal hold with S3 Object Lock, including the governance-mode rehearsal before compliance mode.
• Locking down the network and logging path: VPC endpoints with real policies, KMS key separation, and audit records that name the actual user.

If you have an architecture diagram, a Step Functions definition or a retrieval request you are unsure about, send it over and I will tell you what I would change and why.

The post Legal Document Intelligence on AWS: The Five Boundaries That Have to Hold appeared first on John Nessime.

So you check the zone. There is an MX record. There is an SPF record. There is a DKIM key published under a selector. There is a DMARC record. You paste the domain into three free checkers and every one of them lights up green.

Everything is configured. The domain is still trivially spoofable.

This is the failure mode that makes business email DNS worth writing about. It is not that people forget to publish records. It is that publishing a record and enforcing a policy are different acts, the checkers reward the first one, and nothing anywhere in DNS will tell you that you stopped halfway.

What follows is what each of the four records actually does, what breaks when it is wrong, the domain architecture decision that stops the problem recurring, and an order of operations that gets you to enforcement without losing legitimate mail on the way.

The two jobs your business email DNS actually does

Almost every confused email ticket traces back to one misunderstanding: people treat these four records as a single feature called “email setup.” They are two unrelated systems that happen to share a zone file.

• Receiving mail. This is MX, and only MX. Nothing else in the set participates.
• Authorizing senders. This is SPF, DKIM and DMARC. None of them affect whether a single message reaches your mailbox.

The consequence is blunt. If inbound mail is not arriving, editing SPF cannot help and never will. If your domain is being spoofed, a perfect MX configuration contributes nothing. It is common to see an afternoon spent rewriting an SPF record for a problem that lived entirely in a mail routing rule.

A third job hides inside the second, and it is the one people skip. SPF and DKIM only produce evidence: this server was authorized, this content was signed by this domain. Neither tells a receiving server what to do when the evidence is missing. Only DMARC does, through its policy tag, and that tag is the single enforcement dial in the whole arrangement.

MX: the only record inbound delivery depends on

An MX record carries exactly two values: a preference number and a hostname. Lower preference numbers are tried first. Equal preference numbers let a sender spread load across the listed hosts, which is how most hosted mail providers publish theirs.

dig +short MX example.com

The hostname side has two constraints panels will happily let you violate. The target must be a name resolving to A or AAAA records, not an IP typed into the field, and it must not point at a CNAME. Some panels accept an IP and quietly invent a hostname behind the scenes, which works until you move providers and find the record you thought you published is not the one that exists.

If a domain has no MX record at all, the SMTP specification tells senders to fall back to its A or AAAA address. On a business domain that is your web server. The connection times out or is refused, and the sender queues and retries for days before anyone sees a bounce. The mail is not rejected; it is late, sometimes by a working week.

And subdomains do not inherit MX records from the parent. If example.com has MX records, mail.example.com does not. It falls back to its own A record, or to nothing.

Null MX for domains that should never receive mail

Most businesses own more domains than they use for mail: typo-squat defenses, old brand names, a marketing microsite, the domain from the company they acquired. RFC 7505 defines a way to say “this domain accepts no mail” explicitly, using a single MX record with preference 0 and a target of a lone dot.

example-parked.com.  3600  IN  MX  0  .

A conforming sender sees this and rejects immediately instead of falling back to the A record and retrying for days. The bigger benefit is backscatter: when someone forges a return address at a domain of yours, bounce messages get aimed at it. A null MX turns those into an instant rejection at the sender’s end.

One rule attached to this: a domain advertising a null MX must not publish any other MX records. It is the whole set or nothing.

One operational note: drop MX TTLs to a few minutes before a mail migration and raise them afterwards. A low TTL costs a little query volume. A high TTL during a cutover costs you mail delivered to a server you already decommissioned, with no error anywhere to tell you. Per-record TTL control and fast propagation matter here, which is why I would rather do a cutover on something like Cloudflare than a registrar’s default panel.

SPF: an authorization list with a hard budget

SPF is a TXT record at the domain apex that begins with v=spf1 and lists the systems allowed to send with your domain in the envelope sender. Publish two of them and you have published none: the result is a permanent error, which receivers treat as a failure.

dig +short TXT example.com

The constraint everyone eventually hits is the lookup budget. Evaluation is capped at ten DNS-querying mechanisms. The terms that count are include, a, mx, ptr, exists and redirect, and the count follows nested includes all the way down, so one vendor include: can carry four lookups inside it. Literal ip4 and ip6 ranges cost nothing, and neither does all.

Exceed ten and evaluation stops with a permanent error. It does not fail open. It does not keep counting and pass anyway.

There is a second, quieter limit: void lookups, meaning queries that return nothing. Implementations are told to cap those at two. This is the one that bites long after you stop paying attention, because the trigger is a vendor you cancelled whose include: target eventually stopped resolving. Your record did not change. Your mail started failing anyway.

Over budget, the honest options are to drop vendors you no longer use, replace lookup-heavy mechanisms with the literal IP ranges they resolve to, or move senders onto their own subdomains so each gets a fresh budget. Flattening services that resolve includes and keep the record updated do work, but understand the trade: you have taken on a dependency that must track your vendors’ IP changes, and cloud senders rotate ranges.

Why hard fail is not automatically the safer choice

SPF records end with a qualifier on all. Soft fail is ~all; hard fail is -all. The instinct is that hard fail is the stricter, more secure choice, and it is, but the DMARC specification calls out a specific cost that is easy to miss.

SPF is evaluated early in the SMTP conversation. A hard fail can get a message rejected before the message body is ever transmitted, which means before the From header is visible and before DMARC is evaluated at all. A message that would have passed DMARC on an aligned DKIM signature can be rejected purely on SPF.

Messages rejected that early never reach the DATA phase, so they never appear in your DMARC aggregate reports. You lose the failure and the evidence of the failure in the same move.

That is the argument for staying on ~all while you are still discovering who sends as you, and moving to -all once your reports have gone quiet. On a parked domain with no legitimate senders at all, publish v=spf1 -all immediately; there is nothing to discover.

One more thing SPF does not do, and this is the root of most DMARC confusion. SPF authenticates the envelope sender, the address in the SMTP MAIL FROM command. Your recipients never see that address. They see the From header, which SPF does not check at all.

DKIM: the signature that survives a forward

DKIM adds a cryptographic signature over selected headers and the message body. The public key lives in DNS under a selector, in the form selector._domainkey.example.com, and the signature header names both the selector and a signing domain in its d= tag.

dig +short TXT s1._domainkey.example.com

That d= value is the part that matters and the part that gets ignored. A message can carry a perfectly valid signature from your provider’s own domain. It verifies. It proves nothing about you, because the identity it asserts is theirs. Completing your provider’s domain authentication step, the one that has you publish records under your own zone, is what moves d= onto your domain.

Algorithms and key size

The rules here are settled. Signers must use rsa-sha256. The older rsa-sha1 must not be used for signing or verifying, and signatures using it fail permanently. Keys must be at least 1024 bits and should be at least 2048. Verifiers are required to reject anything under 1024 outright.

A useful field heuristic follows from a DNS quirk. A single TXT character string maxes out around 255 characters, and a 2048-bit RSA key does not fit in one. So if your DKIM record comes back from dig as one quoted string, it is probably 1024-bit; split into two concatenated strings, probably 2048. A hint rather than proof, but it takes two seconds.

An Ed25519 signing algorithm is also defined for DKIM, with compact keys and good cryptographic properties, but support across platforms and verifiers is still uneven. RSA with SHA-256 remains the safe universal choice unless you sign with both.

Rotating a key without breaking anything

Never edit a DKIM key in place. Mail already in flight, in a queue, or sitting in a mailing list’s outbound spool was signed with the old key, and overwriting the record invalidates all of it retroactively.

1. Publish the new key under a new selector name and confirm it resolves.
2. Switch the signing platform to the new selector.
3. Wait out the longest plausible delivery delay, which is days rather than hours if mailing lists are involved.
4. Remove the old selector record.

DKIM has one property SPF does not, and it decides your architecture later. A signature travels with the message. When mail passes through a forwarder, a role alias or a mailing list, the connecting IP is no longer yours and SPF fails; the DKIM signature usually still verifies. This is why the DMARC specification states that domains publishing a reject policy must not rely on SPF alone and must apply valid DKIM signatures.

DMARC: connecting the visible domain to the proof

DMARC is a TXT record at _dmarc.example.com. It asks one question that neither SPF nor DKIM asks: does the domain your recipient can actually see match the domain that proved itself?

dig +short TXT _dmarc.example.com

That match is called alignment. Relaxed alignment, the default, accepts any domain sharing the same organizational domain, so mail.example.com aligns with example.com. Strict requires them to be identical. A message passes DMARC if either SPF or DKIM passes and the domain that passed aligns with the From header. One aligned mechanism is enough. Both passing while neither aligns is a fail, and that is the most common cause of “but all my checkers are green.”

The tags worth knowing

• p is the policy for the domain itself: none, quarantine or reject. Omit it and the record is treated as p=none.
• sp sets a separate policy for existing subdomains.
• np sets a policy for subdomains that do not exist in DNS at all. This is the anti-spoofing tag most zones are missing.
• rua is where aggregate reports are sent. Without it you get no reports, and without reports you are guessing.
• adkim and aspf switch the two alignment modes from relaxed to strict.
• t=y is test mode. It tells receivers to apply one level below your stated policy, so a record saying reject behaves as quarantine, and quarantine behaves as none.

The DMARC specification has been rewritten and promoted onto the IETF standards track as RFC 9989, with aggregate and failure reporting split into RFC 9990 and RFC 9991, obsoleting the original RFC 7489. Existing records keep working and still begin with v=DMARC1, so nothing is urgent. Three things are worth acting on at your next zone edit.

The pct, rf and ri tags are gone. Unknown tags are ignored rather than treated as errors, so leaving them does no damage, but pct no longer means anything and the staged rollout it was used for is now t=y.

The Public Suffix List is no longer how receivers determine your organizational domain. They now walk up the DNS hierarchy looking for a DMARC record at each level, capped at eight queries. For a two-label domain this changes nothing. For a deep subdomain hierarchy it can resolve differently than the old method did, and a name deeper than eight labels will never have its policy record found by the walk. If you run a complex namespace, publish an explicit record at the exact name rather than relying on inheritance.

And if two DMARC TXT records are returned for the same name, both are discarded. A duplicate is not a merge or a tiebreak. It is the same as having none.

Reading the reports

Aggregate reports arrive as gzipped XML, typically once a day per reporting provider. They are technically human-readable and practically not, especially once a dozen providers are reporting on a domain with several sending services.

Two honest choices. Parse them yourself, which is a small satisfying project that gives you exactly the view you want. Or point rua at a hosted processor; Dmarcian, Valimail and PowerDMARC all do this competently, with entry tiers sufficient for one domain. The trade is that you are sending metadata about your mail flows to a third party and inheriting their retention terms. Read those terms first in a regulated environment.

Domain architecture: the decision that stops this recurring

Everything above is records. This part is structure, and it is what separates a zone that stays correct from one that quietly rots.

Put each sending stream on its own subdomain. People mail from the apex. Marketing sends from something like news.example.com. Transactional and application mail sends from mail.example.com or similar. Each subdomain gets its own SPF record, its own DKIM selectors, and its own DMARC record if you want a different policy there.

• Each subdomain gets a fresh ten-lookup SPF budget, which is usually what forced the decision in the first place.
• A marketing platform that gets breached or blocklisted does not drag your invoices and password resets down with it.
• Reputation is scoped. Bulk sending and one-to-one business mail get judged separately, which is what you want, because they behave nothing alike.
• You can hold the apex at a strict policy while a subdomain is still being brought into line.

The cost is more records to maintain and one more thing to explain to whoever inherits the zone. It is worth it the first time a vendor migration happens and you only have to touch one subdomain.

Then close the gaps. Every domain that never sends and never receives should carry a null MX, an SPF record of v=spf1 -all, and DMARC at p=reject. Three records, five minutes, and the domain leaves the pool of things abusable in your name. Add np=reject on live domains too, because an attacker does not need a subdomain to exist in order to put it in a From header.

Rolling it out without losing mail

The order matters, because each step depends on evidence produced by the one before it.

1. Inventory every sender. Mailbox provider, CRM, helpdesk, billing system, marketing platform, monitoring alerts, and the tool a department signed up for without telling anyone. This list is always longer than expected and it is the highest-value step here.
2. Fix MX first, with a low TTL if you are migrating. Inbound is what people notice within minutes.
3. Publish SPF within budget, ending in ~all for now. Count lookups before publishing, not after mail fails.
4. Get DKIM signing on your own domain for every stream. Verify d= in the headers of a real received message rather than trusting a dashboard.
5. Publish DMARC at p=none with a rua address. This changes nothing about delivery and starts the flow of evidence.
6. Read reports for several weeks. Longer if you have quarterly patterns like invoicing runs or annual renewals, which is exactly the mail you cannot afford to break.
7. Move to p=quarantine once every remaining failure is one you recognize as spoofing. Use t=y for a softer first step.
8. Move to p=reject, confirming DKIM is aligned on every stream first, then tighten SPF to -all.

Step six is the one people compress, and it is the only step that generates the information the last two depend on. Skipping it means going to enforcement blind and finding out which stream you broke from a colleague rather than from a report.

Steps one and six are also the two that never finish. Keep the sender inventory current as vendors come and go, and keep reading reports after you reach enforcement, because a new tool bought by a department you never talk to shows up in a report long before it shows up in a conversation.

Troubleshooting, in the order worth checking

When something is wrong, the first question is which of the two jobs failed. Inbound problems and spoofing problems share no diagnostic steps.

Inbound mail is not arriving

1. Confirm the MX records exist and resolve, and that each target has an A or AAAA record.
2. Query your authoritative nameservers directly rather than a resolver, so you are looking at what you published rather than what is cached.
3. Confirm the mail host actually answers on port 25 and offers STARTTLS.
4. If all three pass, the problem is inside the mail platform: routing rules, a filter, an aliasing mistake, a mailbox over quota. It is no longer DNS.

dig +short NS example.com
dig @ns1.example-dns.net example.com MX
openssl s_client -starttls smtp -connect mail.example.com:25

Legitimate mail is being rejected or quarantined

1. Read the SMTP rejection text. Receivers name the reason, and DMARC-related rejections usually say so in plain words.
2. Open the headers of a message that failed and read the authentication results. They tell you which check failed and which domain was evaluated.
3. Compare the d= value in the DKIM signature against the From domain. If they differ and share no organizational domain, alignment is your problem.
4. Count your SPF lookups. A permanent error there presents as a plain SPF failure with nothing in the record that looks wrong.
5. Check the sending IP has a valid reverse DNS record that resolves forward again. Only whoever controls the IP can set this, which means your provider if you are on shared infrastructure, or you if you run your own server.

dig -x 203.0.113.10 +short

That reverse DNS point matters more than it looks. Major mailbox providers expect sending IPs to have valid forward and reverse DNS, so if you run your own mail server, PTR control is a hard requirement rather than a nice-to-have. Confirm it before you buy. Hosts aimed at self-hosting, InterServer and Contabo among them, expose PTR in the panel; plenty of budget instances do not, and some block port 25 outright.

Common mistakes

• Treating p=none as done. It is a monitoring mode. It provides no protection whatsoever, and every checker will still show you a green tick.
• Publishing two SPF records. Usually one from the mailbox provider’s setup wizard and one from a marketing tool. The result is worse than either alone.
• Assuming a passing DKIM signature is your DKIM signature. Check d=, not the presence of a signature.
• Editing a DKIM key in place instead of rotating through a second selector.
• Leaving parked and brand domains bare. No MX, no SPF, no DMARC is an open invitation, and these domains are usually the ones nobody is monitoring.
• Going to p=reject without DKIM aligned on every stream. Forwarded mail will break, and forwarded mail is disproportionately mail from customers.
• Setting rua and never reading it. Reports you do not process are the same as reports you never requested.

Frequently asked questions

Do I need both SPF and DKIM, or is one enough?

DMARC only needs one of them to pass and align. Publish both anyway. They fail in different circumstances, and having only one means any situation that breaks it breaks all your authentication at once. Major mailbox providers also expect both from higher-volume senders, and DKIM specifically is required if you intend to publish a reject policy.

How long should I stay at p=none?

Long enough to see a full cycle of your sending patterns, plus time to fix what the reports reveal. For a small business with a mailbox provider and two SaaS tools, a few weeks. For an organization with quarterly billing runs and departments who buy their own software, months. The signal to move is that you recognize every remaining failure as illegitimate, not that a calendar reminder fired.

Will DMARC stop phishing?

It stops exact-domain spoofing, one technique among several. It does nothing about lookalike domains, nothing about display-name spoofing where the visible name says your CEO but the address is free webmail, and nothing about a compromised mailbox sending properly authenticated mail. One control worth having, not a phishing solution.

Does any of this affect whether I receive email?

No. SPF, DKIM and DMARC describe mail leaving your domain. Only MX affects mail arriving at it. The one indirect connection is that your own inbound filtering may evaluate other people’s SPF and DKIM when deciding what to do with their mail, but that is your mail platform’s behavior, not your DNS records.

Do I need to update my records for the new DMARC RFCs?

Nothing breaks if you do nothing. At your next edit, drop pct, rf and ri if they are present, consider adding np to close the non-existent subdomain gap, and if you run a deep subdomain hierarchy, publish explicit records rather than relying on inheritance. Leave the rest at defaults, because the defaults are sensible.

My mail goes to spam even though everything passes. What now?

Authentication and reputation are different problems. Passing DMARC tells a receiver the mail genuinely came from you; it says nothing about whether recipients want it. If authentication is clean, look at complaint rates, list hygiene, volume ramps and content. Google publishes a spam complaint ceiling and its guidance is to operate well below that ceiling, not near it.

Wrapping up

If you take one thing from this, take the distinction between publishing and enforcing. Business email DNS gives you four records that answer four different questions, and only one of them, the DMARC policy tag, actually instructs a receiving server to do anything. Every free checker will confirm the records exist. None of them will tell you the dial is at zero.

Get MX right so mail arrives. Keep SPF inside its lookup budget. Make DKIM sign with your own domain so d= aligns. Publish DMARC at p=none with a reporting address, read what comes back for as long as it takes, then move the dial until it reaches reject.

And while you are in the zone, spend five minutes on the domains nobody thinks about. A null MX, an empty SPF record and a reject policy on a parked domain is the cheapest security work available to you.

Need a second pair of eyes on your mail domain?

Mail DNS is unusually well suited to outside help, because the failure is almost always somewhere the domain owner cannot see it: in a header, in a report nobody parses, or in a vendor’s include chain. Things I regularly help with:

• Auditing an existing zone across MX, SPF, DKIM and DMARC and telling you plainly what is exposed and what is fine
• Getting an SPF record back under the ten-lookup limit without breaking a sender you forgot about
• Planning and executing the move from p=none to p=quarantine to p=reject, including the sender inventory and the report reading
• Setting up DMARC aggregate report collection and parsing, either self-hosted or on a processor
• Splitting sending streams onto subdomains and migrating each one without a delivery gap
• Locking down parked and brand-protection domains so they stop being usable in your name

If you would rather not describe the problem in the abstract, send me the output of a dig against your domain, or a set of headers from a message that failed, and I will tell you what I see in it.

The post Business Email DNS Done Right: MX, SPF, DKIM and DMARC appeared first on John Nessime.

So you close the ticket. Then it happens again next week, to a different customer, on a different mailbox provider.

This is the shape of almost every WordPress email deliverability problem I’ve had to unpick. The site is not broken and the mail is genuinely leaving. It is being evaluated somewhere downstream and quietly filed into spam by a receiver that has decided it cannot prove who sent it. Nothing in WordPress will ever tell you this happened.

This post covers the four places that failure actually lives: how the message leaves your server, whether SPF survives evaluation, whether DKIM signs and aligns, and what DMARC does with the result. It is organised by failure family rather than by protocol tour, because the protocols are individually well documented and the failures are not.

The failure mode that costs you money: silent success

Start with the thing that makes this hard to debug. wp_mail() returns a boolean. That boolean tells you whether PHPMailer handed the message off without throwing. It tells you nothing about whether the message was delivered, and nothing at all about whether it reached an inbox rather than a spam folder.

So “WordPress says it sent” and “the customer got it” are separated by four or five systems you have no visibility into. That gap is where the money leaks: WooCommerce order confirmations, membership welcome mails, password resets, form notifications. All of them fail the same silent way.

So stop guessing about the boolean. WordPress fires an action when the handoff itself fails, and almost nobody hooks it:

// Log the real reason PHPMailer refused, instead of a bare false.
add_action( 'wp_mail_failed', function ( $error ) {
    error_log( 'wp_mail failed: ' . $error->get_error_message() );
} );

Then force a send from the command line so you are testing the mail path and not a contact form plugin:

wp eval 'var_dump( wp_mail( "you@example.com", "deliverability test", "body" ) );'

If that prints bool(true) and the mail still never arrives, you have ruled out WordPress entirely. Everything from here is transport and DNS.

Failure family one: how the message leaves the box

By default wp_mail() falls through to PHP’s mail(), which shells out to the local mail transfer agent. On a shared host or a default VPS build, that means your marketing site is now an email server with none of the reputation, warm-up, or bounce handling that implies.

Receivers are not being unreasonable about this. A message arriving from a web server IP, with no authentication, from a domain the IP has no relationship with, looks exactly like a compromised WordPress install pumping out spam. Because most of the time, it is.

The three transport options, honestly

Local mail(). Genuine case for it: zero dependencies, no credentials to leak, and it works fine when the box already runs a properly configured MTA with correct reverse DNS and a warmed IP. Where it fails: almost everywhere else. Most hosts either block outbound port 25 or run an MTA nobody has configured for authentication. Skip it unless you own the mail infrastructure and know its reputation.

Authenticated SMTP submission. WordPress connects to a real mail provider over port 587 with STARTTLS, or 465 with implicit TLS, and authenticates. This is the portable option, it works with any provider, and it is what I reach for first on a site I did not build. The cost is that SMTP is a chatty, connection-oriented protocol running inside a page request, so a slow or throttled provider makes checkout feel slow.

Provider HTTP API. The plugin posts the message to an API endpoint over 443 instead of speaking SMTP. Faster, immune to hosts that block SMTP ports outbound, and usually gives you richer delivery events back. The cost is real lock-in: your sending config is now provider-specific and moving means reconfiguring, not just changing a hostname.

On the plugin side, WP Mail SMTP and Post SMTP both cover the common providers and both have usable free tiers. For providers, transactional-focused services like Postmark, Amazon SES, Mailgun, Brevo and SendGrid all do this job well; the differences that matter are bounce webhook quality, log retention, and how quickly a human answers when your account gets throttled. If your host bundles a transactional relay, check whether it authenticates as your domain or the host’s shared domain before you rely on it. Providers like InterServer, Cloudways and Kinsta differ meaningfully here.

Prove the transport before you touch DNS

Before blaming records, confirm the server can actually reach the relay. This opens a TLS session against the submission port and shows you the SMTP banner and advertised capabilities:

openssl s_client -starttls smtp -crlf -connect smtp.example.net:587

-starttls smtp issues the STARTTLS upgrade for you, and -crlf sends proper line endings so the session does not hang. If this times out, your host is filtering outbound SMTP and you want the HTTP API transport instead of a longer argument with support.

To send a full test message end to end, swaks is the tool worth having on every box:

swaks --to you@example.com --from wp@example.com 
      --server smtp.example.net:587 --auth --tls

It prints the entire conversation, so an authentication failure or a policy rejection shows up as an actual SMTP response code rather than a shrug from a plugin.

Failure family two: SPF passes until it silently doesn’t

SPF is a DNS TXT record listing who may send using your domain in the envelope sender. It looks like this:

v=spf1 include:_spf.google.com include:sendgrid.net ~all

Two things about that record cause more WordPress email deliverability incidents than anything else in SPF.

The ten lookup budget

RFC 7208 caps SPF evaluation at ten mechanisms that require DNS lookups. include, a, mx, ptr, exists and redirect all count, and crucially so does everything nested inside someone else’s include. ip4 and ip6 cost nothing.

Cross the limit and the receiver returns permerror. That is not a soft fail. The receiver could not evaluate your record at all, treats SPF as failed, and every message from the domain is affected at once.

Here is why it bites so hard on WordPress sites specifically. Nobody adds ten includes in one sitting. Someone adds Google Workspace. Six months later the marketing team adds a newsletter platform. Then a CRM. Then a help desk. Then you add a transactional provider for the WooCommerce mail, and the record tips over. Nothing in DNS warns you, no plugin surfaces it, and the only visible signal is in DMARC aggregate reports you probably aren’t reading yet.

Check the record and count what it actually resolves to:

dig +short TXT example.com

Counting nested includes by hand is tedious and error-prone, so use a checker that expands them. When you are over budget, the durable fixes are removing senders you no longer use and moving distinct sending systems onto their own subdomains, each with their own SPF record. Flattening includes into raw IPs works, but you have just taken ownership of someone else’s IP ranges and they will change without telling you.

One record, and mind the envelope

Publish exactly one SPF record per domain. Two records is not additive, it is invalid, and receivers may fail the check outright.

Also remember SPF checks the envelope sender, not the address your visitor sees. On WordPress those can easily differ, especially when a plugin sets a friendly From address while the MTA uses the system user as the envelope. If you need to control it explicitly:

add_action( 'phpmailer_init', function ( $phpmailer ) {
    $phpmailer->Sender = 'bounces@example.com';
} );

Setting Sender sets the envelope from, which is what SPF evaluates and where bounces are returned. It is not the same field as From, and conflating the two is a classic source of “but SPF passes when I test it” confusion.

Failure family three: DKIM signs, but alignment is what counts

This is the invisible one, and if you take one thing from this post, take this.

DKIM adds a cryptographic signature to the message. The receiver fetches your public key from DNS at the selector your provider gave you and verifies the signature held. Check yours like this:

dig +short TXT selector1._domainkey.example.com

Straightforward enough. Here is the part that catches people: SPF and DKIM can both pass and DMARC can still fail.

DMARC does not care that a domain authenticated. It cares that the domain the recipient sees in the From header matches the domain that authenticated. That match is called alignment, and it is the whole point of DMARC. SPF aligns when the envelope sender domain matches the From domain. DKIM aligns when the signing domain in the d= tag matches the From domain. You need at least one of them aligned. Not both, but at least one.

Now picture the default setup on a WordPress site with a transactional provider. The From header says orders@example.com. The envelope sender is a bounce address on the provider’s own domain, so SPF authenticates the provider and does not align with you. And if you skipped the DNS records the provider asked you to add, the message is signed with the provider’s shared key, so d= is the provider’s domain and DKIM does not align either.

Both checks pass. Alignment fails. DMARC fails. Every deliverability tester you paste the record into says your DNS is fine.

The fix is nearly always the same: complete your provider’s domain authentication step so mail is signed as your own domain rather than their shared one. That usually means publishing a CNAME or TXT record at a selector under _domainkey, and often a CNAME for a custom return path so SPF aligns too.

Read the headers, don’t trust the dashboard

Send yourself a real message from the site and open the raw source. The Authentication-Results header the receiver added is the ground truth. You are looking for the SPF result, the DKIM result, the d= domain on the DKIM signature, and the DMARC verdict. If d= is not your domain and the SPF domain is not your domain, you have found your problem regardless of what any dashboard claims.

One practical note on the key itself: a 2048-bit DKIM public key does not fit in a single 255 character DNS string and has to be published as multiple concatenated strings. Most DNS providers handle this transparently. Some do not, and the symptom is a key that looks present but fails verification.

Failure family four: DMARC policy, and what just changed

DMARC ties the previous two together. It tells receivers what to do when neither SPF nor DKIM aligns, and it asks them to send you reports about it.

_dmarc.example.com. IN TXT "v=DMARC1; p=none; rua=mailto:dmarc@example.com"

Start at p=none. It changes nothing about how mail is handled and turns on the aggregate reports, which are the only view you will ever get of who is sending as your domain. Read them for a few weeks, find the legitimate senders you forgot about, fix their alignment, then move to p=quarantine and eventually p=reject.

Do not skip straight to p=reject on a domain you have not observed. You will reject your own invoicing system.

The specification itself was updated. RFC 7489, the original DMARC document from 2015, has been obsoleted by RFC 9989 together with RFC 9990 for aggregate reporting and RFC 9991 for failure reporting, and DMARC has moved onto the IETF standards track. Records still begin with v=DMARC1 and nothing you have published stops working, so this is not an emergency. Two changes are worth knowing:

• The pct tag is gone, along with rf and ri. Percentage-based staged rollout was implemented inconsistently by receivers and has been replaced by a simple t testing flag. If your record still carries a partial pct value from a rollout that never finished, that is worth cleaning up.
• Organisational domain discovery no longer depends on the Public Suffix List; receivers now walk the DNS tree. Single-domain owners will not notice. If you run a lot of subdomains, publishing explicit records rather than relying on inheritance removes the ambiguity.

There are also new optional tags. The one worth considering is np, which sets a policy for non-existent subdomains so attackers cannot spoof from a subdomain you never created, without changing your main policy.

Aggregate reports arrive as compressed XML and are unpleasant to read by hand. Feeding them to a processor is worth it; dmarcian, Postmark’s free DMARC digest and MXToolbox all do this, and Cloudflare can host the DNS side.

Are the bulk sender rules your problem?

Probably not, and it is worth saying so plainly because a lot of writing on this topic implies otherwise. Google’s stricter bulk sender requirements apply to domains sending roughly 5,000 or more messages a day to personal Gmail accounts. A typical WordPress business site sends a few dozen.

What changed for everyone is the direction of travel. Authentication that used to be a nice-to-have is now the baseline for being evaluated on your merits rather than treated as suspicious by default. If you do cross into bulk territory, the additional obligations are one-click unsubscribe on marketing mail via the List-Unsubscribe and List-Unsubscribe-Post headers, and keeping spam complaints low. Google’s published guidance is to stay under 0.1% and never reach 0.3%.

Domain strategy: keep resets away from newsletters

One structural decision saves more grief than any single record. Separate your transactional mail from your marketing mail at the domain level.

Send password resets and order confirmations from something like mail.example.com, and send campaigns from news.example.com. Each subdomain gets its own SPF record, which keeps you comfortably inside the lookup budget. More importantly, reputation is largely tracked per sending domain. When a campaign goes badly and complaints spike, the damage stays contained. Your password resets keep landing.

The trade-off is real: more DNS records to maintain, more DMARC reports to read, and a From address that is slightly less clean. On a site where a lost order confirmation costs actual revenue, that is a trade worth making.

Troubleshooting: a working order

1. Does the message leave? Hook wp_mail_failed and send via WP-CLI. A logged error here means credentials, ports or a plugin, not DNS.
2. Can the box reach the relay? openssl s_client -starttls smtp against the submission port. A timeout means outbound filtering.
3. What does the receiver say? Send to a mailbox you control, open the raw source, read Authentication-Results. This single header settles most arguments.
4. Is the DKIM d= your domain? If not, you have not completed provider domain authentication. Fix that before anything else.
5. Does SPF actually evaluate? Expand nested includes and count. A permerror looks like a pass in most naive testers.
6. What do the aggregate reports show? If mail is failing for a subset of recipients only, the reports will name the sending source you forgot about.

If Gmail is deferring rather than rejecting, the enhanced status code in the bounce tells you which check failed. Codes in the 4.7.x range point at authentication or alignment specifically rather than content, which narrows the search considerably.

Common mistakes

• Publishing two SPF records and assuming they merge. They do not.
• Treating a green tick from an online record checker as proof of delivery. Those tools validate syntax, not alignment on your actual mail.
• Jumping to p=reject before reading a single aggregate report.
• Setting a From address on a domain you do not control, such as a customer’s own address on a contact form. Use your domain in From and put theirs in Reply-To.
• Adding a new SaaS include without re-counting the lookup budget.
• Leaving a stale include for a service you stopped using two years ago, which is both a wasted lookup and a live authorisation.
• Configuring an SMTP plugin and never testing what happens when the provider is down. Mail queued nowhere is mail lost.

Best practices

• Route WordPress mail through a real provider, and prefer the HTTP API transport when the host filters SMTP ports.
• Complete provider domain authentication properly so DKIM signs as your domain. This is the single highest-value action on the list.
• Publish DMARC at p=none with rua on day one, even before you intend to enforce. The reports are the visibility you are missing.
• Keep SPF under the ten lookup budget with headroom for the next service someone adds.
• Split transactional and marketing sending onto separate subdomains.
• Enable email logging in your SMTP plugin so you have a record of what was attempted and when.
• Monitor DKIM key expiry and provider selector changes the same way you monitor TLS certificates. Both fail silently and both fail completely.

Frequently asked questions

Do I need an SMTP plugin if my host already sends mail?

Check what the host’s relay authenticates as. If it signs with the host’s shared domain rather than yours, DKIM will not align with your From address and DMARC will fail even though mail is technically going out. If it authenticates as your domain, you are fine without a plugin.

Is SPF enough on its own?

No. SPF breaks on forwarding, because the forwarding server is not in your record, and it authenticates the envelope sender rather than the visible From address. DKIM survives forwarding and gives you a second, independent path to alignment. Publish both.

Why do my emails reach Gmail but not Outlook, or the reverse?

Receivers weight signals differently and each maintains its own reputation view of your sending IP and domain. Divergent behaviour usually means your authentication is borderline rather than solid, so one receiver’s threshold catches it and another’s does not. Fix alignment rather than chasing individual receivers.

How long until DNS changes take effect?

As long as the TTL on the record, plus whatever the resolver in front of you decides to cache. Lower the TTL before you plan changes, not after. Verify with dig against an authoritative nameserver rather than trusting a browser-based checker.

Does DMARC stop people spoofing my domain?

At p=reject it stops exact-domain spoofing at receivers that honour the policy, which is most large ones. It does nothing about lookalike domains, display-name spoofing, or a compromised mailbox sending genuinely authenticated mail. It is one control, not a complete answer to phishing.

Should I update my DMARC record for the new RFCs?

Nothing breaks if you do not. At your next DNS edit, remove pct, rf and ri if present, and consider adding np if you want to close the non-existent subdomain gap. Leave the new tags unset otherwise, because the defaults are sensible.

Why did deliverability get worse after I moved hosts?

Usually because the old host had a relay configured that the new one does not, so PHP mail() is now sending direct from an unwarmed IP with no authentication. It is also common for a migration to move the site but not the DNS records the old provider had added. Re-verify SPF, DKIM selectors and the return path after any move.

Wrapping up

If you remember one thing about WordPress email deliverability, remember that authentication and alignment are different questions. SPF passing and DKIM verifying does not mean DMARC passes. The receiver is asking whether the domain your recipient can see is the domain that proved itself, and on a default WordPress plus transactional provider setup, the honest answer is often no.

Get the transport off PHP mail(), complete your provider’s domain authentication so d= is your own domain, keep SPF inside its lookup budget, and publish DMARC at p=none so you can finally see what is happening. Then read the reports for a month before you enforce anything.

The alternative is closing the same ticket every week and never learning why.

Need someone to fix this properly?

Mail problems are unusually well suited to outside help, because the failure is almost always in a place the site owner cannot see. This is work I take on regularly:

• Auditing an existing setup end to end and telling you exactly which of the four failure families you are in
• Rebuilding SPF records that have drifted past the ten lookup limit, without breaking a live sender
• Completing DKIM and return-path domain authentication with your provider so alignment actually holds
• Taking a domain from p=none to enforcement safely, using the aggregate reports rather than guesswork
• Splitting transactional and marketing sending onto separate subdomains with their own records and reputation
• Migrating WordPress mail off PHP mail() onto a provider, with logging and failure alerting wired up

If you want a second opinion, send me a raw message source with the full headers, or your current SPF and DMARC records, and I will tell you what I see.

The post Why Your WordPress Emails Vanish: SMTP, SPF, DKIM and DMARC appeared first on John Nessime.

The reflex at that point is to reach for history rewriting. That is the wrong first move, and I will come back to why. But the bigger problem is the one that shows up six months after you clean up: teams migrate to Sealed Secrets or SOPS, tick the box, and quietly assume the whole class of problem is now solved. It is not. Encrypting a value before it enters Git closes exactly one door out of about five.

This post covers how to keep Kubernetes secrets out of Git in a way that survives contact with a real cluster: the three families of solution and where each one genuinely wins, the disaster-recovery trap that bites hardest, how to stop plaintext reaching the repo in the first place, and what to actually do about credentials already in your history.

The failure that actually bites: encrypted in Git, wide open in the cluster

Sealed Secrets and SOPS both terminate in the same place. The controller decrypts, the API server accepts, and what lands in etcd is an ordinary Kubernetes Secret. Ordinary means base64-encoded, which is an encoding, not a cipher. Anyone who can run kubectl get secret -o yaml in that namespace reads the value in one pipe through base64 -d.

So the honest description of what these tools buy you is narrower than the marketing suggests. They stop the credential appearing in a public object store with permanent history and a CDN in front of it. They do nothing about the cluster’s own blast radius.

Three things need to be true alongside whichever tool you pick:

• Encryption at rest is on. A stock Kubernetes control plane writes Secrets to etcd unencrypted unless you configure an EncryptionConfiguration, ideally with a KMS provider. Managed distributions vary. On EKS, envelope encryption is something you enable, not something you inherit. Check yours rather than assuming.
• RBAC on secrets is actually restrictive. A depressing number of clusters grant blanket get and list on secrets to service accounts that only ever needed one. That single verb turns a namespace compromise into a credential dump.
• Secrets are mounted as files, not shoved into environment variables where you can help it. Environment variables leak through /proc/<pid>/environ, through anything that dumps the environment on crash, and through every debug endpoint an application framework has ever shipped. A projected volume with tight permissions is the quieter option.

Quick sanity check on the RBAC side, which takes about ten seconds and surprises people:

# Can the default service account in a namespace read every secret in it?
kubectl auth can-i get secrets 
  --namespace production 
  --as system:serviceaccount:production:default

If that returns yes, your Git hygiene is not the weakest link in the chain.

Already leaked? Rotate first, rewrite second, and understand what rewriting does not do

Order matters here more than technique. GitHub’s own guidance is blunt about it: once a credential has been pushed, revoke or rotate it first, because that step alone removes the attacker’s ability to use it. History surgery is the optional follow-up, not the fix.

The reason is that a force-push does not reach everywhere the commit went. Objects survive in forks, which inherit the full object graph at fork time. They survive in colleagues’ clones. On GitHub they remain reachable by SHA through cached views and through pull requests that referenced them, and purging those caches is a support request, not a git command. If the repo was public even briefly, treat the credential as burned and move on.

When you do rewrite, use git filter-repo. It has replaced filter-branch as the recommended tool and it is dramatically faster on any repo with real history.

# Work on a bare mirror, never your day-to-day clone.
git clone --mirror git@github.com:org/infra.git infra-clean.git
cd infra-clean.git

# Strip one path from every commit that ever touched it.
git filter-repo --path clusters/production/values.yaml --invert-paths

# filter-repo deletes the origin remote on purpose, so you cannot
# force-push a half-finished rewrite by muscle memory. Add it back.
git remote add origin git@github.com:org/infra.git
git push --force --all
git push --force --tags

Then the unglamorous part: close or merge open pull requests before you start, because every SHA downstream of the rewrite changes. Tell every collaborator to delete their clone and clone again. A git pull will happily drag the old objects back in.

Two families of solution, and how to tell which one you are in

Every approach to this problem is either encrypt the value and commit the ciphertext, or commit a pointer and fetch the value at runtime. Everything else is implementation detail. The choice is mostly about whether you already run a secret store, and whether you can tolerate the CLI step in the middle of your workflow.

Sealed Secrets: the lowest-friction way to start

A controller in the cluster holds an RSA private key and publishes the matching certificate. The kubeseal CLI encrypts a Secret manifest against that certificate and produces a SealedSecret custom resource. Only that controller can decrypt it, so the resulting YAML is safe to commit.

# Fetch the controller's public certificate. This is safe to commit;
# it can only encrypt, never decrypt.
kubeseal --fetch-cert 
  --controller-name=sealed-secrets 
  --controller-namespace=kube-system 
  > pub-cert.pem

# Build a Secret manifest locally. --dry-run=client means it is rendered
# and never sent to the API server.
kubectl create secret generic db-auth 
  --from-literal=password='<value>' 
  --dry-run=client -o yaml > db-auth.yaml

# Encrypt against the cert. Offline, so no cluster access needed here.
kubeseal --cert pub-cert.pem --format yaml < db-auth.yaml > db-auth-sealed.yaml

rm db-auth.yaml

Because you can hand out pub-cert.pem, developers can seal secrets without any cluster credentials at all. That is the underrated part, and it is the reason this is the one I reach for first on a small team with no existing vault.

Two details that catch people. First, sealing is scoped by default: a SealedSecret is bound to its exact name and namespace, and moving the file to a different namespace makes it undecryptable. That is deliberate, it stops a developer sealing a secret into a namespace they should not have access to, and you can relax it with --scope namespace-wide or --scope cluster-wide if you genuinely need to. Second, the Helm chart names the controller sealed-secrets while the CLI looks for sealed-secrets-controller by default, so --controller-name is not optional in practice.

Where it stops being the right answer: multi-cluster. Each cluster generates its own key pair, so a sealed file is not portable, and a fleet means either copying private keys around or maintaining per-cluster ciphertext for the same value. Rotation is also manual. There is no dynamic secret story here at all.

SOPS with age: encrypt files, not just Kubernetes objects

SOPS encrypts the values inside a structured file and leaves the keys readable, so a diff still tells you which field changed even if it cannot tell you what it changed to. Paired with age, which uses short X25519 keys instead of the GPG keyring experience, it is pleasant to live with.

A .sops.yaml at the repo root drives everything through path matching, which is how you get per-environment key separation without anyone having to remember flags:

creation_rules:
  - path_regex: clusters/production/.*.yaml$
    encrypted_regex: '^(data|stringData)$'
    age: age1...   # production recipient
  - path_regex: clusters/staging/.*.yaml$
    encrypted_regex: '^(data|stringData)$'
    age: age1...   # staging recipient

The encrypted_regex line is the one worth understanding rather than copying. Without it SOPS encrypts everything including metadata, and you lose the readable-diff property that made this approach attractive. Restricting it to data and stringData keeps names, namespaces and labels reviewable in a pull request.

Flux decrypts SOPS natively: you point a Kustomization at a Secret holding the age private key via decryption.provider: sops and it handles the rest. Argo CD does not have equivalent built-in support, so you bolt it on with a config management plugin such as KSOPS or the Argo CD Vault Plugin. Worth confirming against current Argo CD docs before you commit to it, because plugin mechanics there have changed more than once.

SOPS also handles files that are not Kubernetes manifests: Terraform variable files, Ansible vars, plain .env. If you have config living outside the cluster, that reach is a real advantage over Sealed Secrets.

External Secrets Operator: stop putting the value in Git at all

The third approach sidesteps encryption entirely. You commit a reference, and a controller resolves it against a real secret store: HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, Google Secret Manager, or one of the newer hosted options like Doppler, Infisical or 1Password. The repo contains a path, never a value.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: db-auth
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: db-auth
  data:
    - secretKey: password
      remoteRef:
        key: production/postgres
        property: password

This is the only one of the three that gives you rotation without a commit. Change the value in Vault, wait for refreshInterval, and the cluster Secret updates. It also gives you a real audit trail, because reads happen against a system built to log them.

The cost is honest and worth stating. You have introduced a runtime dependency: if the store is unreachable, new workloads cannot get their credentials. You still have a bootstrap credential problem, because the operator needs to authenticate to the store somehow, and that root of trust has to get into the cluster by some other path. And Git is no longer the complete description of your system, which is a genuine philosophical cost if GitOps purity matters to you.

One live upgrade trap. The external-secrets.io/v1beta1 API was removed in ESO 0.17.0. Before that, in the 0.16 series, a conversion webhook served both versions and rewrote stored objects to v1, which produced permanent drift in Argo CD for anyone whose Git manifests still said v1beta1. Argo sees v1 in the cluster, Git says v1beta1, and it reconciles forever. Update your manifests to v1 before you cross that boundary, not after.

Secrets Store CSI Driver: when a Secret object is one object too many

Worth knowing about even if you do not use it. The CSI driver mounts values from an external store directly into the pod as a volume, so no Kubernetes Secret object needs to exist at all. That closes the etcd exposure completely. It costs you the ability to consume the value as an environment variable, and it means the secret is only available to pods that mount it, which is either exactly what you want or a nuisance depending on the workload. Use it where the compliance posture demands that etcd never sees the value.

The key you forgot to back up

This is the failure I would put money on, and it is the one nobody rehearses. Every encrypt-into-Git approach concentrates all your risk into one small piece of key material. Lose it and your repository turns into a directory of well-organised noise.

Concretely: rebuild a cluster from scratch, install a fresh Sealed Secrets controller, and it generates a new key pair. Every SealedSecret in your repo is now undecryptable. Your GitOps repo, the one that was supposed to let you recreate everything, cannot recreate anything that needed a credential. You find this out during the incident, not before it.

• Back up the sealing key out of band. The controller’s private key lives in a Secret in its namespace. Export it, encrypt it, and store it somewhere that is not the cluster and not the repo. A password manager or an offline copy is fine. Test the restore.
• Same for age. The private key is one line of text. Its size makes it feel unimportant. It is the entire thing.
• Add a second recipient for anything that matters. SOPS encrypts the data key once per recipient, so listing both an age key and a KMS key means either path can decrypt. The developer works with age locally, the cluster decrypts through KMS, and no single lost key is fatal.
• Know your re-key procedure before you need it. For SOPS that is sops updatekeys across every file matched by the rule. For Sealed Secrets it means re-sealing everything against the new certificate. Neither is hard. Both are miserable to work out under pressure.

Stopping plaintext from reaching the repo in the first place

All of the above assumes the encryption step happens. The commit that leaks is always the one where somebody skipped it, usually while debugging at the end of a long day.

1. Put a scanner in the pre-commit hook. gitleaks and trufflehog both work well here. Local hooks are bypassable, which is fine, because their job is catching accidents rather than stopping a determined person.
2. Run the same scan in CI, on the whole history, not just the diff. The diff-only scan misses everything that predates the day you added the scanner.
3. Turn on server-side push protection. GitHub’s push protection rejects known credential patterns at push time. GitLab and platforms like GitGuardian offer equivalents. This is the layer that actually holds, because it does not depend on anyone’s local setup.
4. Make the plaintext path short-lived by construction. Never write a plaintext manifest to a tracked path. Use /tmp, or pipe straight into the encrypt step so the value never lands on disk at all.
5. Add *.dec.yaml, secret.yaml and friends to .gitignore as a convention. Not a control, just one less way to be careless.
6. Remember your shell history. A --from-literal with a real password ends up in ~/.bash_history in cleartext. Use --from-file, or read from a variable, or prefix the command with a space if your shell is configured to skip those.

Troubleshooting the four things that go wrong

The SealedSecret applies cleanly but no Secret appears

Almost always a scope mismatch. The resource was sealed for one name or namespace and applied to another. The API server accepts the custom resource happily; the controller is where the failure surfaces.

# The controller logs are the only place this error lives.
kubectl logs -n kube-system deploy/sealed-secrets --tail=50

# Confirm what the CRD actually thinks its name and namespace are.
kubectl get sealedsecret db-auth -n production -o yaml

The second common cause: the file was sealed against a certificate from a different cluster, or from before a key rotation. Re-fetch the cert and re-seal.

The ExternalSecret sits there doing nothing

The status conditions carry the real message, and the events under describe usually name the provider error directly.

kubectl get externalsecret db-auth -n production
kubectl describe externalsecret db-auth -n production
kubectl describe clustersecretstore vault-backend

Check the store before the secret. If the SecretStore is not Ready, nothing referencing it will ever sync, and the error you want is on the store, not the ExternalSecret. When using a ClusterSecretStore, remember that any secret reference inside it needs an explicit namespace, because a cluster-scoped object has no namespace of its own to fall back on.

Argo CD or Flux reports permanent drift

Two usual causes. Either an API version conversion is rewriting stored objects to a version your Git manifests do not use, as with the ESO v1beta1 removal above, or a controller is adding fields to the generated Secret that your reconciler then tries to remove. Compare the live object against the rendered manifest field by field before you start changing anything.

SOPS decryption fails only in the cluster

You encrypted for one recipient and the cluster holds a different key. This is the classic outcome of listing multiple recipients in .sops.yaml but running sops --encrypt with an explicit --age flag that overrides the rule. Decrypt locally to prove your key works, then check which recipients are actually listed in the file’s SOPS metadata block. The recipient list is stored in the file, in the clear, which makes this a fast thing to diagnose.

Common mistakes

• Rewriting history before rotating the credential, and treating the rewrite as the remediation.
• Assuming a private repo is safe enough. Private repos get forked internally, cloned onto laptops, and made public by accident.
• Committing the encrypted file and the plaintext source next to it, because the encrypt step wrote to a new filename and nobody deleted the original.
• Never backing up the sealing key or the age identity.
• Leaving encryption at rest off, so the credential moves from a Git leak to an etcd snapshot leak.
• Encrypting the entire manifest with SOPS instead of just data and stringData, and losing every useful diff.
• Using one age key or one sealing controller for all environments, so a staging compromise reaches production ciphertext.
• Piping a Helm release’s rendered output into a debug artefact in CI. Helm renders secrets in the clear, and CI artefacts are frequently world-readable inside an organisation.

How I would actually decide

Skip the feature matrix and answer three questions in order.

1. Do you already run a secret store? If Vault or a cloud secret manager is already in your environment and someone owns it, use External Secrets Operator. You get rotation and audit for almost no additional conceptual load, and the argument is over.
2. Do you have config outside Kubernetes? Terraform variables, Ansible vars, application .env files. If yes, SOPS with age, because one tool covers all of it and the per-path rules give you environment separation for free.
3. Neither? Sealed Secrets. One cluster, a handful of secrets, a team that wants this solved this afternoon. It is the shortest distance to a repo with no plaintext in it, and migrating away later is straightforward because the plaintext values are recoverable from a running cluster.

Whichever you land on, the cluster-side work is the same and it is not optional: encryption at rest, tight RBAC on secrets, files rather than environment variables, and a tested backup of whatever key material the scheme depends on. On a managed control plane from a provider like DigitalOcean, Linode or a hyperscaler, the encryption-at-rest configuration is partly theirs and partly yours, so read the specific documentation rather than trusting a general answer. On a self-managed cluster running on your own VPS instances, for instance a k3s setup on InterServer or similar, all of it is yours and none of it is on by default.

FAQ

Is base64 in a Kubernetes Secret a form of encryption?

No. Base64 is an encoding that exists so binary values survive YAML parsing. It is reversed with a single command and provides no confidentiality whatsoever. Anyone reading the manifest reads the secret.

Should I use Sealed Secrets or External Secrets Operator?

If you already run Vault, AWS Secrets Manager or an equivalent, use External Secrets Operator: you get rotation without commits and a proper audit trail. If you have no secret store and no appetite to run one, Sealed Secrets is a faster path with fewer moving parts. The trade-off is that Sealed Secrets has no rotation story and its keys are per-cluster.

What happens if I lose the Sealed Secrets private key?

Every SealedSecret in your repository becomes permanently undecryptable, and you regenerate every credential from source. This is the single most common way this setup fails in practice. Back the key up somewhere outside the cluster and the repo, and test restoring it.

Do I still need to rewrite Git history after rotating a leaked credential?

Often not. Once the credential is revoked it cannot be used, which usually resolves the actual risk. Rewriting is worth the disruption when the value is not rotatable, when it is personal data rather than a credential, or when a compliance process requires it. Weigh it against breaking every open pull request and forcing everyone to re-clone.

Can Argo CD decrypt SOPS files on its own?

Not natively, unlike Flux, which has SOPS decryption built into its Kustomize controller. With Argo CD you add a config management plugin such as KSOPS or the Argo CD Vault Plugin. Check the current Argo CD documentation for the supported plugin mechanism before building around it.

Is it safe to commit the Sealed Secrets certificate or the age public key?

Yes, and committing them is the point. Both are public halves of an asymmetric pair and can only encrypt. Putting them in the repo is what lets developers seal new secrets without cluster access. The private halves never go anywhere near Git.

The one thing worth remembering

If you take one idea away, make it this: the work to keep Kubernetes secrets out of Git and the work to keep them safe in the cluster are two separate jobs, and finishing the first one feels a lot like finishing both. It is not. A repo full of SealedSecret resources sitting in front of an unencrypted etcd with permissive RBAC has moved the exposure, not removed it.

Pick the approach that matches what you already run. Back up the key on day one, not after the first rebuild. Turn on encryption at rest. And when something does leak, rotate before you reach for filter-repo, because rotation is the part that actually stops the bleeding.

Need help sorting out secrets in your cluster?

This is work I do regularly, and most of it is less dramatic than it sounds once someone has done it before. Things I can help with:

• Auditing a GitOps repo for committed credentials, including full history, and producing a prioritised rotation list rather than a wall of scanner output
• Setting up Sealed Secrets, SOPS with age, or External Secrets Operator against Vault or a cloud secret manager, including the key backup and restore procedure
• Migrating an existing repo off plaintext or off a scheme that no longer fits, without a big-bang cutover
• Fixing Argo CD or Flux reconciliation loops caused by secret controllers and API version conversions
• Enabling and verifying etcd encryption at rest, and tightening RBAC on secrets across namespaces
• Adding pre-commit and CI secret scanning that people will not immediately disable because of false positives

If you have a controller log, a stuck ExternalSecret, or a scanner report you are not sure how to triage, send it over and I will tell you what I make of it.

The post Keep Kubernetes Secrets Out of Git Without Kidding Yourself appeared first on John Nessime.

What just happened is that your data warehouse now has an endpoint on the public internet, reachable on port 5439, and a third-party SaaS platform holds credentials to it. Usually those credentials belong to whatever user was to hand, which on a lot of clusters means the admin account that can read every schema you have.

That is not an argument against doing it. It is an argument for doing it on purpose. Most guides on how to connect Amazon Redshift to Zoho Analytics walk you through six screenshots and never mention the two decisions that actually matter: which user this connection runs as, and how much of your network it needs.

This covers the whole setup in order, with those decisions made deliberately: the mode you pick and why it is hard to change later, a read-only user scoped to almost nothing, network access done narrowly, and what to actually expose once it works.

What the connection actually requires

Strip away the interface and the mechanics are simple. Zoho Analytics runs on Zoho’s infrastructure. To reach your cluster it opens an inbound TCP connection from Zoho’s own IP addresses to your Redshift endpoint, on the Redshift port. That is it.

Which means three things have to be true, and they are the three places setup fails:

• The endpoint has to be reachable from outside your VPC. A cluster in a private subnet with no public endpoint cannot be reached by a SaaS tool, full stop.
• The security group has to allow it. Zoho publishes the IP addresses it connects from, and those need an inbound rule on the Redshift port. The default port is 5439, though it can be changed at cluster creation, so check yours rather than assuming.
• The credentials have to work and the user has to be able to see the schemas you want to report on.

Zoho keeps the current allowlist on a dedicated help page. Do not copy a list of addresses out of a blog post, including this one. They change, and a stale allowlist produces an intermittent failure that looks like everything except what it is.

Decide the mode before you start

Zoho offers two fundamentally different connection types, and switching later means rebuilding the workspace. Get this one right the first time.

Data Import copies your data into Zoho Analytics on a schedule and reports run against their storage. Dashboards are fast, you can create query tables, and you can import a filtered subset using a custom query. Column additions and deletions sync automatically. The trade is freshness: your numbers are as current as the last sync.

Live Connect stores nothing and queries Redshift each time a report loads. Numbers are always current and your data does not leave AWS. It is available on the paid tiers only. The trade is bigger than it looks: report speed becomes a Redshift performance problem, schema changes need a manual sync each time, you cannot change column types on the Zoho side at all, and no query tables.

My default for most teams is Data Import, because the operational surface is smaller and a scheduled sync is usually fresh enough for a dashboard people look at once a day. Live Connect earns its cost when the numbers genuinely need to be current, or when duplicating the data into a third-party system is a compliance problem.

Step 1: create a user that cannot do much

Do this before you open any network access, so that when the connection works you already know its blast radius.

-- A dedicated user for this connection and nothing else.
CREATE USER zoho_reader PASSWORD 'generate-a-long-random-one';

-- Reach the schema, then read the objects in it.
-- Both grants are needed; USAGE alone gets you nothing.
GRANT USAGE ON SCHEMA analytics TO zoho_reader;
GRANT SELECT ON ALL TABLES IN SCHEMA analytics TO zoho_reader;

-- The grant above covers what exists today. This covers what
-- gets created tomorrow, which is the part people forget and
-- then debug as a mysterious missing table three months later.
ALTER DEFAULT PRIVILEGES IN SCHEMA analytics
  GRANT SELECT ON TABLES TO zoho_reader;

Then check what you actually granted, rather than trusting that the statements did what you meant:

SELECT table_schema, table_name, privilege_type
FROM information_schema.table_privileges
WHERE grantee = 'zoho_reader'
ORDER BY table_schema, table_name;

If that returns rows from schemas you did not intend to publish, fix it now. Once the connection is live, someone will build a report on whatever is visible and removing access becomes a conversation instead of a command.

The password goes into a SaaS platform, so treat it accordingly: long, random, generated, stored in your password manager, and rotated when people leave.

Step 2: open exactly as much network as you need

First find out what your cluster currently looks like, because the console tells you less at a glance than this does:

aws redshift describe-clusters 
  --cluster-identifier my-warehouse 
  --query 'Clusters[0].{Endpoint:Endpoint.Address,Port:Endpoint.Port,Public:PubliclyAccessible,SecurityGroups:VpcSecurityGroups}'

That gives you the hostname Zoho needs, the real port, whether the cluster is currently public, and which security groups govern it. Redshift Serverless has an equivalent setting on the workgroup rather than a cluster, but the same reasoning applies.

Then add one narrow inbound rule per Zoho range. Not 0.0.0.0/0, which is what happens when someone is in a hurry and it never gets tightened afterwards:

aws ec2 authorize-security-group-ingress 
  --group-id sg-0123456789abcdef0 
  --protocol tcp 
  --port 5439 
  --cidr 203.0.113.0/24

Put a description on each rule in the console so that in a year somebody can tell which ranges belong to which vendor. An undocumented allow rule is one nobody will ever dare remove.

While you are in the cluster parameter group, turn on require_ssl so connections that do not use TLS are refused rather than merely discouraged. Parameter group changes need a cluster reboot, so do it in a window rather than mid-afternoon.

Test from outside before you touch Zoho

# Does anything answer on the port, from a machine outside your VPC?
nc -zv my-warehouse.abc123.eu-west-1.redshift.amazonaws.com 5439

# Then prove the credentials work, over TLS, as the new user.
psql "host=my-warehouse.abc123.eu-west-1.redshift.amazonaws.com 
      port=5439 dbname=analytics user=zoho_reader sslmode=require"

If both of those work from your laptop, the remaining variable is Zoho’s IP ranges rather than anything structural. If the first one hangs, it is the security group or public accessibility. If it connects but authentication fails, it is the user. Separating those two takes thirty seconds and saves an hour of guessing inside a web form that reports one generic error for both.

If you cannot make the cluster public

Plenty of organisations have a policy against it, and that policy is not unreasonable. Zoho’s answer for databases behind a firewall is Zoho Databridge, a lightweight agent you install on a machine inside your own network. It opens an outbound connection on 443 to Zoho and waits for requests, so nothing inbound has to be allowed at all. It runs on Windows, macOS and Linux, and it is designed for exactly this situation.

Worth checking before you commit to it: Databridge is documented against a long list of relational databases, and Redshift is not always named explicitly in that list. Since Redshift speaks the PostgreSQL wire protocol and has a JDBC driver, it is usually workable, but confirm with Zoho support for your plan rather than taking my word for it. If it fits, an outbound-only agent is a meaningfully better architecture than a public endpoint, and almost nobody setting this up knows the option exists.

Step 3: make the connection

With the groundwork done, this part is genuinely a form. In Zoho Analytics, create a workspace, choose to import from or connect to Amazon Redshift, and supply the endpoint hostname, the port, the database name, and the zoho_reader credentials.

Two expectations worth setting so you do not diagnose a non-problem:

• The first fetch takes a while, depending on volume and how fast your cluster responds. Zoho emails you when it finishes.
• An empty workspace before that fetch completes is normal. If you open it early it will show nothing, which looks exactly like a failure and is not one.

Step 4: choose what to expose, carefully

This is the screen people click through fastest and the one with the longest consequences. Whatever tables you select here become the interface between your warehouse and your dashboards, and every future upstream change becomes a potential BI incident.

Select views, not base tables. A reporting view with explicit casts, timezone conversion done in Redshift, and semi-structured columns already flattened gives you a stable contract. The tables underneath stay free to change.

CREATE OR REPLACE VIEW analytics.v_orders_bi AS
SELECT
    CAST(order_id     AS BIGINT)        AS order_id,
    CAST(order_status AS VARCHAR(64))   AS order_status,
    CAST(total_amount AS DECIMAL(18,2)) AS total_amount,
    created_at                          AS created_at_utc,
    CAST(CONVERT_TIMEZONE('UTC','Europe/London', created_at) AS DATE)
                                        AS order_date_local
FROM analytics.orders
WITH NO SCHEMA BINDING;

WITH NO SCHEMA BINDING makes it a late-binding view, so a nightly job can drop and rebuild the underlying table without Redshift refusing or the view disappearing. On a warehouse with a rebuild-style load, that alone prevents a recurring class of broken-dashboard morning.

Grant your reader access to the views specifically, and skip the blanket grant on the whole schema if you can. Fewer objects visible means fewer things somebody can accidentally build a report on.

Step 5: schedule it, then verify a real number

In Data Import mode, set the sync schedule to match how the data is actually produced. Syncing hourly when the warehouse loads once at 03:00 just means twenty-three pointless queries a day against your cluster, and Redshift bills by uptime and workload.

Then do the step everyone skips. Pick a closed period, run the same aggregate in both places, and compare:

SELECT DATE_TRUNC('day', created_at) AS day,
       COUNT(*)                      AS orders,
       SUM(total_amount)             AS revenue
FROM analytics.v_orders_bi
WHERE created_at_utc >= DATEADD(month, -1, DATE_TRUNC('month', GETDATE()))
  AND created_at_utc <  DATE_TRUNC('month', GETDATE())
GROUP BY 1
ORDER BY 1;

A connection that returns data is not the same as a connection that returns correct data. If daily figures agree but monthly ones do not, you have a timezone problem rather than a connection problem, and it is far easier to find now than six weeks into someone’s board pack.

Troubleshooting

Connection times out

Network, not credentials. A timeout means nothing answered. Check public accessibility, then the security group rule, then whether Zoho’s allowlist has changed since you configured it. A refused connection rather than a timeout usually means you reached the right host on the wrong port.

Authentication failed

The network is fine, which is genuine progress. Test the same credentials with psql from your own machine. Also check the database name: Redshift clusters often have both a default database and the one you actually use, and connecting to the wrong one authenticates fine and then shows you nothing.

Connected, but no tables listed

A permissions gap. USAGE on the schema and SELECT on the objects are separate grants and you need both. Run the information_schema.table_privileges query as a check rather than re-running the grants and hoping.

Worked on setup, fails intermittently later

Classic symptom of a partially stale IP allowlist: some of Zoho’s ranges reach you and some do not, so syncs succeed or fail depending on which host tries. Re-check the published list against your security group rules.

Sync succeeds but a new column never appears

In Live Connect, schema changes need a manual Sync Design, and an unresolved mismatch anywhere in the connection stops it fetching new column information at all. Clear the Mismatch tab completely, then sync again.

Reports are slow

In Live Connect that is your cluster, not Zoho. Look at the queries hitting Redshift and at sort and distribution keys on the underlying tables. Zoho also offers per-workspace caching for Live Connect with a configurable refresh interval, which trades freshness for speed.

Common mistakes

• Connecting with the cluster admin user because it was already in the password manager.
• Opening the Redshift port to 0.0.0.0/0 to make the test pass, then never tightening it.
• Copying an IP allowlist out of a blog post instead of Zoho’s own page.
• Granting USAGE without SELECT, or the reverse, and concluding the connector is broken.
• Forgetting ALTER DEFAULT PRIVILEGES, so tables created later are invisible.
• Pointing the connection at base tables rather than reporting views.
• Picking Live Connect for the freshness without accounting for the manual sync on every schema change.
• Assuming the setup failed because the workspace is empty during the initial fetch.
• Scheduling syncs far more often than the warehouse is actually loaded.
• Not enforcing TLS on the cluster.
• Declaring it done without reconciling a single number against Redshift.
• Leaving no note anywhere about which security group rules belong to which vendor.

Best practices

• A dedicated read-only user per integration, never a shared or admin account.
• Grant access to reporting views only, not whole schemas, where you can.
• One narrow security group rule per vendor range, described and dated.
• Enforce TLS with require_ssl rather than trusting the client to ask for it.
• Prefer an outbound-only agent over a public endpoint if your plan and setup support it.
• Decide Data Import versus Live Connect deliberately, because changing it means rebuilding.
• Expose late-binding views with explicit casts and timezone conversion done in Redshift.
• Match the sync schedule to your load schedule, not to how fresh you wish the data were.
• Reconcile at least one aggregate before anyone builds a dashboard on it.
• Rotate the connection password when people leave, and document where it lives.
• Test connectivity and credentials separately, from outside the VPC, before blaming the connector.

FAQ

Does Redshift have to be publicly accessible?

For a direct connection, yes: Zoho reaches your cluster inbound from its own IP addresses, so there has to be an endpoint it can resolve and reach. The alternative is Zoho Databridge, an agent inside your network that connects outbound on 443, which removes the inbound requirement entirely. Confirm Redshift support for it with Zoho before planning around it.

Which port does Zoho need open?

Whichever port your cluster listens on. 5439 is the Redshift default, but it can be set to something else when the cluster is created, so read it from describe-clusters rather than assuming.

Should I use Data Import or Live Connect?

Data Import for most cases: faster dashboards, query tables, automatic column syncing, and a smaller operational surface. Live Connect when the numbers must be current or copying data into a third party is a compliance issue, accepting slower reports and manual schema syncs. Decide before you build, because switching means starting the workspace over.

What permissions does the Zoho user need?

USAGE on the schema and SELECT on the objects you want reported on. Nothing else. Add ALTER DEFAULT PRIVILEGES so objects created later are covered without anybody having to remember.

Why is the workspace empty after I finish setup?

The initial fetch has not finished. Zoho emails you when it completes, and the workspace shows nothing until then. Give it time before you start pulling the configuration apart.

Can I limit which tables Zoho can see?

Yes, and you should. Grant the reader access only to a set of reporting views in a dedicated schema. That is a stronger control than the table picker in the interface, because it holds even if somebody later edits the connection.

Will this increase my Redshift bill?

Live Connect can, since every report load is a query. Data Import costs you one scheduled query run per sync. If cost matters, Data Import on a schedule matched to your load window is the cheaper shape by a wide margin.

The one thing to remember

Getting Zoho Analytics to talk to Redshift takes about twenty minutes. Getting it to talk to Redshift without handing a SaaS platform a superuser account and an open port takes about an hour, and that hour is the entire difference between a connection you can defend in an audit and one you quietly hope nobody asks about.

So do it in this order: least-privilege user first, narrow network access second, connection third, views rather than tables fourth, and a reconciled number before anyone builds a dashboard on it. Every step after the first is easier when the first one is already done.

Want this set up properly the first time?

Most of these connections get built under time pressure and inherit whatever shortcuts made the test pass. Work I take on:

• Setting up a Redshift to Zoho Analytics connection end to end, with least-privilege credentials and scoped network access.
• Reviewing an existing connection and reporting what it can actually reach, then narrowing it without breaking reports.
• Building the reporting view layer in Redshift: explicit casts, timezone handling, flattened semi-structured columns, late binding.
• Network architecture for BI access, including outbound-agent options where a public endpoint is not acceptable.
• Choosing between Data Import and Live Connect based on your actual freshness, cost and compliance constraints.
• Reconciliation checks so dashboard numbers are verified against the warehouse rather than assumed.

Tell me whether your cluster is currently public and which user the connection runs as, and I will tell you what I would change first.

The post Connect Amazon Redshift to Zoho Analytics Without Putting Your Warehouse on the Internet appeared first on John Nessime.