<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Firewall | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/firewall/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/firewall/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Tue, 13 Oct 2026 06:06:40 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Firewall | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/firewall/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Your WordPress 403 Isn&#8217;t the Rule You Think: Tuning ModSecurity False Positives</title>
		<link>https://john-nessime.com/blog/web-security/modsecurity-false-positives-wordpress/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 13 Oct 2026 06:06:40 +0000</pubDate>
				<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[WordPress]]></category>
		<category><![CDATA[Anomaly Scoring]]></category>
		<category><![CDATA[Apache]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[Coraza]]></category>
		<category><![CDATA[False Positives]]></category>
		<category><![CDATA[Firewall]]></category>
		<category><![CDATA[ModSecurity]]></category>
		<category><![CDATA[Nginx]]></category>
		<category><![CDATA[OWASP CRS]]></category>
		<category><![CDATA[Rule Exclusions]]></category>
		<category><![CDATA[Server Hardening]]></category>
		<category><![CDATA[Web Application Firewall]]></category>
		<category><![CDATA[WordPress Security]]></category>
		<category><![CDATA[wp-admin]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=434</guid>

					<description><![CDATA[<p>A 403 from ModSecurity almost never names the rule you need to change. Under anomaly scoring, detection rules only add to a score and a separate blocking rule returns the error. This guide covers reading the audit log backwards, choosing between the four exclusion mechanisms, the WordPress failure families that cause most false positives, and why raising the anomaly threshold quietly destroys your coverage.</p>
<p>The post <a href="https://john-nessime.com/blog/web-security/modsecurity-false-positives-wordpress/">Your WordPress 403 Isn&#8217;t the Rule You Think: Tuning ModSecurity False Positives</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket almost always reads the same way. Someone spent twenty minutes writing a post, hit Update, and got a bare <code>403 Forbidden</code>. Nothing in the WordPress debug log. Nothing in the PHP error log. The plugin they blame was not updated. The post they wrote yesterday saved fine.</p>



<p class="wp-block-paragraph">Then you open the ModSecurity audit log and find eleven rule matches on a single request, none of them obviously the culprit.</p>



<p class="wp-block-paragraph">Dealing with ModSecurity false positives on WordPress is not hard once you understand how the OWASP Core Rule Set decides to block. It is very hard if you go looking for &#8220;the rule that blocked me,&#8221; because under the default configuration that rule does not exist. This post covers reading the log, choosing the right exclusion mechanism, the WordPress failure families that generate most of the noise, and the tuning knobs that quietly gut your coverage.</p>



<h2 class="wp-block-heading">The invisible part: no single rule blocked the request</h2>



<p class="wp-block-paragraph">The Core Rule Set runs in anomaly scoring mode by default. Individual detection rules do not deny anything. They match, they log, and they add to a running score with <code>setvar</code>. At the end of the request phase, a separate blocking evaluation rule compares the accumulated score against a threshold and returns the 403.</p>



<p class="wp-block-paragraph">Scores come from rule severity. A CRITICAL match adds 5, ERROR adds 4, WARNING adds 3, NOTICE adds 2. The inbound threshold ships at 5 and the outbound threshold at 4. So one critical match is enough on its own, and so are two notices plus a warning.</p>



<p class="wp-block-paragraph">That is the failure mode nobody warns you about. The log line that produced your 403 lives in <code>REQUEST-949-BLOCKING-EVALUATION.conf</code> and it says the inbound anomaly score was exceeded. It does not name what matched. Take that ID and disable it and you have not fixed a false positive, you have turned off blocking for the whole site while leaving every detection rule running and logging. The site will look fixed. It is not protected.</p>



<p class="wp-block-paragraph">First rule of tuning, then: the blocking rule is a symptom. Read backwards from it.</p>



<h2 class="wp-block-heading">Read the log before you touch a rule</h2>



<p class="wp-block-paragraph">Every alert carries a <code>unique_id</code>. That value ties the error log line to the full audit log record for the same transaction, and it is the only reliable way to see every rule that fired on one request instead of a scattering of lines from different visitors.</p>



<pre class="wp-block-code"><code># Paths vary by build. Check SecAuditLog in modsecurity.conf for yours.
tail -n 200 /var/log/httpd/error_log | grep -i modsecurity

# Then pull the whole transaction using the unique_id from that output
grep -F 'Zm9vYmFyCg' /var/log/httpd/modsec_audit.log</code></pre>



<p class="wp-block-paragraph">From each match, four fields matter and the rest is noise:</p>



<ul class="wp-block-list">
<li><strong>The rule ID</strong>, as <code>[id "941320"]</code>. This is what you exclude.</li>



<li><strong>The matched variable</strong>, at the end of the pattern-match message as something like <code>at ARGS:content</code>. This is what you exclude it <em>from</em>.</li>



<li><strong>The URI</strong>, which tells you how narrowly you can scope the exclusion.</li>



<li><strong>The paranoia level tag</strong>, as <code>[tag "paranoia-level/2"]</code>. If every alert is tagged level 2 or higher, the real fix may be a paranoia level change rather than an exclusion.</li>
</ul>



<p class="wp-block-paragraph">One warning most guides skip: audit logs of false positives contain the payload that triggered them. If a user&#8217;s password happens to look like a shell command, it lands in your log in plaintext. Treat these files as sensitive, rotate them, and redact before pasting anything into a public bug report.</p>



<h2 class="wp-block-heading">Pick the right exclusion mechanism</h2>



<p class="wp-block-paragraph">ModSecurity gives you four ways to write an exclusion, differing on two axes: applied once at startup or evaluated per request, and removing the whole rule or just one variable from it.</p>



<ul class="wp-block-list">
<li><code>SecRuleRemoveById</code> and <code>SecRuleRemoveByTag</code> delete a rule or a whole tagged category at configure time. Blunt, cheap, global.</li>



<li><code>SecRuleUpdateTargetById</code> and <code>SecRuleUpdateTargetByTag</code> strip one variable from a rule at configure time, leaving the rule live for everything else.</li>



<li><code>ctl:ruleRemoveById</code> and <code>ctl:ruleRemoveByTag</code> do the same removal, but only for transactions matching a condition you write.</li>



<li><code>ctl:ruleRemoveTargetById</code> and <code>ctl:ruleRemoveTargetByTag</code> strip one variable, conditionally. This is the surgical option and the one to reach for first.</li>
</ul>



<p class="wp-block-paragraph">Runtime exclusions cost a little, since each is an extra rule evaluated on every request. That overhead is small next to the regex work the rule set already does, and the precision is worth it.</p>



<h3 class="wp-block-heading">Placement decides whether an exclusion works at all</h3>



<p class="wp-block-paragraph">This is where most people lose an afternoon. The two types go in opposite places, for mechanical reasons rather than arbitrary ones.</p>



<ul class="wp-block-list">
<li><strong>Configure-time exclusions go after the CRS include.</strong> They remove rules, and a rule has to exist before you can remove it.</li>



<li><strong>Runtime exclusions go before the CRS include.</strong> They change how rules behave for this transaction, and changing a rule after it has already run does nothing.</li>
</ul>



<p class="wp-block-paragraph">The Core Rule Set ships two files for exactly this, both with an <code>.example</code> suffix you drop before they load. <code>REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf</code> holds runtime exclusions, and <code>RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf</code> holds configure-time ones. Use them. Local files that survive a rule set update are the entire point.</p>



<h2 class="wp-block-heading">Five failure families behind most ModSecurity false positives</h2>



<p class="wp-block-paragraph">Almost everything you will hit on WordPress falls into one of these. Recognizing the family tells you the shape of the fix before you read a single log line.</p>



<h3 class="wp-block-heading">1. Post content that legitimately contains markup</h3>



<p class="wp-block-paragraph">WordPress lets editors put HTML in posts. It maintains its own allowed-tag list, sanitizes server side, and that mechanism has been reviewed hard over the years. The Core Rule Set knows none of this. It sees an <code>&lt;h1&gt;</code> or an inline <code>style</code> attribute in a POST parameter and matches an XSS rule.</p>



<p class="wp-block-paragraph">This produces the &#8220;it saved yesterday&#8221; ticket, because it is content-dependent. Plain paragraphs pass. A pasted table with attributes does not. The fix is a target exclusion scoped to the save endpoint and the specific parameter, never a global rule removal:</p>



<pre class="wp-block-code"><code># Runtime exclusion. Goes in REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf
# Replace 941320 and the parameter with what your audit log actually shows.
SecRule REQUEST_URI "@beginsWith /wp-admin/post.php" 
    "id:1000,
    phase:1,
    pass,
    t:none,
    nolog,
    ctl:ruleRemoveTargetById=941320;ARGS:content"</code></pre>



<p class="wp-block-paragraph">Pick your own IDs from a range CRS will never use. Reusing an existing ID is a startup error rather than a silent one, which is the good kind of mistake.</p>



<h3 class="wp-block-heading">2. REST and admin-ajax calls that fail with no visible error</h3>



<p class="wp-block-paragraph">This family is nastier, because there is no error page. The browser hits <code>/wp-json/</code> or <code>/wp-admin/admin-ajax.php</code>, gets a 403, and the JavaScript either swallows it or shows a generic &#8220;updating failed&#8221; toast. Autosave stops. An editor panel stays empty. A form plugin silently drops submissions. Users report this as flakiness rather than a security block, so it sits unfixed for months. When someone says the editor is &#8220;just slow sometimes,&#8221; open the browser network tab and look for 403s before going anywhere near PHP.</p>



<p class="wp-block-paragraph">Scoping differs here. REST routes are structured, so use <code>@rx</code> against <code>REQUEST_FILENAME</code> to hit a route family rather than one path. For <code>admin-ajax.php</code>, the useful discriminator is usually the <code>action</code> parameter, since that one endpoint multiplexes dozens of unrelated plugin handlers.</p>



<h3 class="wp-block-heading">3. Media uploads that break at a size boundary</h3>



<p class="wp-block-paragraph">Multipart uploads have two independent failure paths that look identical from the browser. The first is a genuine rule match on the multipart structure, or on binary content containing a byte sequence some rule dislikes. That shows up with a rule ID and you tune it normally.</p>



<p class="wp-block-paragraph">The second is an engine limit, not a rule. <code>SecRequestBodyLimit</code> caps the total body ModSecurity will buffer, <code>SecRequestBodyNoFilesLimit</code> caps the non-file portion, and <code>SecRequestBodyLimitAction</code> decides whether an oversized body is rejected outright or inspected partially. If uploads fail above a consistent size and no CRS rule ID appears anywhere in the log, stop tuning rules. That is a limit, and the fix is in engine config.</p>



<p class="wp-block-paragraph">Raising them costs memory per concurrent request, and partial inspection leaves part of the body unscanned. Decide which trade you are making rather than bumping the number until the ticket closes.</p>



<h3 class="wp-block-heading">4. Cookies with dynamic names</h3>



<p class="wp-block-paragraph">WordPress authentication cookies carry a hash in the name, so a literal exclusion is impossible. Configure-time directives accept a regular expression in the target if you wrap it in forward slashes:</p>



<pre class="wp-block-code"><code># Configure-time exclusion. Goes in RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf
SecRuleUpdateTargetById 942440 "!REQUEST_COOKIES:/^wordpress_logged_in_.*/"</code></pre>



<p class="wp-block-paragraph">The catch is significant: the <code>ctl</code> action supports no regular expressions at all. If you need a pattern match on a variable name, you are forced into the configure-time form, which applies site-wide rather than to one path. That single constraint drives more design decisions than people expect.</p>



<h3 class="wp-block-heading">5. Chained rules that refuse to be tuned</h3>



<p class="wp-block-paragraph">Some rules are written as a chain of conditions. Target exclusions only apply to the first rule in the chain, so if the variable you want to strip is referenced further down, nothing you write has any effect and you will burn an hour convincing yourself the config file is not loading.</p>



<p class="wp-block-paragraph">When a target exclusion visibly does nothing on a rule you can watch matching, check whether it is chained. If it is, your only option is removing the whole rule under a narrow condition. Accept the wider blast radius, or change the request instead.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Install the WordPress plugin before writing anything custom</h2>



<p class="wp-block-paragraph">The Core Rule Set maintains a WordPress rule exclusions package. In CRS 3 it was built in and switched on with a variable in <code>crs-setup.conf</code>. In CRS 4 it moved out into a separate plugin published in the CRS plugin registry, and it is not installed by default.</p>



<p class="wp-block-paragraph">That catches people upgrading. Exclusions that used to work vanish, WordPress starts throwing 403s again, and the setup variable they remember is gone from the config file. The reasoning behind the split is sound: fewer rules loaded means less code that can carry a bug, and one of the old exclusion packages did carry a bypass.</p>



<p class="wp-block-paragraph">Plugins load through wildcard includes in a specific order. Get the order wrong and the plugin silently does nothing:</p>



<pre class="wp-block-code"><code>Include /path/to/crs/crs-setup.conf

Include /path/to/crs/plugins/*-config.conf
Include /path/to/crs/plugins/*-before.conf

Include /path/to/crs/rules/*.conf

Include /path/to/crs/plugins/*-after.conf</code></pre>



<p class="wp-block-paragraph">Two limits before you lean on it. The plugin covers vanilla WordPress only, so false positives from third-party plugins, page builders and form plugins are still yours to tune. And the comment field is deliberately left out of the exclusions, because comment forms have a long history as a real attack surface. If your comment form throws false positives, that is a decision the maintainers made on purpose, and overriding it is a genuine risk trade rather than a gap in their package.</p>



<p class="wp-block-paragraph">On a server hosting more than one application, scope the plugin to the WordPress vhost using <code>WebAppID</code> or a Host header condition. WordPress exclusions applied to your unrelated API are a hole you did not mean to open.</p>



<h2 class="wp-block-heading">The knobs that look like a fix and are not</h2>



<p class="wp-block-paragraph">When exclusions get tedious, two shortcuts start looking attractive. Both cost more than they appear to.</p>



<p class="wp-block-paragraph"><strong>Raising the anomaly threshold.</strong> Going from 5 to 20 makes most false positives vanish, and it also means any attack category that scores low can never trigger a block. Detections that only ever produce a single critical match become unblockable. The CRS documentation is direct here: keep the threshold at the default and change the paranoia level instead. Raising it is a temporary tuning aid, not a configuration.</p>



<p class="wp-block-paragraph"><strong>Dropping the paranoia level.</strong> Legitimate, but often applied backwards. Level 1 is designed to produce few false positives. Levels 2 and up add progressively more aggressive rules and progressively more tuning work. If your alerts are all tagged level 2, dropping to level 1 is the correct fix and it is honest about what you gave up.</p>



<p class="wp-block-paragraph">The better tool for moving up is the split between blocking and detection paranoia levels. Set <code>tx.blocking_paranoia_level</code> to what you actually enforce and <code>tx.detection_paranoia_level</code> higher. Rules from the higher level execute and log, but only rules up to the blocking level count toward the score. You get to see what level 2 would have caught, in production, on real traffic, without anyone getting blocked. Then you tune, then you promote. The performance cost lands as soon as you raise the detection level, though, since those rules run either way.</p>



<p class="wp-block-paragraph">Two smaller settings help here. <code>tx.sampling_percentage</code> sends only a fraction of requests into the rule set, a gentler introduction for a busy site than blocking everything on day one. And <code>tx.reporting_level</code> can log requests that <em>would</em> have been blocked at a higher paranoia level, which is exactly the signal you want while climbing.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<p class="wp-block-paragraph"><strong>The exclusion has no effect.</strong> Check placement first. A runtime exclusion loaded after the rules, or a configure-time exclusion loaded before them, fails silently. Then check for a chained rule. Then confirm the file is being included at all by breaking it deliberately with a syntax error and seeing whether the server refuses to start.</p>



<p class="wp-block-paragraph"><strong>It works on one site and not another.</strong> Different vhosts load different config. On control panels, per-domain overrides usually sit in a separate file from the global rule set.</p>



<p class="wp-block-paragraph"><strong>It worked and then stopped.</strong> Something updated the rule set. If you edited CRS rule files directly, your edits are gone, which is the whole argument against editing them.</p>



<p class="wp-block-paragraph"><strong>Nothing appears in the log at all.</strong> Confirm the rule engine is on, and that auditing is enabled with enough log parts to include the request body. An engine in detection only mode logs without blocking, which is confusing if you did not know you were in it.</p>



<p class="wp-block-paragraph"><strong>You are on shared hosting and cannot edit any of this.</strong> Some hosts expose a per-domain exclusion box, some let you disable rule IDs, some give you nothing. That is a real ceiling. Providers that hand you root, such as InterServer or Contabo, let you own the rule set and its update cadence, which on a site with genuine tuning needs beats the panel convenience you trade away.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Disabling the blocking evaluation rule because it showed up in the log. It reports the score, it detects nothing, and removing it disables blocking site-wide.</li>



<li>Editing CRS rule files directly. Every update reverts you, and you now maintain a fork by hand.</li>



<li>Excluding by message rather than by ID. Messages carry dynamic data and special characters, so the match is fragile.</li>



<li>Removing an entire attack tag because two rules misfired. <code>SecRuleRemoveByTag attack-sqli</code> is defensible when nothing behind the WAF touches SQL. On WordPress it is not.</li>



<li>Excluding <code>/wp-admin/</code> wholesale, or stacking multiple rule set vendors. Admin is where a compromised session does the most damage, and overlapping rule sets make attribution guesswork.</li>



<li>Leaving known false positives in place. Real attacks get lost in the noise, and the logs accumulate user data you never intended to collect.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Start in detection only. Collect real traffic, tune what you find, then enable blocking. Enforcing first is how you learn about broken checkout at two in the morning.</li>



<li>Install the WordPress exclusions plugin before writing custom rules. Do not hand-write what upstream already maintains.</li>



<li>Write the narrowest exclusion that resolves the alert: one rule, one variable, one path. Comment it with the rule ID, the endpoint, and why you judged it safe. Six months later that comment is the only thing between you and a blind revert.</li>



<li>Keep exclusion files in version control and deploy them with the rest of your config. Test rule set updates on staging, since new signatures land regularly.</li>



<li>Ship audit logs somewhere queryable. Grepping a growing text file stops scaling fast, and a platform like Grafana Cloud turns &#8220;which rule fires most&#8221; into a query instead of an afternoon.</li>



<li>If an edge WAF such as Cloudflare sits in front of the origin, decide deliberately which layer owns which rule. Two WAFs scanning the same body doubles your false positive surface for no extra coverage.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How do I find which ModSecurity rule blocked my WordPress request?</h3>



<p class="wp-block-paragraph">Take the <code>unique_id</code> from the 403 entry in the error log and grep the audit log for it. That gives you every rule that fired on the transaction. The blocking evaluation rule is what returned the 403, but the detection rules listed alongside it are what you tune.</p>



<h3 class="wp-block-heading">Should I disable ModSecurity for WordPress instead of tuning it?</h3>



<p class="wp-block-paragraph">It is a defensible choice if the alternative is blocking mode with unresolved false positives, since a WAF everyone has learned to route around is worse than none. But detection only mode gives you the logs with none of the breakage, and that is almost always the better middle ground while you work through the tuning.</p>



<h3 class="wp-block-heading">Is the WordPress rule exclusions plugin enough on its own?</h3>



<p class="wp-block-paragraph">For a stock install on a standard theme, usually. It covers vanilla WordPress and both block-based and classic themes. It does not cover third-party plugins, page builders, or theme frameworks, and those generate most of the false positives on a real site.</p>



<h3 class="wp-block-heading">Why do false positives come back after a rule set update?</h3>



<p class="wp-block-paragraph">Either you edited rule files directly and the update overwrote them, or the update shipped new signatures your existing exclusions do not cover. The first is avoidable by keeping every change in the exclusion files. The second is the cost of running a maintained rule set, and it is why updates belong on staging first.</p>



<h3 class="wp-block-heading">Does ModSecurity slow WordPress down?</h3>



<p class="wp-block-paragraph">It adds per-request work, mostly regular expression evaluation against request data, and the cost scales with paranoia level and body size. The settings that matter are the paranoia level, the request body limits, and whether response body inspection is on. Measure on your own traffic rather than trusting someone else&#8217;s benchmark.</p>



<h3 class="wp-block-heading">Is ModSecurity still maintained?</h3>



<p class="wp-block-paragraph">The engine moved to the OWASP Foundation after Trustwave stepped back from commercial support, and it still receives fixes, though feature work is slower than it once was. The Core Rule Set is a separate project on its own cadence and is actively developed. Coraza is a compatible engine running the same rule language, worth evaluating for new deployments.</p>



<h2 class="wp-block-heading">Wrapping up</h2>



<p class="wp-block-paragraph">The thing to hold on to about ModSecurity false positives on WordPress is that the rule which returned your 403 is almost never the rule you need to change. Anomaly scoring separates detection from blocking on purpose, and reading the log backwards from the blocking decision to the detection rules that fed it is the whole skill.</p>



<p class="wp-block-paragraph">Everything else follows. Install the maintained WordPress exclusions instead of reinventing them. Write the narrowest exclusion you can, in the file that survives updates. Move the paranoia level, not the threshold.</p>



<h2 class="wp-block-heading">Need help tuning your WAF?</h2>



<p class="wp-block-paragraph">Most of the WAF work I take on is untangling exactly this: a rule set that is technically installed and practically ignored, because nobody had time to work through the false positives. I can help with:</p>



<ul class="wp-block-list">
<li>Triaging your audit logs and mapping every recurring false positive to the rule, variable and endpoint that produced it</li>



<li>Writing scoped exclusion sets for WooCommerce, form plugins, page builders and custom REST routes that upstream packages do not cover</li>



<li>Moving a site off direct rule file edits and onto exclusion files that survive rule set updates</li>



<li>Planning a CRS 3 to CRS 4 migration, including the exclusion package to plugin transition and the paranoia level variable renames</li>



<li>Running a staged rollout from detection only to blocking, with the reporting and sampling settings that make it safe</li>



<li>Shipping ModSecurity audit logs into a queryable log platform so rule tuning becomes a dashboard instead of a grep</li>
</ul>



<p class="wp-block-paragraph">If you have a blocked request you cannot explain, send me the audit log entry and your exclusion file and I will tell you what is firing and why.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/web-security/modsecurity-false-positives-wordpress/">Your WordPress 403 Isn&#8217;t the Rule You Think: Tuning ModSecurity False Positives</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Bootstrap a VPS With Ansible in One Run Without Locking Yourself Out</title>
		<link>https://john-nessime.com/blog/devops/bootstrap-vps-with-ansible/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 13 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Linux]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Ansible]]></category>
		<category><![CDATA[Ansible Collections]]></category>
		<category><![CDATA[Ansible Vault]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[cloud-init]]></category>
		<category><![CDATA[Fail2ban]]></category>
		<category><![CDATA[Firewall]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Infrastructure as Code]]></category>
		<category><![CDATA[OpenSSH]]></category>
		<category><![CDATA[Server Hardening]]></category>
		<category><![CDATA[Server Provisioning]]></category>
		<category><![CDATA[SSH]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Systemd]]></category>
		<category><![CDATA[UFW]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=204</guid>

					<description><![CDATA[<p>A single Ansible run can take a blank VPS to a hardened, reproducible server. The hard part is not the tasks, it is the order. Here is the ordering that keeps you logged in, the failures that pass green and bite two weeks later, and how to make the second run as safe as the first.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/bootstrap-vps-with-ansible/">Bootstrap a VPS With Ansible in One Run Without Locking Yourself Out</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The playbook finishes green. Thirty-odd tasks, nothing failed, and the recap says the box is done. Two weeks later it reboots after a kernel update and nobody can get in on the port the playbook supposedly set.</p>



<p class="wp-block-paragraph">That is the real problem with trying to bootstrap a VPS with Ansible in a single run. The failures that cost you are not the ones that turn the recap red. They are the tasks that report <code>changed</code>, do nothing useful, and leave you believing the server is hardened when it is not.</p>



<p class="wp-block-paragraph">This post covers the ordering that keeps you connected while you dismantle your own access, the silent failure modes that only surface on reboot, and how to make the second run of the playbook as safe as the first. The examples are Debian and Ubuntu flavoured with notes for RHEL-family images, and they assume a plain provider VPS from somewhere like InterServer, Hetzner, DigitalOcean or Vultr, not a pre-baked golden image.</p>



<h2 class="wp-block-heading">What a one-run bootstrap actually has to survive</h2>



<p class="wp-block-paragraph">A bootstrap playbook is unusual because it modifies the thing it is standing on. Every other playbook you write assumes a working connection. This one has to change the user it connects as, the port it connects to, and the firewall that lets it connect at all, without dropping the run.</p>



<p class="wp-block-paragraph">Four things break it, and they break it in different ways:</p>



<ul class="wp-block-list">
<li><strong>The chicken-and-egg problem.</strong> Ansible needs Python on the target to run almost any module, and a minimal image may not have it.</li>

<li><strong>Lockout.</strong> You remove your own access before proving the replacement access works.</li>

<li><strong>Non-idempotent tasks.</strong> The first run is fine. The second run is where the damage happens.</li>

<li><strong>Missing collections.</strong> The playbook that worked on your laptop fails on the CI runner because half the modules do not ship with <code>ansible-core</code>.</li>
</ul>



<p class="wp-block-paragraph">Everything below is organised around those four, because that is how they actually show up.</p>



<h2 class="wp-block-heading">The chicken-and-egg problem: no Python, no modules</h2>



<p class="wp-block-paragraph">Most modern cloud images ship Python 3 and this never comes up. Minimal Debian netinstall images and some stripped provider templates do not, and then your very first task fails with a message about the interpreter before it has done anything at all.</p>



<p class="wp-block-paragraph">The escape hatch is <code>ansible.builtin.raw</code>, which pipes a command straight down the SSH connection without needing Python on the other end. Pair it with <code>gather_facts: false</code>, because fact gathering is itself a Python module and will fail before your bootstrap task ever runs.</p>



<pre class="wp-block-code"><code>- name: Make the host manageable
  hosts: new_servers
  gather_facts: false
  become: true
  tasks:
    - name: Check whether a Python interpreter exists
      ansible.builtin.raw: test -e /usr/bin/python3
      register: python_present
      changed_when: false
      failed_when: false

    - name: Install Python if it is missing
      ansible.builtin.raw: apt-get update &amp;&amp; apt-get install -y python3
      when: python_present.rc != 0
      changed_when: true

    - name: Gather facts now that we can
      ansible.builtin.setup:</code></pre>



<p class="wp-block-paragraph">Two details worth understanding rather than copying. <code>failed_when: false</code> on the check turns a non-zero exit into data instead of a failure, so you can branch on it. And <code>ansible.builtin.setup</code> is the fact-gathering module called explicitly, which means from that point on <code>ansible_distribution</code> and friends are available even though the play started with gathering disabled.</p>



<p class="wp-block-paragraph">If you are provisioning through a provider that supports user-data, doing this in cloud-init instead is cleaner. Let cloud-init handle first-boot truths, then let Ansible own everything that will change later.</p>



<h2 class="wp-block-heading">Failure family one: locking yourself out while the playbook says ok</h2>



<p class="wp-block-paragraph">This is the one that actually costs a rebuild, and it is entirely an ordering problem. The tasks are all correct in isolation. Run them in the wrong sequence and the playbook cheerfully removes your way back in before it has confirmed the replacement works.</p>



<h3 class="wp-block-heading">The order that survives</h3>



<p class="wp-block-paragraph">The rule is simple: <strong>nothing that removes access happens until something has proven the new access works.</strong> In practice:</p>



<ol class="wp-block-list">
<li>Connect as whatever the provider handed you, usually <code>root</code> or a sudo-capable default user.</li>

<li>Create the admin account, add it to the sudo or wheel group, install its authorized key.</li>

<li>Write the sudoers drop-in and validate it with <code>visudo -cf</code>.</li>

<li><strong>Prove you can log in as that account and escalate.</strong> This is the step everyone skips.</li>

<li>Install the firewall and allow SSH <em>before</em> enabling it.</li>

<li>Write the sshd configuration, validate it, restart, reconnect on the new port and confirm.</li>

<li>Only now disable root login and password authentication.</li>
</ol>



<p class="wp-block-paragraph">Step four is a real task, not a comment. It runs from the control node against the target, using the credentials you are about to depend on:</p>



<pre class="wp-block-code"><code>- name: Prove the new admin account can log in and escalate
  ansible.builtin.command: id -u
  become: true
  changed_when: false
  vars:
    ansible_user: "{{ admin_user }}"
    ansible_ssh_private_key_file: "{{ admin_key_path }}"</code></pre>



<p class="wp-block-paragraph">Setting connection variables at the task level means this single task uses the new identity while the rest of the play carries on as root. If the key is wrong or sudo is misconfigured, the play stops here with root login still intact, which is exactly what you want.</p>



<h3 class="wp-block-heading">The socket activation trap</h3>



<p class="wp-block-paragraph">This is the failure that passes green and bites on reboot. On Ubuntu images from 22.10 onward, sshd is started through systemd socket activation rather than as a persistent service. <code>ssh.socket</code> owns the listening port, and <code>Port</code> in <code>sshd_config</code> is not what decides where the daemon listens.</p>



<p class="wp-block-paragraph">So your playbook writes <code>Port 2222</code>, restarts the service, reports changed, and the box carries on listening on 22. Your existing session still works. Your firewall rule for 2222 looks fine. Nothing indicates a problem until something forces a reconnect.</p>



<p class="wp-block-paragraph">The fix is to write a socket override as well, and to check which mechanism the image actually uses rather than assuming:</p>



<pre class="wp-block-code"><code># Is the socket unit in play on this host?
systemctl is-enabled ssh.socket

# What is actually listening, and under which unit?
ss -tlnp | grep -E 'ssh|:22'</code></pre>



<p class="wp-block-paragraph">If <code>ssh.socket</code> is enabled, the override needs an empty <code>ListenStream=</code> first to clear the inherited default, then the real value. Miss the empty line and the host listens on both ports, which is worse than either outcome alone because your hardening looks applied and is not.</p>



<p class="wp-block-paragraph">Honestly, the simpler answer for most single-server setups is to leave SSH on 22 and lean on key-only authentication plus a firewall source restriction. Moving the port cuts log noise from opportunistic scanners. It does not stop anyone who is actually looking at you.</p>



<h3 class="wp-block-heading">Firewall ordering</h3>



<p class="wp-block-paragraph">UFW defaults to denying inbound traffic when you enable it. Enable first, add the SSH rule second, and the run dies mid-task with an <code>UNREACHABLE</code> that will not recover. Rules go in first, always:</p>



<pre class="wp-block-code"><code>- name: Allow SSH before the firewall is armed
  community.general.ufw:
    rule: allow
    port: "{{ ssh_port }}"
    proto: tcp

- name: Set default inbound policy
  community.general.ufw:
    direction: incoming
    policy: deny

- name: Enable UFW
  community.general.ufw:
    state: enabled</code></pre>



<p class="wp-block-paragraph">Two extra things bite here. Provider-level firewalls, the ones configured in the control panel rather than on the host, are a second layer that Ansible has no visibility into. If you open a port on the host and traffic still does not arrive, check there before debugging iptables. And if you moved the SSH port, the host firewall rule and the provider firewall rule both need updating, in that order, before the daemon restarts.</p>



<h2 class="wp-block-heading">Failure family two: the second run is the dangerous one</h2>



<p class="wp-block-paragraph">A bootstrap playbook you run once is a shell script with extra syntax. The value only appears when you can run it again on the same host, in six months, and have it either do nothing or fix drift. That is idempotency, and the usual killers are <code>shell</code> and <code>command</code> tasks with no guard.</p>



<p class="wp-block-paragraph">An unguarded <code>shell</code> task reports changed every single time, which means any handler it notifies fires every single time. That is how a playbook you thought was a no-op ends up restarting sshd on a production box during a routine drift check.</p>



<p class="wp-block-paragraph">Guard them properly:</p>



<ul class="wp-block-list">
<li><code>creates:</code> or <code>removes:</code> makes the task skip when the target state already exists. Prefer this.</li>

<li><code>changed_when: false</code> for read-only commands so they stop lying about changing things.</li>

<li><code>when:</code> driven by a registered check, for anything conditional.</li>
</ul>



<p class="wp-block-paragraph">The other idempotency trap is <code>lineinfile</code> against <code>sshd_config</code>. It works, and it is fragile in a specific way: the regex has to match the shipped file, and different distributions ship different defaults, commented or not. Writing a drop-in file is more predictable:</p>



<pre class="wp-block-code"><code>- name: Write the sshd hardening drop-in
  ansible.builtin.template:
    src: 99-hardening.conf.j2
    dest: /etc/ssh/sshd_config.d/99-hardening.conf
    owner: root
    group: root
    mode: "0600"
  notify: Restart sshd

- name: Validate the full sshd configuration
  ansible.builtin.command: sshd -t
  changed_when: false</code></pre>



<p class="wp-block-paragraph">Note that the validation runs against the whole configuration rather than the drop-in in isolation. A drop-in is not a valid standalone config file, so <code>validate:</code> on the template task with <code>sshd -t -f %s</code> will fail on a temporary file that has no host keys and no context. Run <code>sshd -t</code> as its own task afterwards instead, and let the handler restart only if it passes.</p>



<p class="wp-block-paragraph">Two prerequisites for the drop-in approach. The base <code>sshd_config</code> needs an <code>Include /etc/ssh/sshd_config.d/*.conf</code> line, which recent Debian, Ubuntu and RHEL 9 images have but older ones may not. And in sshd configuration the <em>first</em> value for a keyword wins, so the include has to appear near the top of the file for your drop-in to override anything below it. Check both before relying on the mechanism.</p>



<h2 class="wp-block-heading">Failure family three: the collections are not where you think</h2>



<p class="wp-block-paragraph"><code>ansible-core</code> ships the <code>ansible.builtin</code> namespace and essentially nothing else. The full <code>ansible</code> package bundles a large set of community collections, which is why a playbook that runs on your laptop can fail on a lean CI runner with a module-not-found error.</p>



<p class="wp-block-paragraph">Things people assume are built in and are not: <code>ufw</code> and <code>timezone</code> live in <code>community.general</code>. <code>authorized_key</code>, <code>sysctl</code> and <code>mount</code> live in <code>ansible.posix</code>. Declare them and pin them:</p>



<pre class="wp-block-code"><code># requirements.yml
collections:
  - name: community.general
  - name: ansible.posix

# install before the run, and in CI
ansible-galaxy collection install -r requirements.yml</code></pre>



<p class="wp-block-paragraph">Use fully qualified names everywhere. Short names still resolve, but they resolve through a lookup that can pick a different module when two collections define the same name. <code>ansible-lint</code> flags this and it is worth listening to.</p>



<h2 class="wp-block-heading">Failure family four: secrets in the repository</h2>



<p class="wp-block-paragraph">Bootstrap playbooks accumulate secrets faster than anything else you write: initial passwords, API tokens for monitoring agents, TLS material, backup credentials. They also tend to be the first repository someone shares with a client.</p>



<p class="wp-block-paragraph">Ansible Vault is the low-friction option and it is genuinely fine for a small estate. Encrypt the variable file, not the playbook, so diffs stay readable:</p>



<pre class="wp-block-code"><code>ansible-vault encrypt group_vars/all/vault.yml
ansible-playbook site.yml --ask-vault-pass</code></pre>



<p class="wp-block-paragraph">The honest trade-off: Vault gives you one shared passphrase with no rotation story and no audit trail. Once more than two people run the playbook, or a client needs their own credentials kept separate from yours, move to something with real access control. That is more moving parts, so do not reach for it on day one.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Putting it together: how to bootstrap a VPS with Ansible end to end</h2>



<p class="wp-block-paragraph">Structure matters less than order, but a shape that works well is a single <code>site.yml</code> calling roles in dependency order, with the risky access changes fenced off at the end.</p>



<pre class="wp-block-code"><code># inventory.yml
new_servers:
  hosts:
    web01:
      ansible_host: 203.0.113.10
      ansible_user: root
      ansible_port: 22
  vars:
    admin_user: deploy
    ssh_port: 22</code></pre>



<p class="wp-block-paragraph">Then the run itself, in this order:</p>



<ol class="wp-block-list">
<li><strong>Reachability.</strong> Python check, fact gathering, timezone, hostname, package cache refresh.</li>

<li><strong>Identity.</strong> Admin user, group membership, authorized key, sudoers drop-in with <code>visudo -cf</code> validation.</li>

<li><strong>Verification gate.</strong> The task shown earlier. Nothing destructive runs before this passes.</li>

<li><strong>Firewall.</strong> Rules first, policy second, enable third.</li>

<li><strong>Unattended updates.</strong> <code>unattended-upgrades</code> on Debian and Ubuntu, <code>dnf-automatic</code> with its systemd timer on RHEL-family hosts.</li>

<li><strong>Brute-force protection.</strong> fail2ban with your own <code>/etc/fail2ban/jail.local</code>, never edits to <code>jail.conf</code>.</li>

<li><strong>Workload.</strong> Docker, a web server, a monitoring agent, whatever the box is for.</li>

<li><strong>Access teardown.</strong> Disable root SSH login, disable password authentication, restart sshd, reconnect, confirm.</li>
</ol>



<p class="wp-block-paragraph">When step eight changes the port or user, the play needs to actually reconnect rather than reusing its cached SSH connection. Ansible keeps a persistent connection per host, so a config change alone will not be picked up:</p>



<pre class="wp-block-code"><code>- name: Point subsequent tasks at the new identity
  ansible.builtin.set_fact:
    ansible_user: "{{ admin_user }}"
    ansible_port: "{{ ssh_port }}"

- name: Drop the cached connection
  ansible.builtin.meta: reset_connection

- name: Confirm the host answers on the new settings
  ansible.builtin.wait_for_connection:
    timeout: 60</code></pre>



<p class="wp-block-paragraph"><code>meta: reset_connection</code> does not accept a <code>when</code> conditional, so it runs on every pass. That is a small cost for a play that works on the first run and every run after.</p>



<h2 class="wp-block-heading">Test it before you point it at anything real</h2>



<p class="wp-block-paragraph">The cheapest test is a disposable VPS. Spin one up, run the playbook, reboot it, run the playbook again, then destroy it. The reboot is the important part, because that is what exposes the socket activation problem and any service you enabled but never actually started.</p>



<pre class="wp-block-code"><code># catch YAML and structural errors instantly
ansible-playbook site.yml --syntax-check

# style and correctness, including FQCN and idempotency smells
ansible-lint

# show what would change without changing it
ansible-playbook -i inventory.yml site.yml --check --diff</code></pre>



<p class="wp-block-paragraph">One caveat on <code>--check</code>: it is not fully reliable on a bootstrap playbook. Tasks that depend on earlier tasks having run will fail or report nonsense in check mode, because the earlier change did not really happen. Treat a clean check run as a weak signal, not proof.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong><code>UNREACHABLE</code> on the very first task.</strong> Usually host key verification against a rebuilt host reusing an old IP. Remove the stale entry with <code>ssh-keygen -R</code> and re-add the correct one with <code>ssh-keyscan</code>. Disabling host key checking globally makes this go away and removes a real protection, so do not.</li>

<li><strong><code>UNREACHABLE</code> immediately after a firewall task.</strong> You armed the firewall before allowing SSH. Recover through the provider console, not SSH.</li>

<li><strong>&#8220;Missing sudo password&#8221;.</strong> The sudoers drop-in is not in effect or has bad syntax. Validate with <code>visudo -cf</code> on the file itself before trusting it.</li>

<li><strong>Module not found under an FQCN.</strong> The collection is not installed on the machine running the playbook. Confirm with <code>ansible-galaxy collection list</code>.</li>

<li><strong>New group membership ignored.</strong> Adding a user to <code>docker</code> mid-play does not affect the already-open connection. Insert <code>meta: reset_connection</code> after the group change.</li>

<li><strong>fail2ban installed but not banning.</strong> On images without rsyslog, <code>/var/log/auth.log</code> may not exist, so a jail configured to read it never starts. Check <code>fail2ban-client status</code> and switch the jail to the systemd backend if the log file is absent.</li>

<li><strong>Everything green, port unchanged.</strong> Socket activation. Check <code>systemctl is-enabled ssh.socket</code> and <code>ss -tlnp</code>.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Disabling root login in the same play that creates the replacement user, with no verification step between them.</li>

<li>Enabling UFW before the SSH allow rule exists.</li>

<li>Assuming <code>Port</code> in <code>sshd_config</code> is authoritative on every distribution.</li>

<li>Unguarded <code>shell</code> tasks that notify handlers, causing service restarts on every run.</li>

<li>Editing <code>jail.conf</code> instead of creating <code>jail.local</code>, so the next package update silently reverts it.</li>

<li>Committing plaintext secrets because &#8220;it is only a bootstrap playbook&#8221;.</li>

<li>Testing on a host you cannot afford to lose, with no console access configured.</li>

<li>Relying on <code>--check</code> as a substitute for a real run on a disposable host.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Know your recovery path before the first run. Provider console, serial console or recovery mode, tested once so you are not learning it under pressure.</li>

<li>Put a verification task before every irreversible one, and let the play fail there rather than after.</li>

<li>Prefer drop-in files over line edits for sshd, sudoers, fail2ban and sysctl. They survive package upgrades and are trivially removable.</li>

<li>Validate configuration before restarting the service that reads it. <code>sshd -t</code>, <code>visudo -cf</code>, <code>nginx -t</code>.</li>

<li>Pin collections in <code>requirements.yml</code> and install them in CI, not just locally.</li>

<li>Run the playbook twice on a fresh host and read the second recap. Anything still reporting changed is a bug.</li>

<li>Reboot once during testing. Half of the silent failures only appear after a restart.</li>

<li>Get a monitoring agent on the box in the same run. A server that is hardened but unobserved is a server you will hear about from a customer first.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can you really bootstrap a VPS with Ansible in a single playbook run?</h3>



<p class="wp-block-paragraph">Yes, including changing the SSH user and port mid-run. The mechanism is <code>set_fact</code> to update the connection variables, <code>meta: reset_connection</code> to drop the cached connection, and <code>wait_for_connection</code> to confirm the new settings work before continuing. What you cannot do safely is change access and skip the confirmation.</p>



<h3 class="wp-block-heading">Should I use cloud-init or Ansible for initial server setup?</h3>



<p class="wp-block-paragraph">Both, split by lifetime. cloud-init handles things that are true exactly once at first boot: the admin user, the SSH key, a minimal package set. Ansible handles everything that will be tuned again later. Putting hardening rules in cloud-init means the only way to change them is to rebuild the host.</p>



<h3 class="wp-block-heading">Do I need Terraform or OpenTofu as well?</h3>



<p class="wp-block-paragraph">Only if you are creating the infrastructure, not just configuring it. Terraform and OpenTofu create the VPS, the DNS records and the provider firewall. Ansible configures what runs inside. For a handful of long-lived servers ordered through a provider panel, Ansible alone is enough and adding a second tool costs more than it returns.</p>



<h3 class="wp-block-heading">Why does my playbook work the first time and fail the second?</h3>



<p class="wp-block-paragraph">Almost always because the first run changed the connection parameters and the inventory still describes the original ones. Either update the inventory to the post-bootstrap values and make the play tolerate both, or split bootstrap and steady-state into separate playbooks with separate inventory entries.</p>



<h3 class="wp-block-heading">Is changing the SSH port worth doing?</h3>



<p class="wp-block-paragraph">It reduces log noise from untargeted scanning, which makes real events easier to spot. It is not a security control on its own. Key-only authentication, a default-deny firewall and fail2ban do the actual work. If moving the port complicates your automation, the honest answer is to leave it on 22.</p>



<h3 class="wp-block-heading">How do I recover if the playbook locks me out?</h3>



<p class="wp-block-paragraph">Through the provider&#8217;s console or recovery environment, which is why you should know where it is before you need it. Most VPS providers offer a web-based console, some offer a rescue boot with the disk mounted. From there, revert the offending drop-in file or firewall rule, restart the service, and reconnect. Then add the verification step the playbook was missing.</p>



<h3 class="wp-block-heading">Can this run against several servers at once?</h3>



<p class="wp-block-paragraph">Yes, but use <code>serial: 1</code> the first few times. If there is an ordering bug in the play, running it against one host at a time means you lock yourself out of one server instead of a fleet.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">When you bootstrap a VPS with Ansible, the individual tasks are the easy part. Creating a user, writing a firewall rule and hardening sshd are all well-documented and hard to get wrong in isolation. What decides whether the run leaves you with a working server or a rebuild is the sequence, and specifically whether you verify new access before removing old access.</p>



<p class="wp-block-paragraph">Add the verification gate. Run the playbook twice. Reboot the box once. Those three habits catch nearly every silent failure described here, and they cost about ten minutes on a disposable host.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a bootstrap playbook you can actually rerun?</h2>



<p class="wp-block-paragraph">Most of the provisioning work I take on starts with a playbook that worked once and has been too frightening to run since. Things I help with:</p>



<ul class="wp-block-list">
<li>Building a bootstrap role that takes a blank VPS to a hardened, monitored server in one run, with a verification gate before every irreversible step.</li>

<li>Auditing an existing playbook for idempotency, so the second and hundredth runs are as safe as the first.</li>

<li>Untangling SSH, firewall and socket activation interactions on mixed Debian, Ubuntu and RHEL-family estates.</li>

<li>Splitting cloud-init and Ansible responsibilities cleanly so hardening stays changeable after first boot.</li>

<li>Moving secrets out of a repository into Vault or a managed secret store without breaking existing runs.</li>

<li>Wiring the playbook into CI with linting, collection pinning and a disposable-host test that runs on every change.</li>
</ul>



<p class="wp-block-paragraph">Send me the playbook, a recap from a second run, or the output of <code>ss -tlnp</code> from the host that is behaving oddly, and I will tell you what I would change first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/bootstrap-vps-with-ansible/">Bootstrap a VPS With Ansible in One Run Without Locking Yourself Out</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Connect Amazon Redshift to Zoho Analytics Without Putting Your Warehouse on the Internet</title>
		<link>https://john-nessime.com/blog/web-security/connect-redshift-zoho-analytics/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 02 Aug 2026 07:59:30 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Amazon Redshift]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Business Intelligence]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Warehouse]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Firewall]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[SQL]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[VPC]]></category>
		<category><![CDATA[Zoho Analytics]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=80</guid>

					<description><![CDATA[<p>Every guide tells you to tick "Publicly Accessible" and move on. That works, and it also puts your warehouse on the internet with whatever credentials were to hand. Here's the full setup done deliberately: least-privilege user, narrow network access, and the outbound-only option nobody mentions.</p>
<p>The post <a href="https://john-nessime.com/blog/web-security/connect-redshift-zoho-analytics/">Connect Amazon Redshift to Zoho Analytics Without Putting Your Warehouse on the Internet</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">You fill in the connection form, click test, and it sits there until it times out. You search the error. Every result says the same thing: go into the Redshift console and tick <strong>Publicly Accessible</strong>. You do it. The connection succeeds. Dashboards start working and everyone moves on.</p>



<p class="wp-block-paragraph">What just happened is that your data warehouse now has an endpoint on the public internet, reachable on port 5439, and a third-party SaaS platform holds credentials to it. Usually those credentials belong to whatever user was to hand, which on a lot of clusters means the admin account that can read every schema you have.</p>



<p class="wp-block-paragraph">That is not an argument against doing it. It is an argument for doing it on purpose. Most guides on how to <strong>connect Amazon Redshift to Zoho Analytics</strong> walk you through six screenshots and never mention the two decisions that actually matter: which user this connection runs as, and how much of your network it needs.</p>



<p class="wp-block-paragraph">This covers the whole setup in order, with those decisions made deliberately: the mode you pick and why it is hard to change later, a read-only user scoped to almost nothing, network access done narrowly, and what to actually expose once it works.</p>



<h2 class="wp-block-heading">What the connection actually requires</h2>



<p class="wp-block-paragraph">Strip away the interface and the mechanics are simple. Zoho Analytics runs on Zoho&#8217;s infrastructure. To reach your cluster it opens an inbound TCP connection from Zoho&#8217;s own IP addresses to your Redshift endpoint, on the Redshift port. That is it.</p>



<p class="wp-block-paragraph">Which means three things have to be true, and they are the three places setup fails:</p>



<ul class="wp-block-list">
<li><strong>The endpoint has to be reachable from outside your VPC.</strong> A cluster in a private subnet with no public endpoint cannot be reached by a SaaS tool, full stop.</li>
<li><strong>The security group has to allow it.</strong> Zoho publishes the IP addresses it connects from, and those need an inbound rule on the Redshift port. The default port is 5439, though it can be changed at cluster creation, so check yours rather than assuming.</li>
<li><strong>The credentials have to work</strong> and the user has to be able to see the schemas you want to report on.</li>
</ul>



<p class="wp-block-paragraph">Zoho keeps the current allowlist on a dedicated help page. Do not copy a list of addresses out of a blog post, including this one. They change, and a stale allowlist produces an intermittent failure that looks like everything except what it is.</p>



<h2 class="wp-block-heading">Decide the mode before you start</h2>



<p class="wp-block-paragraph">Zoho offers two fundamentally different connection types, and switching later means rebuilding the workspace. Get this one right the first time.</p>



<p class="wp-block-paragraph"><strong>Data Import</strong> copies your data into Zoho Analytics on a schedule and reports run against their storage. Dashboards are fast, you can create query tables, and you can import a filtered subset using a custom query. Column additions and deletions sync automatically. The trade is freshness: your numbers are as current as the last sync.</p>



<p class="wp-block-paragraph"><strong>Live Connect</strong> stores nothing and queries Redshift each time a report loads. Numbers are always current and your data does not leave AWS. It is available on the paid tiers only. The trade is bigger than it looks: report speed becomes a Redshift performance problem, schema changes need a manual sync each time, you cannot change column types on the Zoho side at all, and no query tables.</p>



<p class="wp-block-paragraph">My default for most teams is Data Import, because the operational surface is smaller and a scheduled sync is usually fresh enough for a dashboard people look at once a day. Live Connect earns its cost when the numbers genuinely need to be current, or when duplicating the data into a third-party system is a compliance problem.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Step 1: create a user that cannot do much</h2>



<p class="wp-block-paragraph">Do this before you open any network access, so that when the connection works you already know its blast radius.</p>



<pre class="wp-block-code"><code>-- A dedicated user for this connection and nothing else.
CREATE USER zoho_reader PASSWORD 'generate-a-long-random-one';

-- Reach the schema, then read the objects in it.
-- Both grants are needed; USAGE alone gets you nothing.
GRANT USAGE ON SCHEMA analytics TO zoho_reader;
GRANT SELECT ON ALL TABLES IN SCHEMA analytics TO zoho_reader;

-- The grant above covers what exists today. This covers what
-- gets created tomorrow, which is the part people forget and
-- then debug as a mysterious missing table three months later.
ALTER DEFAULT PRIVILEGES IN SCHEMA analytics
  GRANT SELECT ON TABLES TO zoho_reader;</code></pre>



<p class="wp-block-paragraph">Then check what you actually granted, rather than trusting that the statements did what you meant:</p>



<pre class="wp-block-code"><code>SELECT table_schema, table_name, privilege_type
FROM information_schema.table_privileges
WHERE grantee = 'zoho_reader'
ORDER BY table_schema, table_name;</code></pre>



<p class="wp-block-paragraph">If that returns rows from schemas you did not intend to publish, fix it now. Once the connection is live, someone will build a report on whatever is visible and removing access becomes a conversation instead of a command.</p>



<p class="wp-block-paragraph">The password goes into a SaaS platform, so treat it accordingly: long, random, generated, stored in your password manager, and rotated when people leave.</p>



<h2 class="wp-block-heading">Step 2: open exactly as much network as you need</h2>



<p class="wp-block-paragraph">First find out what your cluster currently looks like, because the console tells you less at a glance than this does:</p>



<pre class="wp-block-code"><code>aws redshift describe-clusters 
  --cluster-identifier my-warehouse 
  --query 'Clusters[0].{Endpoint:Endpoint.Address,Port:Endpoint.Port,Public:PubliclyAccessible,SecurityGroups:VpcSecurityGroups}'</code></pre>



<p class="wp-block-paragraph">That gives you the hostname Zoho needs, the real port, whether the cluster is currently public, and which security groups govern it. Redshift Serverless has an equivalent setting on the workgroup rather than a cluster, but the same reasoning applies.</p>



<p class="wp-block-paragraph">Then add one narrow inbound rule per Zoho range. Not <code>0.0.0.0/0</code>, which is what happens when someone is in a hurry and it never gets tightened afterwards:</p>



<pre class="wp-block-code"><code>aws ec2 authorize-security-group-ingress 
  --group-id sg-0123456789abcdef0 
  --protocol tcp 
  --port 5439 
  --cidr 203.0.113.0/24</code></pre>



<p class="wp-block-paragraph">Put a description on each rule in the console so that in a year somebody can tell which ranges belong to which vendor. An undocumented allow rule is one nobody will ever dare remove.</p>



<p class="wp-block-paragraph">While you are in the cluster parameter group, turn on <code>require_ssl</code> so connections that do not use TLS are refused rather than merely discouraged. Parameter group changes need a cluster reboot, so do it in a window rather than mid-afternoon.</p>



<h3 class="wp-block-heading">Test from outside before you touch Zoho</h3>



<pre class="wp-block-code"><code># Does anything answer on the port, from a machine outside your VPC?
nc -zv my-warehouse.abc123.eu-west-1.redshift.amazonaws.com 5439

# Then prove the credentials work, over TLS, as the new user.
psql "host=my-warehouse.abc123.eu-west-1.redshift.amazonaws.com 
      port=5439 dbname=analytics user=zoho_reader sslmode=require"</code></pre>



<p class="wp-block-paragraph">If both of those work from your laptop, the remaining variable is Zoho&#8217;s IP ranges rather than anything structural. If the first one hangs, it is the security group or public accessibility. If it connects but authentication fails, it is the user. Separating those two takes thirty seconds and saves an hour of guessing inside a web form that reports one generic error for both.</p>



<h3 class="wp-block-heading">If you cannot make the cluster public</h3>



<p class="wp-block-paragraph">Plenty of organisations have a policy against it, and that policy is not unreasonable. Zoho&#8217;s answer for databases behind a firewall is Zoho Databridge, a lightweight agent you install on a machine inside your own network. It opens an <em>outbound</em> connection on 443 to Zoho and waits for requests, so nothing inbound has to be allowed at all. It runs on Windows, macOS and Linux, and it is designed for exactly this situation.</p>



<p class="wp-block-paragraph">Worth checking before you commit to it: Databridge is documented against a long list of relational databases, and Redshift is not always named explicitly in that list. Since Redshift speaks the PostgreSQL wire protocol and has a JDBC driver, it is usually workable, but confirm with Zoho support for your plan rather than taking my word for it. If it fits, an outbound-only agent is a meaningfully better architecture than a public endpoint, and almost nobody setting this up knows the option exists.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Step 3: make the connection</h2>



<p class="wp-block-paragraph">With the groundwork done, this part is genuinely a form. In Zoho Analytics, create a workspace, choose to import from or connect to Amazon Redshift, and supply the endpoint hostname, the port, the database name, and the <code>zoho_reader</code> credentials.</p>



<p class="wp-block-paragraph">Two expectations worth setting so you do not diagnose a non-problem:</p>



<ul class="wp-block-list">
<li><strong>The first fetch takes a while</strong>, depending on volume and how fast your cluster responds. Zoho emails you when it finishes.</li>
<li><strong>An empty workspace before that fetch completes is normal.</strong> If you open it early it will show nothing, which looks exactly like a failure and is not one.</li>
</ul>



<h2 class="wp-block-heading">Step 4: choose what to expose, carefully</h2>



<p class="wp-block-paragraph">This is the screen people click through fastest and the one with the longest consequences. Whatever tables you select here become the interface between your warehouse and your dashboards, and every future upstream change becomes a potential BI incident.</p>



<p class="wp-block-paragraph">Select views, not base tables. A reporting view with explicit casts, timezone conversion done in Redshift, and semi-structured columns already flattened gives you a stable contract. The tables underneath stay free to change.</p>



<pre class="wp-block-code"><code>CREATE OR REPLACE VIEW analytics.v_orders_bi AS
SELECT
    CAST(order_id     AS BIGINT)        AS order_id,
    CAST(order_status AS VARCHAR(64))   AS order_status,
    CAST(total_amount AS DECIMAL(18,2)) AS total_amount,
    created_at                          AS created_at_utc,
    CAST(CONVERT_TIMEZONE('UTC','Europe/London', created_at) AS DATE)
                                        AS order_date_local
FROM analytics.orders
WITH NO SCHEMA BINDING;</code></pre>



<p class="wp-block-paragraph"><code>WITH NO SCHEMA BINDING</code> makes it a late-binding view, so a nightly job can drop and rebuild the underlying table without Redshift refusing or the view disappearing. On a warehouse with a rebuild-style load, that alone prevents a recurring class of broken-dashboard morning.</p>



<p class="wp-block-paragraph">Grant your reader access to the views specifically, and skip the blanket grant on the whole schema if you can. Fewer objects visible means fewer things somebody can accidentally build a report on.</p>



<h2 class="wp-block-heading">Step 5: schedule it, then verify a real number</h2>



<p class="wp-block-paragraph">In Data Import mode, set the sync schedule to match how the data is actually produced. Syncing hourly when the warehouse loads once at 03:00 just means twenty-three pointless queries a day against your cluster, and Redshift bills by uptime and workload.</p>



<p class="wp-block-paragraph">Then do the step everyone skips. Pick a closed period, run the same aggregate in both places, and compare:</p>



<pre class="wp-block-code"><code>SELECT DATE_TRUNC('day', created_at) AS day,
       COUNT(*)                      AS orders,
       SUM(total_amount)             AS revenue
FROM analytics.v_orders_bi
WHERE created_at_utc &gt;= DATEADD(month, -1, DATE_TRUNC('month', GETDATE()))
  AND created_at_utc &lt;  DATE_TRUNC('month', GETDATE())
GROUP BY 1
ORDER BY 1;</code></pre>



<p class="wp-block-paragraph">A connection that returns data is not the same as a connection that returns correct data. If daily figures agree but monthly ones do not, you have a timezone problem rather than a connection problem, and it is far easier to find now than six weeks into someone&#8217;s board pack.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Connection times out</h3>



<p class="wp-block-paragraph">Network, not credentials. A timeout means nothing answered. Check public accessibility, then the security group rule, then whether Zoho&#8217;s allowlist has changed since you configured it. A refused connection rather than a timeout usually means you reached the right host on the wrong port.</p>



<h3 class="wp-block-heading">Authentication failed</h3>



<p class="wp-block-paragraph">The network is fine, which is genuine progress. Test the same credentials with <code>psql</code> from your own machine. Also check the database name: Redshift clusters often have both a default database and the one you actually use, and connecting to the wrong one authenticates fine and then shows you nothing.</p>



<h3 class="wp-block-heading">Connected, but no tables listed</h3>



<p class="wp-block-paragraph">A permissions gap. <code>USAGE</code> on the schema and <code>SELECT</code> on the objects are separate grants and you need both. Run the <code>information_schema.table_privileges</code> query as a check rather than re-running the grants and hoping.</p>



<h3 class="wp-block-heading">Worked on setup, fails intermittently later</h3>



<p class="wp-block-paragraph">Classic symptom of a partially stale IP allowlist: some of Zoho&#8217;s ranges reach you and some do not, so syncs succeed or fail depending on which host tries. Re-check the published list against your security group rules.</p>



<h3 class="wp-block-heading">Sync succeeds but a new column never appears</h3>



<p class="wp-block-paragraph">In Live Connect, schema changes need a manual Sync Design, and an unresolved mismatch anywhere in the connection stops it fetching new column information at all. Clear the Mismatch tab completely, then sync again.</p>



<h3 class="wp-block-heading">Reports are slow</h3>



<p class="wp-block-paragraph">In Live Connect that is your cluster, not Zoho. Look at the queries hitting Redshift and at sort and distribution keys on the underlying tables. Zoho also offers per-workspace caching for Live Connect with a configurable refresh interval, which trades freshness for speed.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Connecting with the cluster admin user because it was already in the password manager.</li>
<li>Opening the Redshift port to <code>0.0.0.0/0</code> to make the test pass, then never tightening it.</li>
<li>Copying an IP allowlist out of a blog post instead of Zoho&#8217;s own page.</li>
<li>Granting <code>USAGE</code> without <code>SELECT</code>, or the reverse, and concluding the connector is broken.</li>
<li>Forgetting <code>ALTER DEFAULT PRIVILEGES</code>, so tables created later are invisible.</li>
<li>Pointing the connection at base tables rather than reporting views.</li>
<li>Picking Live Connect for the freshness without accounting for the manual sync on every schema change.</li>
<li>Assuming the setup failed because the workspace is empty during the initial fetch.</li>
<li>Scheduling syncs far more often than the warehouse is actually loaded.</li>
<li>Not enforcing TLS on the cluster.</li>
<li>Declaring it done without reconciling a single number against Redshift.</li>
<li>Leaving no note anywhere about which security group rules belong to which vendor.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>A dedicated read-only user per integration, never a shared or admin account.</li>
<li>Grant access to reporting views only, not whole schemas, where you can.</li>
<li>One narrow security group rule per vendor range, described and dated.</li>
<li>Enforce TLS with <code>require_ssl</code> rather than trusting the client to ask for it.</li>
<li>Prefer an outbound-only agent over a public endpoint if your plan and setup support it.</li>
<li>Decide Data Import versus Live Connect deliberately, because changing it means rebuilding.</li>
<li>Expose late-binding views with explicit casts and timezone conversion done in Redshift.</li>
<li>Match the sync schedule to your load schedule, not to how fresh you wish the data were.</li>
<li>Reconcile at least one aggregate before anyone builds a dashboard on it.</li>
<li>Rotate the connection password when people leave, and document where it lives.</li>
<li>Test connectivity and credentials separately, from outside the VPC, before blaming the connector.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Does Redshift have to be publicly accessible?</h3>



<p class="wp-block-paragraph">For a direct connection, yes: Zoho reaches your cluster inbound from its own IP addresses, so there has to be an endpoint it can resolve and reach. The alternative is Zoho Databridge, an agent inside your network that connects outbound on 443, which removes the inbound requirement entirely. Confirm Redshift support for it with Zoho before planning around it.</p>



<h3 class="wp-block-heading">Which port does Zoho need open?</h3>



<p class="wp-block-paragraph">Whichever port your cluster listens on. 5439 is the Redshift default, but it can be set to something else when the cluster is created, so read it from <code>describe-clusters</code> rather than assuming.</p>



<h3 class="wp-block-heading">Should I use Data Import or Live Connect?</h3>



<p class="wp-block-paragraph">Data Import for most cases: faster dashboards, query tables, automatic column syncing, and a smaller operational surface. Live Connect when the numbers must be current or copying data into a third party is a compliance issue, accepting slower reports and manual schema syncs. Decide before you build, because switching means starting the workspace over.</p>



<h3 class="wp-block-heading">What permissions does the Zoho user need?</h3>



<p class="wp-block-paragraph"><code>USAGE</code> on the schema and <code>SELECT</code> on the objects you want reported on. Nothing else. Add <code>ALTER DEFAULT PRIVILEGES</code> so objects created later are covered without anybody having to remember.</p>



<h3 class="wp-block-heading">Why is the workspace empty after I finish setup?</h3>



<p class="wp-block-paragraph">The initial fetch has not finished. Zoho emails you when it completes, and the workspace shows nothing until then. Give it time before you start pulling the configuration apart.</p>



<h3 class="wp-block-heading">Can I limit which tables Zoho can see?</h3>



<p class="wp-block-paragraph">Yes, and you should. Grant the reader access only to a set of reporting views in a dedicated schema. That is a stronger control than the table picker in the interface, because it holds even if somebody later edits the connection.</p>



<h3 class="wp-block-heading">Will this increase my Redshift bill?</h3>



<p class="wp-block-paragraph">Live Connect can, since every report load is a query. Data Import costs you one scheduled query run per sync. If cost matters, Data Import on a schedule matched to your load window is the cheaper shape by a wide margin.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">Getting Zoho Analytics to talk to Redshift takes about twenty minutes. Getting it to talk to Redshift without handing a SaaS platform a superuser account and an open port takes about an hour, and that hour is the entire difference between a connection you can defend in an audit and one you quietly hope nobody asks about.</p>



<p class="wp-block-paragraph">So do it in this order: least-privilege user first, narrow network access second, connection third, views rather than tables fourth, and a reconciled number before anyone builds a dashboard on it. Every step after the first is easier when the first one is already done.</p>



<h2 class="wp-block-heading">Want this set up properly the first time?</h2>



<p class="wp-block-paragraph">Most of these connections get built under time pressure and inherit whatever shortcuts made the test pass. Work I take on:</p>



<ul class="wp-block-list">
<li>Setting up a Redshift to Zoho Analytics connection end to end, with least-privilege credentials and scoped network access.</li>
<li>Reviewing an existing connection and reporting what it can actually reach, then narrowing it without breaking reports.</li>
<li>Building the reporting view layer in Redshift: explicit casts, timezone handling, flattened semi-structured columns, late binding.</li>
<li>Network architecture for BI access, including outbound-agent options where a public endpoint is not acceptable.</li>
<li>Choosing between Data Import and Live Connect based on your actual freshness, cost and compliance constraints.</li>
<li>Reconciliation checks so dashboard numbers are verified against the warehouse rather than assumed.</li>
</ul>



<p class="wp-block-paragraph">Tell me whether your cluster is currently public and which user the connection runs as, and I will tell you what I would change first.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/web-security/connect-redshift-zoho-analytics/">Connect Amazon Redshift to Zoho Analytics Without Putting Your Warehouse on the Internet</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>You Probably Didn&#8217;t Harden That Server: A VPS Setup and Hardening Checklist That Verifies Itself</title>
		<link>https://john-nessime.com/blog/system-administration/vps-hardening-checklist/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 01 Aug 2026 09:24:24 +0000</pubDate>
				<category><![CDATA[Linux]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Bash]]></category>
		<category><![CDATA[Fail2ban]]></category>
		<category><![CDATA[Firewall]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[OpenSSH]]></category>
		<category><![CDATA[Production]]></category>
		<category><![CDATA[Self Hosting]]></category>
		<category><![CDATA[Server Hardening]]></category>
		<category><![CDATA[SSH]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Systemd]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[UFW]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=62</guid>

					<description><![CDATA[<p>You set PasswordAuthentication no, restarted SSH, and moved on. Months later the logs show successful password logins, because a vendor drop-in file sorted earlier and won. A VPS setup and hardening checklist where every step comes with the command that proves it worked.</p>
<p>The post <a href="https://john-nessime.com/blog/system-administration/vps-hardening-checklist/">You Probably Didn&#8217;t Harden That Server: A VPS Setup and Hardening Checklist That Verifies Itself</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">You spun up a new box, worked through a hardening guide, set <code>PasswordAuthentication no</code>, restarted SSH, and moved on. Six months later you are reading auth logs for an unrelated reason and there it is: successful password logins. Not attempts. Logins.</p>



<p class="wp-block-paragraph">The line you wrote was real. The file was saved. What you did not know is that your distro puts an <code>Include /etc/ssh/sshd_config.d/*.conf</code> near the top of <code>sshd_config</code>, that sshd uses the <em>first</em> value it finds for any given keyword, and that your cloud provider&#8217;s image dropped a <code>50-cloud-init.conf</code> in that directory setting <code>PasswordAuthentication yes</code>. It sorts earlier. It wins. Your line at the bottom of the main file was never reached.</p>



<p class="wp-block-paragraph">You never noticed because you log in with a key anyway. Nothing broke. The control just was not there.</p>



<p class="wp-block-paragraph">That is the thing most guides get wrong, and it is the idea this <strong>VPS hardening checklist</strong> is built around: a hardening step is not done when you have edited the file. It is done when you have checked the effective configuration and seen the value you expected. Everything below comes with the command that proves it.</p>



<p class="wp-block-paragraph">This covers the order to do things in so you do not lock yourself out, SSH keys and config, the firewall, automatic updates, fail2ban and whether it is worth it, the unglamorous things that matter more than any of it, and what I would skip.</p>



<h2 class="wp-block-heading">Before you touch anything</h2>



<p class="wp-block-paragraph">Two minutes of preparation that saves an evening.</p>



<p class="wp-block-paragraph"><strong>Find your provider&#8217;s console before you need it.</strong> Whether you are on InterServer, Hetzner, DigitalOcean or anything else, there is a web-based serial or VNC console that works when SSH does not. Open it now, log in as root, and confirm it works. This is your parachute. If you are going to lock yourself out, you want to already know where it is.</p>



<p class="wp-block-paragraph"><strong>Never close the session you are working in.</strong> Every SSH change gets tested from a second terminal while the first one stays open. If the new settings are broken, the old session is still authenticated and you can undo them. Close it too early and your only route back in is the console.</p>



<p class="wp-block-paragraph">Then update the system, because everything else assumes current packages:</p>



<pre class="wp-block-code"><code>sudo apt update &amp;&amp; sudo apt upgrade -y
sudo reboot   # if a kernel update landed</code></pre>



<h2 class="wp-block-heading">1. A real user, and keys instead of passwords</h2>



<p class="wp-block-paragraph">Generate the key on your own machine, not on the server. A private key that has been on a server is not a private key any more.</p>



<pre class="wp-block-code"><code># On your laptop. ed25519 is the sane default: short keys,
# fast verification, no parameter choices to get wrong.
ssh-keygen -t ed25519 -C "deploy@my-laptop"</code></pre>



<pre class="wp-block-code"><code># On the server, as root.
adduser deploy
usermod -aG sudo deploy        # 'wheel' instead of 'sudo' on RHEL-family

# Back on your laptop. ssh-copy-id appends to authorized_keys
# and sets the permissions correctly, which is the part people
# get wrong when they paste the key by hand.
ssh-copy-id deploy@203.0.113.10</code></pre>



<p class="wp-block-paragraph">Now open a second terminal and confirm <code>ssh deploy@203.0.113.10</code> works, and that <code>sudo -v</code> works once you are in. Only then move on. If either fails, fix it before you disable anything, because in a minute you will be removing the fallback.</p>



<h2 class="wp-block-heading">2. SSH config, and why your edit might do nothing</h2>



<p class="wp-block-paragraph">This is the section that actually matters. Two mechanisms silently override what you write.</p>



<h3 class="wp-block-heading">Drop-in files and first-match-wins</h3>



<p class="wp-block-paragraph">Modern OpenSSH packages ship an <code>Include</code> line near the top of <code>/etc/ssh/sshd_config</code>, which pulls in every <code>.conf</code> file from <code>/etc/ssh/sshd_config.d/</code> in alphabetical order, before the rest of the main file is read. Combine that with sshd&#8217;s rule that the first value found for a keyword wins, and the consequence is specific: <strong>your drop-in has to sort earlier than the vendor&#8217;s, not later.</strong></p>



<p class="wp-block-paragraph">Most guides tell you to name it <code>99-something.conf</code>, which is the convention almost everywhere else in Linux and exactly backwards here.</p>



<pre class="wp-block-code"><code># /etc/ssh/sshd_config.d/00-hardening.conf
# Sorts before 50-cloud-init.conf, so these values are seen first.

PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes

# Optional: restrict to the accounts that should have shell access.
AllowUsers deploy</code></pre>



<p class="wp-block-paragraph">A note on <code>KbdInteractiveAuthentication</code>: older guides and older OpenSSH call this <code>ChallengeResponseAuthentication</code>. Setting only <code>PasswordAuthentication no</code> can still leave a keyboard-interactive path open on some configurations, which is why both are here.</p>



<p class="wp-block-paragraph">Now the part nobody does. Validate the syntax, then read the effective config:</p>



<pre class="wp-block-code"><code># Syntax check. Prints nothing if the config is valid.
# Run this BEFORE restarting, always.
sudo sshd -t

# Dump the effective configuration with every include resolved.
# This is the only output that tells you the truth.
sudo sshd -T | grep -Ei 'permitrootlogin|passwordauthentication|kbdinteractive|pubkeyauth'</code></pre>



<p class="wp-block-paragraph">If <code>passwordauthentication yes</code> comes back, your file lost to something else in that directory. Look at what is in there with <code>ls /etc/ssh/sshd_config.d/</code> and either rename yours to sort earlier or edit the offending file directly.</p>



<h3 class="wp-block-heading">Socket activation, and the port that will not change</h3>



<p class="wp-block-paragraph">The second trap. On several current distro releases, SSH is started by systemd socket activation rather than as a persistent service: <code>ssh.socket</code> holds the listening port and starts <code>sshd</code> only when a connection arrives. On those systems the <code>Port</code> and <code>ListenAddress</code> directives in <code>sshd_config</code> may be ignored entirely, because systemd owns the socket.</p>



<p class="wp-block-paragraph">People change the port, restart the service, see no error, and assume it worked. Check what is actually bound:</p>



<pre class="wp-block-code"><code># -t TCP, -l listening, -n numeric, -p show the owning process
sudo ss -tlnp</code></pre>



<p class="wp-block-paragraph">If you genuinely want a different port on a socket-activated system, override the socket unit with a proper drop-in rather than editing files under <code>/lib/systemd/system/</code>, which a package update will overwrite:</p>



<pre class="wp-block-code"><code>sudo systemctl edit ssh.socket</code></pre>



<pre class="wp-block-code"><code># In the editor that opens:
[Socket]
# The empty assignment clears the inherited value. Without it
# you end up listening on both the old port and the new one.
ListenStream=
ListenStream=2222</code></pre>



<pre class="wp-block-code"><code>sudo systemctl daemon-reload
sudo systemctl restart ssh.socket
sudo ss -tlnp        # confirm before you trust it</code></pre>



<p class="wp-block-paragraph">Honest assessment of port changes: this is not security. Anyone scanning your address finds an SSH daemon on a non-standard port in seconds. What it genuinely does is cut the volume of automated login noise in your logs by a large margin, which makes real events easier to see. That is a maintenance benefit, not a defence, and it is worth doing for that reason alone. Just do not count it as one of your controls.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">3. Firewall, in the right order</h2>



<p class="wp-block-paragraph">The ordering here is the whole trick. Set the default policy and add your rules <em>before</em> you enable the firewall. Enable first and you will drop your own connection with the policy that denies everything.</p>



<pre class="wp-block-code"><code>sudo ufw default deny incoming
sudo ufw default allow outgoing

# Allow SSH BEFORE enabling. If you changed the port,
# allow that port instead of the OpenSSH profile.
sudo ufw allow OpenSSH
# sudo ufw allow 2222/tcp

sudo ufw allow 80,443/tcp

sudo ufw enable
sudo ufw status verbose</code></pre>



<p class="wp-block-paragraph">Read that <code>status verbose</code> output rather than assuming. It shows the default policies and every rule, and it is the proof that this step is done.</p>



<p class="wp-block-paragraph">One thing worth knowing if you run containers: Docker writes its own rules directly into netfilter and bypasses ufw. A container published with <code>-p 5432:5432</code> is reachable from the internet even though ufw says that port is denied. Bind container ports to <code>127.0.0.1</code> explicitly, or put the service behind a reverse proxy, and verify from outside the box with a port scan rather than trusting the firewall status.</p>



<p class="wp-block-paragraph">A stronger option worth considering: do not expose SSH to the internet at all. Put the server on a private overlay network with something like Tailscale or a WireGuard tunnel, bind sshd to that interface, and close port 22 to the world entirely. More setup, and it removes an entire class of problem rather than filtering it.</p>



<h2 class="wp-block-heading">4. Automatic security updates</h2>



<p class="wp-block-paragraph">Unpatched packages are a far more realistic route into your server than an SSH brute force. This is the highest-value item on the list and it takes two commands.</p>



<pre class="wp-block-code"><code>sudo apt install unattended-upgrades
sudo dpkg-reconfigure --priority=low unattended-upgrades

# Verify. Note the binary is singular, the package is plural.
# --dry-run makes no changes.
sudo unattended-upgrade --dry-run --debug</code></pre>



<p class="wp-block-paragraph">By default this applies security updates only, which is the right scope. Enabling automatic reboots is a real trade-off: it closes kernel vulnerabilities without you doing anything, and it will restart your server at whatever hour you configure. On a single web server with a proper service startup order, I turn it on. On anything holding state that does not recover cleanly from a hard restart, I leave it off and patch by hand.</p>



<h2 class="wp-block-heading">5. fail2ban, and what it is actually for</h2>



<p class="wp-block-paragraph">Install it, but be clear about the role. Once password authentication is off and root login is disabled, a brute force against sshd cannot succeed. There is no password to guess. fail2ban is not what is keeping those attackers out; your key-only config is.</p>



<p class="wp-block-paragraph">What it does earn its place for is reducing log noise and CPU spent on connection handling, and protecting the services where credentials <em>do</em> exist: a mail server, a web application login form, an FTP daemon.</p>



<pre class="wp-block-code"><code>sudo apt install fail2ban

# Do not edit jail.conf; it gets replaced on upgrade.
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

sudo systemctl enable --now fail2ban
sudo systemctl status fail2ban
sudo fail2ban-client status sshd</code></pre>



<p class="wp-block-paragraph">Check that <code>status</code> output properly. fail2ban has shipped in states where the service fails to start on a given distro release because the packaged version does not match the system Python, and it fails loudly in the journal and silently everywhere else. A stopped fail2ban looks exactly like a working one from the outside. If it is not running, <code>journalctl -u fail2ban</code> will tell you why.</p>



<p class="wp-block-paragraph">Set <code>ignoreip</code> in <code>jail.local</code> to include your office or home address. Banning yourself is a rite of passage but it is avoidable.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">6. The boring items that matter more than the exciting ones</h2>



<p class="wp-block-paragraph">Nobody writes blog posts about these, and they cause more real outages than any of the above.</p>



<h3 class="wp-block-heading">Provider account security</h3>



<p class="wp-block-paragraph">Turn on two-factor authentication in your hosting control panel and your domain registrar. Someone with your provider login does not need to defeat sshd; they can reset the root password from the console or attach your disk to a machine they control. This is the weakest link on most setups and it takes two minutes.</p>



<h3 class="wp-block-heading">Time and timezone</h3>



<pre class="wp-block-code"><code>sudo timedatectl set-timezone UTC
timedatectl status       # look for "System clock synchronized: yes"</code></pre>



<p class="wp-block-paragraph">UTC on servers, always. Correlating logs across machines in different local times, one of which observes daylight saving, is a genuinely miserable way to spend an incident.</p>



<h3 class="wp-block-heading">Swap on small instances</h3>



<pre class="wp-block-code"><code># A little swap stops the OOM killer choosing your database
# during a traffic spike. It is not a substitute for RAM.
sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
free -h</code></pre>



<h3 class="wp-block-heading">Backups that leave the machine</h3>



<p class="wp-block-paragraph">Your provider&#8217;s snapshots live in the same account as the server. If that account is compromised, or you delete the wrong thing, they go with it. A snapshot is a rollback mechanism, not a backup.</p>



<p class="wp-block-paragraph">Get database dumps and application data onto storage under a different set of credentials, on a schedule, and then do the part everyone skips: restore one. A backup you have never restored is a hypothesis. Point a dead man&#8217;s switch service at the backup job so you find out when it stops running, rather than the day you need it.</p>



<h3 class="wp-block-heading">Knowing the box is alive</h3>



<p class="wp-block-paragraph">At minimum, an external uptime check that alerts you. Better, node metrics into Prometheus with Grafana dashboards, or a hosted equivalent. Disk filling up is the single most common way a small VPS falls over, and it is completely predictable if anything is watching the graph.</p>



<h2 class="wp-block-heading">What I would skip</h2>



<p class="wp-block-paragraph">Not everything in the long hardening guides earns its cost on a single web server.</p>



<ul class="wp-block-list">
<li><strong>Large sysctl hardening blocks pasted from a gist.</strong> Most of the values are already the default on a current kernel, several are cargo-culted from a decade ago, and one of them will break something in a way that takes hours to trace. Change kernel parameters when you understand what each one does.</li>
<li><strong>AIDE, auditd and file-integrity monitoring</strong> on a small server nobody is watching. These produce a large volume of output that needs a human to review. If nobody reads it, it is not a control, it is just disk usage.</li>
<li><strong>Rebuilding OpenSSH&#8217;s cipher list.</strong> Current defaults are good. Guides that hand you a <code>Ciphers</code> line are usually copying advice written for a version that shipped years ago, and you can pin yourself to something weaker than the default without realising.</li>
<li><strong>Disabling ping.</strong> It breaks your own monitoring and hides nothing.</li>
</ul>



<h2 class="wp-block-heading">Troubleshooting</h2>



<h3 class="wp-block-heading">Locked out after an SSH change</h3>



<p class="wp-block-paragraph">Provider console, log in as root, and revert. If you cannot remember what you changed, <code>ls -lt /etc/ssh/sshd_config.d/</code> shows the most recently modified file. Run <code>sshd -t</code> to see whether the config is even valid, since a syntax error stops sshd starting at all.</p>



<h3 class="wp-block-heading">Key auth rejected, password prompt appears instead</h3>



<p class="wp-block-paragraph">Almost always permissions. The home directory must not be group-writable, <code>~/.ssh</code> needs 700 and <code>authorized_keys</code> 600, both owned by the user. sshd refuses keys from world-writable locations and does not explain itself on the client side. Run <code>ssh -v</code> to see how far it gets, and check <code>journalctl -u ssh</code> on the server for the actual reason.</p>



<h3 class="wp-block-heading">The setting I applied is not in effect</h3>



<p class="wp-block-paragraph"><code>sudo sshd -T</code> and search for the keyword. If the effective value differs from your file, something in <code>sshd_config.d/</code> sorts earlier, or a <code>Match</code> block later in the config is overriding it for your connection.</p>



<h3 class="wp-block-heading">A service is reachable that the firewall says is blocked</h3>



<p class="wp-block-paragraph">Docker, or another daemon writing netfilter rules directly. Check with <code>sudo iptables -S</code> or <code>sudo nft list ruleset</code>, and always verify from a machine outside the server rather than trusting <code>ufw status</code>.</p>



<h3 class="wp-block-heading">Nothing obviously wrong, but who has been logging in?</h3>



<pre class="wp-block-code"><code>last              # successful logins
sudo lastb        # failed attempts, from /var/log/btmp
journalctl -u ssh --since "24 hours ago"</code></pre>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Closing your only SSH session before testing the new configuration from a second one.</li>
<li>Enabling the firewall before adding the SSH rule.</li>
<li>Naming your sshd drop-in <code>99-</code>, so a vendor file that sorts earlier wins.</li>
<li>Trusting the file you edited instead of running <code>sshd -T</code>.</li>
<li>Generating the SSH key on the server.</li>
<li>Changing the SSH port and counting it as a security control.</li>
<li>Disabling password auth for SSH while leaving a web control panel with a weak password exposed.</li>
<li>No two-factor authentication on the hosting account that can reset root.</li>
<li>Treating provider snapshots as backups.</li>
<li>Installing fail2ban and never checking that it started.</li>
<li>Publishing container ports without binding them to localhost, then assuming ufw covers it.</li>
<li>Pasting a sysctl block you cannot explain line by line.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Verify every control after applying it, with a command that reads the running state.</li>
<li>Keep a second session open and the provider console tested before making SSH changes.</li>
<li>Key-only authentication, root login disabled, and an explicit <code>AllowUsers</code> list.</li>
<li>Default-deny inbound firewall, with rules added before the firewall is enabled.</li>
<li>Automatic security updates on, with a deliberate decision about automatic reboots.</li>
<li>Two-factor authentication on the hosting account and the registrar.</li>
<li>Backups off the machine, on separate credentials, and restore-tested at least once.</li>
<li>UTC everywhere, with clock sync confirmed.</li>
<li>Monitoring that pages you on disk usage before it pages you on downtime.</li>
<li>Script the whole build, so the next server is identical and you are not repeating this from memory.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">What is the single most important step?</h3>



<p class="wp-block-paragraph">Key-only SSH with root login disabled, verified with <code>sshd -T</code>. It removes the entire category of credential guessing. Automatic security updates is a close second, because unpatched software is the more realistic way in once passwords are off the table.</p>



<h3 class="wp-block-heading">Should I change the SSH port?</h3>



<p class="wp-block-paragraph">For quieter logs, yes. As a security measure, no. A scanner finds the daemon regardless. The practical benefit is that failed-login noise drops enough that a genuine anomaly becomes visible, which has real value for anyone actually reading their logs.</p>



<h3 class="wp-block-heading">Do I need fail2ban if password authentication is off?</h3>



<p class="wp-block-paragraph">Not for SSH specifically. Install it for the services that still accept credentials, and for the reduction in log volume and connection churn. Just do not let it occupy the slot in your head reserved for an actual defence.</p>



<h3 class="wp-block-heading">Why did my sshd_config change get ignored?</h3>



<p class="wp-block-paragraph">Two likely causes. A file in <code>/etc/ssh/sshd_config.d/</code> that sorts alphabetically earlier set the same keyword first, and sshd keeps the first value it sees. Or the setting is <code>Port</code> or <code>ListenAddress</code> on a socket-activated system, where systemd owns the listener. <code>sshd -T</code> answers the first case and <code>ss -tlnp</code> answers the second.</p>



<h3 class="wp-block-heading">Is ufw enough, or should I use nftables directly?</h3>



<p class="wp-block-paragraph">ufw is enough for a single server with a handful of open ports, and it is a front end to the same machinery underneath. Write nftables rules directly when you need something ufw cannot express cleanly, such as rate limiting or complex source-based rules. Do not switch tools for its own sake; a firewall you understand beats an elegant one you do not.</p>



<h3 class="wp-block-heading">How do I harden a server I did not build?</h3>



<p class="wp-block-paragraph">Audit before you change anything. <code>sshd -T</code>, <code>ufw status verbose</code>, <code>ss -tlnp</code> for unexpected listeners, <code>systemctl list-units --type=service --state=running</code>, and the contents of every <code>authorized_keys</code> on the box. Old keys belonging to people who left are extremely common and nobody ever removes them.</p>



<h3 class="wp-block-heading">Should this be automated?</h3>



<p class="wp-block-paragraph">Yes, once you have done it manually enough times to understand each step. An Ansible playbook or a cloud-init script makes the next server identical and turns your hardening into something reviewable in version control. Automating a process you do not yet understand just makes the mistakes reproducible.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">Editing a config file is not the same as changing a system&#8217;s behaviour. Between your file and the running daemon sit include directives, drop-in ordering, systemd units and packages that ship their own opinions, and any one of them can quietly discard what you wrote without an error message.</p>



<p class="wp-block-paragraph">So treat this VPS hardening checklist as a set of claims to be tested rather than a list to be ticked. <code>sshd -T</code> for SSH, <code>ufw status verbose</code> for the firewall, <code>ss -tlnp</code> for what is listening, <code>systemctl status</code> for anything you installed. If you cannot show the control from the running system, you have not applied it. You have only intended to.</p>



<h2 class="wp-block-heading">Want a second pair of eyes on your server?</h2>



<p class="wp-block-paragraph">Most of the servers I get handed were hardened once, by someone who has since moved on, and nobody has verified anything since. Work I take on:</p>



<ul class="wp-block-list">
<li>Auditing an existing VPS and reporting which controls are genuinely in effect versus configured but overridden.</li>
<li>Building a new server from scratch: users, SSH, firewall, automatic updates, monitoring, backups, all verified.</li>
<li>Turning a manual build into an Ansible playbook or cloud-init script so the next server matches this one.</li>
<li>Moving SSH off the public internet onto a private overlay network or bastion.</li>
<li>Backup strategy that survives the hosting account being compromised, including a tested restore.</li>
<li>Firewall and container networking, especially where published Docker ports are bypassing the firewall.</li>
</ul>



<p class="wp-block-paragraph">Send me the output of <code>sshd -T</code>, <code>ufw status verbose</code> and <code>ss -tlnp</code>, and I will tell you what stands out.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/system-administration/vps-hardening-checklist/">You Probably Didn&#8217;t Harden That Server: A VPS Setup and Hardening Checklist That Verifies Itself</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
