<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Secrets Management | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/secrets-management/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/secrets-management/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Tue, 03 Nov 2026 07:06:02 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Secrets Management | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/secrets-management/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>AWS Secrets Manager vs Parameter Store: Stop Deciding on Price</title>
		<link>https://john-nessime.com/blog/devops/aws-secrets-manager-vs-parameter-store/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 03 Nov 2026 07:06:02 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Amazon ECS]]></category>
		<category><![CDATA[Amazon RDS]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[AWS Secrets Manager]]></category>
		<category><![CDATA[CloudTrail]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[HashiCorp Vault]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[Infrastructure as Code]]></category>
		<category><![CDATA[Key Rotation]]></category>
		<category><![CDATA[Least Privilege]]></category>
		<category><![CDATA[Parameter Store]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Terraform]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=442</guid>

					<description><![CDATA[<p>Most teams pick between AWS Secrets Manager and SSM Parameter Store on price, then get bitten months later by rotation semantics, a shared throughput quota, or a recovery window that blocks a redeploy. Here are the four levers that actually decide it, per value, plus a decision procedure you can run against a real inventory.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-secrets-manager-vs-parameter-store/">AWS Secrets Manager vs Parameter Store: Stop Deciding on Price</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The pull request comment was one line: why is the database password in Parameter Store and the Stripe key in Secrets Manager?</p>



<p class="wp-block-paragraph">Nobody had a good answer. Both were added in a hurry, months apart, by different people, and each choice made sense on the day it was made. That is how most AWS accounts end up with secrets in two places. Not by design, by accumulation.</p>



<p class="wp-block-paragraph">The comparison usually gets framed as a pricing question: Parameter Store&#8217;s standard tier has no storage charge, Secrets Manager bills per secret, so use Parameter Store until somebody complains. That framing is wrong, and it stays invisible for months before it costs you anything. What separates these two services is what happens when a value <em>changes</em>, not what it costs to sit there.</p>



<p class="wp-block-paragraph">This post covers the failure modes that actually bite, the four levers that decide <strong>AWS Secrets Manager vs Parameter Store</strong> for any given value, a decision procedure you can run against a real inventory, and the arguments about this that do not survive contact with a running system.</p>



<h2 class="wp-block-heading">The two failure modes that actually bite</h2>



<h3 class="wp-block-heading">Rotation that succeeds while your application quietly breaks</h3>



<p class="wp-block-paragraph">Secrets Manager&#8217;s single user rotation strategy changes the password on the existing database user. AWS documents a short window between the password changing in the database and the secret being updated, during which calls using the rotated credentials can be denied. Retry logic covers that window.</p>



<p class="wp-block-paragraph">That is the documented gap. The one that catches people is quieter.</p>



<p class="wp-block-paragraph">Your application fetched the credential once at boot, cached it in a module-level variable, and never looked again. Rotation completes. The rotation function logs success. Nothing breaks. Then a container recycles, picks up the new value and works fine, while older containers keep running on connections opened before the change. It looks healthy right up until a connection pool churns and part of your fleet starts failing authentication against a service you have not deployed in weeks.</p>



<p class="wp-block-paragraph">The alternating users strategy exists for exactly this. Instead of changing one user&#8217;s password, Secrets Manager clones the user and alternates which one it updates, so a valid credential exists on both sides of every change. The price is a second database user and a separate superuser secret with permission to do the cloning. AWS recommends single user for most cases and for interactive or one-off credentials, and alternating users where the application needs high availability.</p>



<p class="wp-block-paragraph">The rule I use: if consumers cache and you rotate on a schedule, you need alternating users or an explicit refresh path. A shorter cache TTL narrows the window, it does not close it.</p>



<h3 class="wp-block-heading">A throughput ceiling you do not own</h3>



<p class="wp-block-paragraph">Parameter Store read throughput is a quota on the account and Region, not on the parameter and not on the caller. Every cold start, every task boot, every CI job in that account draws from the same budget. The default is sized for low-volume workloads, and there is an opt-in higher throughput setting that raises the ceiling for an additional charge.</p>



<p class="wp-block-paragraph">The failure presents as an application bug. A scale-out event or a chatty pipeline fans out, <code>GetParameter</code> starts returning <code>ThrottlingException</code>, and services fail at boot for reasons unrelated to the code anybody just shipped. Nothing in the console tells you that you are near the line.</p>



<p class="wp-block-paragraph">Both services have quotas. The difference is blast radius: Secrets Manager throttling is annoying, Parameter Store throttling is a shared resource problem where one team&#8217;s new batch job degrades another team&#8217;s deploys. Alarm on throttling for both before you need to. CloudWatch does it natively, and if your dashboards live in Grafana Cloud or a self-hosted Grafana, pull the metric across so the alert lands where people actually look.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Where Parameter Store wins, and where it hurts</h2>



<p class="wp-block-paragraph">Parameter Store is not primarily a secrets store. It is a hierarchical configuration store that encrypts when you ask it to. Hold that framing and its behaviour becomes predictable.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong></p>



<ul class="wp-block-list">
<li><strong>Configuration that is not confidential.</strong> AMI IDs, feature flags, queue names, tuning values. Putting these in Secrets Manager means paying a per-secret charge for data with no confidentiality requirement.</li>



<li><strong>Hierarchies and bulk reads.</strong> Path-based naming plus <code>GetParametersByPath</code> pulls a whole subtree in one call, which is what a service wants at startup.</li>



<li><strong>No storage charge on the standard tier,</strong> so nobody is incentivized to cram unrelated values together to save money.</li>



<li><strong>Parameter policies and AWS public parameters</strong> live here, including read-only values AWS publishes for things like current AMI IDs.</li>
</ul>



<p class="wp-block-paragraph"><strong>Where it hurts:</strong></p>



<ul class="wp-block-list">
<li><strong>No rotation.</strong> You write the function, schedule it, and own verification and rollback yourself.</li>



<li><strong>CloudFormation cannot create an encrypted parameter.</strong> The <code>AWS::SSM::Parameter</code> resource accepts <code>String</code> and <code>StringList</code> only. You can reference an existing encrypted parameter through the <code>ssm-secure</code> dynamic reference, and only on the resource properties that support that pattern.</li>



<li><strong>No recovery window.</strong> Delete is delete, and parameter history does not survive deleting the parameter.</li>



<li><strong>Cross-account sharing requires the advanced tier,</strong> through Resource Access Manager.</li>
</ul>



<h2 class="wp-block-heading">Where AWS Secrets Manager wins, and where it hurts</h2>



<p class="wp-block-paragraph">Secrets Manager is a lifecycle service. Storage is the least interesting thing it does.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong></p>



<ul class="wp-block-list">
<li><strong>Managed rotation</strong> for Amazon RDS, Aurora, Redshift and DocumentDB, using AWS-published rotation functions and two documented strategies.</li>



<li><strong>Versioning with staging labels.</strong> <code>AWSCURRENT</code>, <code>AWSPENDING</code> and <code>AWSPREVIOUS</code> give rotation a defined state machine and give you a last known good version to fall back to.</li>



<li><strong>A recovery window on delete,</strong> between seven and thirty days, defaulting to thirty.</li>



<li><strong>Cross-region replication and resource policies</strong> as first-class features, so multi-Region and cross-account access are properties of the secret rather than things you build.</li>
</ul>



<p class="wp-block-paragraph"><strong>Where it hurts:</strong></p>



<ul class="wp-block-list">
<li><strong>Cost scales by multiplication.</strong> Storage bills per secret per month, prorated hourly, and every cross-region replica bills as its own secret. The multiplier is secrets times environments times Regions.</li>



<li><strong>The recovery window is a deployment trap.</strong> Destroy an environment, apply again, and the create fails because the name is still reserved.</li>



<li><strong>Managed rotation only covers databases.</strong> Anything else means writing your own function, so you carry the operational burden without the free tier.</li>



<li><strong>The pricing model pushes you toward bundling.</strong> One secret per value is the clean design and the expensive one, so teams pack fields into JSON and lose per-value IAM granularity.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Four levers that decide AWS Secrets Manager vs Parameter Store</h2>



<h3 class="wp-block-heading">1. Does the value have to change on a schedule?</h3>



<p class="wp-block-paragraph">If yes, and it is a credential for RDS, Aurora, Redshift or DocumentDB, use Secrets Manager and stop deliberating. You are not paying for storage. You are paying to avoid writing, testing and maintaining a rotation function, and to inherit a staging-label state machine that makes a failed rotation recoverable instead of an outage.</p>



<p class="wp-block-paragraph">If the value never changes on a schedule, an AMI ID, a webhook URL, a vendor key you replace manually once a year, then rotation machinery buys you nothing.</p>



<h3 class="wp-block-heading">2. How many independent gates guard the read path?</h3>



<p class="wp-block-paragraph">This is the lever people skip, and it matters more than the service choice.</p>



<p class="wp-block-paragraph">Both services encrypt through KMS. The question is whether KMS is a second, independent authorization decision or a formality. An encrypted parameter under the AWS managed key for Systems Manager is decrypted through that key on behalf of the service, so in practice the SSM permission is the gate. Encrypt under a customer managed key instead and the caller must also be allowed <code>kms:Decrypt</code> on that specific key. Now two policies in two services both have to say yes, and you can scope the key policy to a named set of roles independently of who may call Systems Manager. The first gate protects the API surface, the second protects the plaintext.</p>



<p class="wp-block-paragraph">The same split exists in Secrets Manager, between the AWS managed key and one you create.</p>



<pre class="wp-block-code"><code># Store an encrypted parameter under a customer managed key
# instead of the AWS managed default.
aws ssm put-parameter 
  --name "/prod/billing/db-password" 
  --type SecureString 
  --key-id "alias/prod-app-secrets" 
  --value "REPLACE_ME" 
  --overwrite

# Reading it now needs ssm:GetParameter AND
# kms:Decrypt on alias/prod-app-secrets.
aws ssm get-parameter 
  --name "/prod/billing/db-password" 
  --with-decryption 
  --query "Parameter.Value" 
  --output text</code></pre>



<p class="wp-block-paragraph">There is a second payoff. A customer managed key makes the decrypt call attributable in CloudTrail against a key policy you control, which turns &#8220;someone read a secret&#8221; into &#8220;this role decrypted this value with this key at this time&#8221;. That is usually what an auditor is asking for, and it is independent of which store you land on.</p>



<h3 class="wp-block-heading">3. What does the retrieval path look like under load?</h3>



<p class="wp-block-paragraph">Neither service is designed to be called on every request. Both punish you for it in different currencies: Parameter Store with a shared throughput ceiling, Secrets Manager with a per-call charge.</p>



<p class="wp-block-paragraph">The answer for both is caching, and AWS ships the mechanism rather than making you build it. The Secrets Manager Agent, now published under the name AWS Workload Credentials Provider, runs as a sidecar or local service, holds secrets in memory, and serves them to your application over a local HTTP endpoint, so the app makes one call instead of thousands and needs no AWS SDK dependency to do it. For Lambda, the AWS Parameters and Secrets Lambda Extension does the equivalent inside the execution environment and covers both services.</p>



<p class="wp-block-paragraph">Caching moves the problem rather than removing it. A cached credential is a stale credential during rotation, which loops straight back to lever one.</p>



<h3 class="wp-block-heading">4. How does the value get created and destroyed?</h3>



<p class="wp-block-paragraph">Creation first. CloudFormation cannot create an encrypted parameter, so if your pipeline is CloudFormation or CDK and the value must be encrypted from the moment it exists, Secrets Manager is the path of least resistance. The usual workarounds, a Lambda-backed custom resource or creating the parameter out of band, both move the secret outside the template that claims to own it. That is a real cost, not a detail.</p>



<p class="wp-block-paragraph">Destruction is where the surprise lives. Secrets Manager schedules deletion after a recovery window with a minimum of seven days and a default of thirty, and the name stays reserved for the duration.</p>



<pre class="wp-block-code"><code># Shorten the recovery window to the documented minimum.
aws secretsmanager delete-secret 
  --secret-id prod/billing/db 
  --recovery-window-in-days 7

# Or skip the window entirely. There is no undo after this.
aws secretsmanager delete-secret 
  --secret-id ephemeral/pr-1482/db 
  --force-delete-without-recovery</code></pre>



<p class="wp-block-paragraph">AWS notes the actual deletion runs as an asynchronous background process, so deleting and immediately recreating a secret with the same name needs backoff and retry even with the force flag. This is why per-branch environments built with Terraform or OpenTofu fail on their second run. Use the force flag deliberately, put a unique suffix in the name, or keep ephemeral values in Parameter Store where deletion is immediate.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Cost levers, without quoting numbers that will age</h2>



<p class="wp-block-paragraph">Published rates change. The billing mechanisms are stable, and the mechanisms are what let you predict the bill.</p>



<ul class="wp-block-list">
<li><strong>Parameter Store standard tier</strong> carries no storage charge. The advanced tier is charged per parameter, and the higher throughput setting is charged separately as an account and Region level switch.</li>



<li><strong>Secrets Manager</strong> charges per stored secret per month, prorated hourly, plus a charge per block of API calls. Replicas count as separate stored secrets.</li>



<li><strong>KMS</strong> charges a monthly fee per customer managed key plus a request charge on cryptographic operations. This applies to both services and it is the line most estimates forget.</li>



<li><strong>Rotation</strong> has no separate line item, but the rotation function bills as ordinary Lambda usage.</li>
</ul>



<p class="wp-block-paragraph">Two consequences. The Secrets Manager multiplier is not the number of secrets, it is secrets across environments across Regions, and replicas are the sneaky term because nobody remembers enabling them. And the advanced parameter tier is a one-way door: you can promote a standard parameter to advanced, but you cannot demote it, because reverting would truncate the value. Undoing that means delete and recreate.</p>



<p class="wp-block-paragraph">If you cannot see which term is growing, break the bill down by service and tag before you migrate anything. Cost Explorer handles the single-account case; for cross-account views without building the aggregation yourself, tools like Vantage or CloudZero will answer &#8220;is this secret count, replication, or retrieval frequency&#8221; in a few minutes. That is a better use of an afternoon than guessing.</p>



<h2 class="wp-block-heading">A decision procedure you can run today</h2>



<p class="wp-block-paragraph">Take an actual list of values and walk each one through this. It takes about a minute each.</p>



<ol class="wp-block-list">
<li><strong>Is it confidential at all?</strong> If no, Parameter Store, plain <code>String</code> type, done. Do not encrypt configuration for the aesthetics.</li>



<li><strong>Is it a database credential that should rotate on a schedule?</strong> Secrets Manager with managed rotation. Choose single user or alternating users based on whether consumers cache.</li>



<li><strong>Does it need scheduled rotation but is not a supported database?</strong> You are writing a function either way. Secrets Manager if you want versioning, staging labels and a recovery window; Parameter Store if the free tier matters more and you accept owning the lifecycle.</li>



<li><strong>Does it need to exist in more than one Region or account?</strong> Secrets Manager gives you replication and resource policies directly. Parameter Store shares across accounts through Resource Access Manager, advanced tier only.</li>



<li><strong>Does CloudFormation or CDK create it?</strong> Encrypted parameters are not creatable there, so it is Secrets Manager or an accepted out-of-band step.</li>



<li><strong>Is it read frequently?</strong> Cache it either way, then return to step two, because caching and rotation interact and that interaction is where outages come from.</li>



<li><strong>Everything else</strong> goes to Parameter Store, encrypted under a customer managed key.</li>
</ol>



<p class="wp-block-paragraph">Most accounts land with the large majority of values in Parameter Store and a short list of rotating database credentials in Secrets Manager. That split is healthy. What is not healthy is being unable to say why a particular value is where it is.</p>



<p class="wp-block-paragraph">Once you accept you will use both, the operational work is making them look like one system: one naming convention across both stores, IAM scoped to the path rather than the service, KMS keys separated by trust boundary rather than by service, and the rule written down where the next person will find it. Two lines in a repository README beats a policy document nobody opens.</p>



<h2 class="wp-block-heading">Arguments that do not survive contact</h2>



<p class="wp-block-paragraph"><strong>&#8220;Parameter Store is free, so use it for everything.&#8221;</strong> The standard tier has no storage charge, which is not the same as free. KMS requests bill, the higher throughput setting bills, advanced parameters bill, and the rotation function you write instead of using a managed one has a real cost in engineering hours and incident risk.</p>



<p class="wp-block-paragraph"><strong>&#8220;Secrets Manager is expensive.&#8221;</strong> Per secret it is small. The bill grows through multiplication, not unit price. Before migrating anything, count secrets across environments and Regions and find out how many replicas exist that nobody asked for. Deleting unused replicas is usually a bigger win than a migration project.</p>



<p class="wp-block-paragraph"><strong>&#8220;Just use Vault and skip the argument.&#8221;</strong> HashiCorp Vault is a better answer if you are multi-cloud, need dynamic short-lived credentials, or want secrets engines AWS does not offer. It also hands you a cluster to run, upgrade, seal and unseal, back up and monitor, plus a licensing decision. On a single-cloud AWS estate with a handful of rotating credentials, the AWS services are less capable and less work. Both statements are true at once, and which dominates depends on how much platform team you have.</p>



<p class="wp-block-paragraph"><strong>&#8220;It is encrypted at rest, so it is secure.&#8221;</strong> Encryption at rest protects against someone reading the storage layer, which is not the threat you have. Your threat is an over-permissive role and a compromised workload. That is an IAM policy and a KMS key policy question, and the answer is the same on either service.</p>



<p class="wp-block-paragraph"><strong>&#8220;We will add rotation later.&#8221;</strong> Rotation is not a feature flag, it is a coordination problem between the store, the target system and every consumer&#8217;s caching behaviour. Deciding that a credential will eventually rotate changes how you write the consumer today. Retrofitting it means touching every consumer anyway.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I migrate a value between the two without downtime?</h3>



<p class="wp-block-paragraph">Yes, with a dual read. Write the value to the new location, deploy consumers that read the new location and fall back to the old one, then use CloudTrail to confirm nothing still reads the old path before deleting it. Do not skip the CloudTrail step; the consumer you forgot is always a job that runs monthly. If the value passed through a local file during the move, erase that file properly rather than with <code>rm</code>, using <code>shred</code> on Linux or a secure deletion tool such as those from O&amp;O Software on Windows.</p>



<h3 class="wp-block-heading">Does Parameter Store support rotation?</h3>



<p class="wp-block-paragraph">Not as a built-in feature. You write a function and invoke it on a schedule, typically with an EventBridge rule. You also inherit everything managed rotation handles for you: verifying the new credential works before committing to it, rolling back cleanly when it does not, and giving consumers an overlap window.</p>



<h3 class="wp-block-heading">Which one should I use for third-party API keys?</h3>



<p class="wp-block-paragraph">It depends on whether you rotate them. Most vendor keys get rotated manually and rarely, in which case an encrypted parameter under a customer managed key is enough. If the provider supports programmatic key rotation and you want that automated, Secrets Manager&#8217;s version staging is what makes the swap safe.</p>



<h3 class="wp-block-heading">Can workloads outside AWS read from either service?</h3>



<p class="wp-block-paragraph">Both, over the API, with AWS credentials. That means a long-lived access key or a federated identity, and you should federate wherever the platform supports it. If part of your estate runs on VPS hosts at a provider like Contabo or InterServer, this is the boundary to think hardest about, because you have just placed an AWS credential on a machine whose network you do not control. Scope the role narrowly, apply condition keys where the network path allows, and treat that credential as the one most likely to leak.</p>



<h3 class="wp-block-heading">What happens if I delete something by accident?</h3>



<p class="wp-block-paragraph">Secrets Manager lets you restore during the recovery window, which is the single best argument for putting genuinely important credentials there. Parameter Store has no equivalent. Once the parameter is gone your options are whatever exists in state files, backups or someone&#8217;s terminal history, and none of those is a plan.</p>



<h3 class="wp-block-heading">Should ECS or EKS workloads pull secrets differently?</h3>



<p class="wp-block-paragraph">ECS can inject values from either service directly into a task&#8217;s environment, which removes SDK code from the container but reads the value at task start, so a rotation does not reach a running task. On EKS you have both the Secrets Store CSI driver with the AWS provider, which mounts values as files, and the agent sidecar pattern, which serves them over localhost and can refresh without restarting the pod. Choose based on whether your application can reload a credential mid-life. If it cannot, mounting and injection are equivalent, and rotation means a rollout.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The <strong>AWS Secrets Manager vs Parameter Store</strong> decision is a lifecycle decision wearing a pricing costume. What separates the two is not what they store, it is what they do when the value changes, when it gets deleted, when a Region falls over, and when ten thousand containers ask for it at once.</p>



<p class="wp-block-paragraph">Decide per value, not per project. Configuration and rarely-changing keys belong in Parameter Store. Database credentials that rotate on a schedule belong in Secrets Manager. Put a customer managed key in front of both so the second gate does real work, cache reads so you are not fighting a quota, and make sure whoever wrote the caching understands what happens the next time a rotation runs. If you cannot explain why a given value lives where it does, that is the value that will surprise you.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your secrets setup?</h2>



<p class="wp-block-paragraph">Most of the work here is not choosing a service, it is untangling what already exists. Things I can help with:</p>



<ul class="wp-block-list">
<li>Auditing an AWS account for where secrets actually live, which roles can decrypt them, and which values are duplicated across environments or replicated into Regions nobody uses</li>



<li>Designing a naming hierarchy and path-scoped IAM so one policy covers a service instead of one policy per value</li>



<li>Setting up managed rotation for RDS and Aurora credentials, including the alternating users pattern when the application caches and cannot be changed</li>



<li>Writing and testing custom rotation functions for non-database secrets, with the rollback path actually exercised rather than assumed</li>



<li>Moving values between the two stores with a dual-read cutover and CloudTrail verification that nothing is left reading the old path</li>



<li>Working out whether a growing Secrets Manager line item is secret count, replication or retrieval frequency, and fixing the one actually responsible</li>
</ul>



<p class="wp-block-paragraph">Send me a redacted IAM policy, a CloudTrail query result, or the output of listing your parameters by path, and I will tell you what I would change first and why.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-secrets-manager-vs-parameter-store/">AWS Secrets Manager vs Parameter Store: Stop Deciding on Price</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>PHP2Shell Vulnerability: What Happened, How to Patch, and How to Prove It Worked</title>
		<link>https://john-nessime.com/blog/system-administration/php2shell-vulnerability-fix/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 24 Oct 2026 12:36:55 +0000</pubDate>
				<category><![CDATA[Linux]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[WordPress]]></category>
		<category><![CDATA[Cloudflare]]></category>
		<category><![CDATA[CloudLinux]]></category>
		<category><![CDATA[cPanel]]></category>
		<category><![CDATA[CustomBuild]]></category>
		<category><![CDATA[CVE Management]]></category>
		<category><![CDATA[CVE-2026-109618]]></category>
		<category><![CDATA[CVE-2026-109619]]></category>
		<category><![CDATA[DirectAdmin]]></category>
		<category><![CDATA[Docker]]></category>
		<category><![CDATA[Incident Response]]></category>
		<category><![CDATA[mod_php]]></category>
		<category><![CDATA[Patch Management]]></category>
		<category><![CDATA[PHP]]></category>
		<category><![CDATA[PHP-FPM]]></category>
		<category><![CDATA[PHP2Shell]]></category>
		<category><![CDATA[Remote Code Execution]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Server Hardening]]></category>
		<category><![CDATA[Vulnerability Management]]></category>
		<category><![CDATA[Web Application Firewall]]></category>
		<category><![CDATA[WordPress Security]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=1217</guid>

					<description><![CDATA[<p>PHP2Shell is a pair of critical flaws inside PHP itself, so updating WordPress or your framework fixes nothing. Here is what broke, who is exposed, how to patch every common setup, and how to prove the fix is actually running.</p>
<p>The post <a href="https://john-nessime.com/blog/system-administration/php2shell-vulnerability-fix/">PHP2Shell Vulnerability: What Happened, How to Patch, and How to Prove It Worked</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">On the evening of 20 October 2026, php.net posted something it almost never posts: an advance warning that an emergency security release was coming the next day, for two critical bugs it would not yet describe. By the next morning the details had leaked, a proof-of-concept video was circulating, and the same message was landing in every hosting inbox: &#8220;We updated WordPress last week. We&#8217;re fine, right?&#8221;</p>



<p class="wp-block-paragraph">No. And that question is the entire problem with this one.</p>



<p class="wp-block-paragraph">This post is a reference for the <strong>PHP2Shell vulnerability</strong>: what actually broke inside PHP, which servers are exposed, how to patch each common setup, how to prove the fix is really running, and which quick fixes give you false comfort. Keep it open while you work through your servers.</p>



<h2 class="wp-block-heading">What happened: the PHP2Shell vulnerability in plain terms</h2>



<p class="wp-block-paragraph">PHP2Shell is the nickname researchers gave to two flaws in the PHP engine itself, <strong>CVE-2026-109618</strong> and <strong>CVE-2026-109619</strong>. PHP rates both 9.2 on CVSS v4.0, and both can let an unauthenticated attacker run code on a web server. The name is the researchers&#8217;, not the PHP project&#8217;s.</p>



<p class="wp-block-paragraph">The dates matter here, because they decide if you were exposed and for how long:</p>



<ul class="wp-block-list">
<li><strong>20 October 2026:</strong> PHP announced an emergency release for two critical issues. Details leaked publicly the same day.</li>



<li><strong>21 October 2026:</strong> fixed releases shipped: <strong>PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35</strong>, with a full advisory.</li>



<li><strong>23 October 2026:</strong> PHP published an informational advisory for a third, much narrower issue, CVE-2026-115560, with a one-line mitigation and no emergency release.</li>
</ul>



<p class="wp-block-paragraph">The flaws were reported by Wiz Research, Calif and OpenAI&#8217;s Codex Security team, who used AI models to turn long-standing weaknesses into working exploits. The affected code goes back to PHP 5.4, so effectively every PHP version from the last decade is in scope.</p>



<h3 class="wp-block-heading">The two bugs, without the jargon</h3>



<p class="wp-block-paragraph"><strong>CVE-2026-109618 (out-of-memory corruption).</strong> PHP enforces <code>memory_limit</code> by aborting a request the instant it crosses the limit. If that abort lands while PHP is still building internal request data, for example while parsing a multipart file upload, the data is left half-initialized. Request cleanup then treats it as valid and works with memory that can still hold attacker-controlled leftovers.</p>



<p class="wp-block-paragraph"><strong>CVE-2026-109619 (stale global state).</strong> Some state owned by one request can survive after its memory has been freed, for example after a fatal error. The next request handled by the same worker can use that stale state before normal setup resets it.</p>



<p class="wp-block-paragraph">Here is why this is worse than the usual plugin CVE. The bug lives in the runtime, not in your code. A site whose own code is flawless is still exposed, and updating WordPress, Laravel or Drupal changes nothing, because the PHP binary underneath comes from your OS packages, your host or your container image. The researchers also demonstrated reading memory from the PHP worker process, which can contain passwords, tokens and keys belonging to other people&#8217;s requests, without ever getting a shell.</p>



<h2 class="wp-block-heading">Who is exposed, and why mod_php is the worst case</h2>



<p class="wp-block-paragraph">PHP&#8217;s advisory covers every installation that serves web requests: PHP-FPM, Apache with mod_php, php-cgi, FrankenPHP and LiteSpeed. The risk is not equal across them, though.</p>



<p class="wp-block-paragraph"><strong>mod_php is the highest risk.</strong> Apache with mod_php runs request shutdown even when request startup already failed. That means simply running out of memory while PHP parses the request body is enough, and one unauthenticated request can trigger the attack before a single line of your application runs. PHP demonstrated this with default memory and input limits.</p>



<p class="wp-block-paragraph"><strong>PHP-FPM is harder to hit, not safe.</strong> FPM shuts the worker down when a request fails early in setup, which blocks the cleanest path. Public reporting says one of the two bugs was still shown to reach code execution under FPM, through a path that depends on something the application does. Treat &#8220;we run FPM&#8221; as &#8220;we have slightly more time,&#8221; nothing more.</p>



<p class="wp-block-paragraph">If you are not sure which mode you run, these two checks answer it on most Linux servers. The first lists Apache modules and shows a PHP module only if mod_php is loaded. The second lists running FPM services.</p>



<pre class="wp-block-code"><code>apachectl -M | grep -i php
systemctl list-units --type=service | grep -i fpm</code></pre>



<h2 class="wp-block-heading">How to fix PHP2Shell: it depends on where your PHP comes from</h2>



<p class="wp-block-paragraph">The fix is always the same in principle: run a patched PHP build. Getting there depends entirely on who builds your PHP. And there is a trap that runs in both directions. Linux distributions and long-term-support vendors received the patch early and backported it, so a <strong>patched</strong> package can carry an old-looking version like 8.3.6. Meanwhile, an <strong>unpatched</strong> process can keep running after the package updated. The version string answers neither question reliably.</p>



<h3 class="wp-block-heading">Debian and Ubuntu packages</h3>



<p class="wp-block-paragraph">Debian, Ubuntu and Ondřej Surý&#8217;s widely used repository (packages.sury.org, also published as ppa:ondrej/php) all shipped fixed packages, including builds for some end-of-life branches. Update normally, then confirm the CVE is in the package changelog instead of reading the version number. Replace 8.3 with your branch.</p>



<pre class="wp-block-code"><code>sudo apt update &amp;&amp; sudo apt upgrade
zgrep CVE-2026-109618 /usr/share/doc/php8.3-common/changelog.Debian.gz
sudo systemctl restart php8.3-fpm
sudo systemctl restart apache2</code></pre>



<p class="wp-block-paragraph">If <code>zgrep</code> prints a line, the fix is in the installed package. If it prints nothing, you are either on an older distro release that needs a paid extended-support channel, or on a repository that has not shipped the fix.</p>



<h3 class="wp-block-heading">RHEL, AlmaLinux and Rocky Linux</h3>



<p class="wp-block-paragraph">Same idea with RPM. Update, then search the package changelog for the CVE. If you use Remi&#8217;s parallel packages, the package name follows its own pattern (for example <code>php83-php-common</code>), so query that name instead.</p>



<pre class="wp-block-code"><code>sudo dnf upgrade 'php*'
rpm -q --changelog php-common | grep CVE-2026-109618
sudo systemctl restart php-fpm httpd</code></pre>



<p class="wp-block-paragraph">One warning from PHP&#8217;s own advisory: the volunteer-run rpms.remirepo.net repository was not expected to have updates on release day, and PHP suggested switching to distribution packages or building from source if you depend on it. Check before assuming the update landed.</p>



<h3 class="wp-block-heading">cPanel, CloudLinux and LiteSpeed</h3>



<p class="wp-block-paragraph">cPanel ships its own <code>ea-php</code> packages through EasyApache 4, and CloudLinux ships <code>alt-php</code>. CloudLinux backported the fix into supported and end-of-life branches, so most of its fixed builds keep version numbers below the php.net release. CloudLinux&#8217;s published procedure is to update both package families, then restart the services that run PHP so no worker keeps the old binary loaded.</p>



<pre class="wp-block-code"><code>yum clean all &amp;&amp; yum update 'alt-php*' 'ea-php*' -y
/scripts/restartsrv_httpd</code></pre>



<p class="wp-block-paragraph">If your domains run PHP-FPM through cPanel, restart the FPM service as well. On LiteSpeed, restart the web server so every <code>lsphp</code> process respawns from the new build. Public analysis says LiteSpeed&#8217;s LSAPI behaves more like FPM than mod_php, which buys time but does not remove the need to patch.</p>



<h3 class="wp-block-heading">DirectAdmin with CustomBuild</h3>



<p class="wp-block-paragraph">CustomBuild compiles PHP from source, so you get the fix when CustomBuild offers the patched release and you rebuild. Refresh the version list, confirm the fixed release appears for each PHP version you run, then build.</p>



<pre class="wp-block-code"><code>cd /usr/local/directadmin/custombuild
./build update
./build versions
./build update_versions</code></pre>



<p class="wp-block-paragraph">Read the <code>./build versions</code> output before running the last command. If a branch you use still shows an unpatched release, apply the mitigations below for that branch until it updates. Compiling takes a while on small servers, so plan for it rather than starting it at peak traffic.</p>



<h3 class="wp-block-heading">Docker and container images</h3>



<p class="wp-block-paragraph">A container never patches itself. If your image is built <code>FROM</code> an official <code>php</code> tag, rebuild with a fresh pull of the base image and redeploy. If your Dockerfile pins the base by digest, the digest has to change or nothing happens.</p>



<pre class="wp-block-code"><code>docker compose build --pull
docker compose up -d</code></pre>



<p class="wp-block-paragraph">Then run <code>php -v</code> inside the running container, not on the host, to confirm the version the app actually uses.</p>



<h3 class="wp-block-heading">Managed hosting versus unmanaged VPS</h3>



<p class="wp-block-paragraph">On managed hosting, the host owns the PHP runtime, and several large hosts announced they had patched their fleets. Your job there is to get written confirmation, not to assume. On an unmanaged VPS from Contabo, Interserver or any similar provider, nobody patches PHP but you. That is the trade you accepted for the lower price and full root access, and this week is when it gets collected.</p>



<h3 class="wp-block-heading">End-of-life PHP: 8.1 and older</h3>



<p class="wp-block-paragraph">The PHP project released no fixed builds below 8.2. Because of the severity, its security team published minimal source backports for PHP 7.4, 8.0 and 8.1 for people who build from source. Beyond that, your options are a distribution or vendor that backports (Ubuntu&#8217;s own PHP 8.1 on 22.04, the Surý repository, Freexian ELTS for older Debian, CloudLinux and TuxCare, HeroDevs), or upgrading. Note that PHP 8.2 itself reaches end of life on 31 December 2026, so if you are touching an 8.2 server now, plan the next upgrade at the same time.</p>



<h2 class="wp-block-heading">Prove the patch is actually running</h2>



<p class="wp-block-paragraph">This is the failure mode that will bite most people. The package manager says &#8220;upgraded,&#8221; the ticket gets closed, and the old PHP code keeps serving traffic because the processes that loaded it were never restarted. Linux happily keeps running a binary whose file has been replaced on disk.</p>



<ol class="wp-block-list">
<li><strong>Confirm the fix is in the package</strong> with the changelog check above, not the version string.</li>



<li><strong>Restart everything that runs PHP:</strong> every FPM pool, Apache (for mod_php), LiteSpeed, and any long-running PHP workers such as queue consumers managed by Supervisor or systemd.</li>



<li><strong>Check the restart time is after the upgrade.</strong> <code>systemctl status php8.3-fpm</code> shows when the service started. If that is earlier than your package install, it is still running old code.</li>



<li><strong>Ask the system what still needs a restart.</strong> On Debian and Ubuntu, <code>sudo needrestart -r l</code> lists services still using outdated files, without restarting anything.</li>



<li><strong>Check the right binary.</strong> <code>php -v</code> on the command line reports the CLI build, which can differ from the FPM build serving your site. On Debian and Ubuntu, <code>php-fpm8.3 -v</code> asks the FPM binary directly.</li>
</ol>



<h2 class="wp-block-heading">Mitigations if you cannot patch today, and what they don&#8217;t do</h2>



<p class="wp-block-paragraph">PHP is blunt about this: only an update is a complete fix. The settings below reduce exposure to the specific attack paths it has confirmed. They do not make a server safe, and some of them have real costs. Use them to buy hours, not weeks.</p>



<p class="wp-block-paragraph">Set them where applications cannot override them: <code>php_admin_value</code> or <code>php_admin_flag</code> in the FPM pool, or the Apache configuration for mod_php. Do not rely on <code>ini_set()</code> or an FPM <code>.user.ini</code> file, because those apply too late in the request. Here is the FPM pool version on a Debian-style path (adjust the file and branch to yours):</p>



<pre class="wp-block-code"><code>; /etc/php/8.3/fpm/pool.d/www.conf
php_admin_value[memory_limit] = -1
php_admin_value[post_max_size] = 8M
php_admin_value[upload_max_filesize] = 6M
php_admin_flag[session.upload_progress.enabled] = off</code></pre>



<p class="wp-block-paragraph">And the Apache equivalent for mod_php, inside the virtual host or a <code>Directory</code> block:</p>



<pre class="wp-block-code"><code>php_admin_value memory_limit -1
php_admin_value post_max_size 8M
php_admin_value upload_max_filesize 6M
php_admin_flag session.upload_progress.enabled off</code></pre>



<p class="wp-block-paragraph">Restart every worker afterwards. Now the reasoning behind each line, because each one protects a different path:</p>



<ul class="wp-block-list">
<li><strong><code>memory_limit = -1</code></strong> targets CVE-2026-109618. The confirmed exploit needs a predictable per-request limit to hit, so removing the limit removes the trigger. The cost: one request can now consume all available memory. A genuine allocation failure can also still reach the vulnerable code.</li>



<li><strong>Low <code>post_max_size</code></strong> shrinks the request bodies an attacker can use. PHP suggests 8M where uploads are not needed and 32M where they are, with <code>upload_max_filesize</code> kept below it.</li>



<li><strong><code>session.upload_progress.enabled = off</code></strong> targets CVE-2026-109619. It blocks the confirmed session paths but not other variants. It does nothing at all for CVE-2026-109618.</li>



<li><strong>On PHP 8.5 only, <code>fatal_error_backtraces = off</code></strong> blocks the confirmed fatal-backtrace path for CVE-2026-109619, and nothing else.</li>
</ul>



<p class="wp-block-paragraph">Because removing the memory limit opens a denial-of-service risk, PHP recommends a container or cgroup memory limit as the availability boundary, ideally one that kills only the affected worker. It specifically warns against <code>RLIMIT_AS</code> or <code>RLIMIT_DATA</code>, since those make allocations fail at the limit, which is exactly the condition the bug needs. Enforcing request-size, field-count and multipart-part limits at your reverse proxy helps too.</p>



<p class="wp-block-paragraph">A web application firewall is a reasonable stopgap, not a fix. Cloudflare deployed rules before the patch shipped and reported attackers changing payloads to get around them. If your site already sits behind Cloudflare, those rules help. They do not replace the update.</p>



<h3 class="wp-block-heading">The third issue: CVE-2026-115560</h3>



<p class="wp-block-paragraph">Two days after the release, PHP disclosed one more possible code-execution path. It needs an unusual setup: the application sets <code>session.serialize_handler</code> to <code>php_binary</code> or <code>php_serialize</code> in configuration, then switches it to <code>php</code> at runtime with <code>ini_set()</code> before calling <code>session_start()</code>, with session upload progress still enabled. The fixed releases above do not cover it. PHP&#8217;s advice is to keep <code>session.upload_progress.enabled=0</code> until the next regular bugfix release. If you already set it for CVE-2026-109619, leave it in place and you are done.</p>



<h2 class="wp-block-heading">After patching: assume someone tried</h2>



<p class="wp-block-paragraph">Amazon reported a honeypot hit against these flaws on 17 October, four days before disclosure, and exploitation attempts climbed fast once details were public. Most reports so far describe scanning and failed attempts rather than confirmed break-ins, but &#8220;nobody has reported it&#8221; is not evidence about your server. If you ran an unpatched, internet-facing PHP site during that window, do three things.</p>



<ul class="wp-block-list">
<li><strong>Rotate secrets a PHP worker could have held in memory:</strong> database passwords, API keys, payment provider keys, SMTP credentials and, for WordPress, the salts in <code>wp-config.php</code> (which logs every user out, so warn people first).</li>



<li><strong>Read your logs for the patterns researchers flagged:</strong> unusual multipart requests followed by memory-limit errors or bursts of HTTP 500 responses, oddly large <code>PHPSESSID</code> cookies in responses, and PHP workers restarting with no explanation.</li>



<li><strong>Look for anything new and executable</strong> in your web roots and upload directories, especially PHP files that no deploy put there.</li>
</ul>



<h2 class="wp-block-heading">Troubleshooting</h2>



<p class="wp-block-paragraph"><strong>&#8220;<code>php -v</code> still shows the old version.&#8221;</strong> Either it is a distro backport (check the changelog for the CVE), or you are looking at the CLI binary while a different build serves the site. Check the FPM binary directly and the container, not the host.</p>



<p class="wp-block-paragraph"><strong>&#8220;The site throws 500 errors after the update.&#8221;</strong> The fix changes how PHP handles running out of memory. Crossing <code>memory_limit</code> now raises a soft error at a safe point, and a second, higher threshold ends the process outright, without running shutdown handlers. A plugin or import job that used to squeak by can now fail differently. Check the error log, then review <code>memory_limit</code> and <code>post_max_size</code> against what the application genuinely needs.</p>



<p class="wp-block-paragraph"><strong>&#8220;Cleanup or logging code stopped running on big jobs.&#8221;</strong> Same cause. If anything important lives in a shutdown function, it will not run when the hard limit is hit. Fix the memory usage rather than raising limits blindly.</p>



<p class="wp-block-paragraph"><strong>&#8220;The package manager says there is nothing to update.&#8221;</strong> You are probably on a repository that has not shipped the fix, an end-of-life distro release, or an end-of-life PHP branch with no vendor backport. That is a migration problem, not a patching problem.</p>



<p class="wp-block-paragraph"><strong>&#8220;I set the mitigation but the old value still shows.&#8221;</strong> Look for overrides in other pool files, per-directory configuration or application code, and make sure you used the admin variants and restarted the workers.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Updating WordPress, plugins or the framework and calling it done. None of that touches the PHP runtime.</li>



<li>Trusting the version number in either direction, and declaring a backported package &#8220;vulnerable&#8221; or a stale process &#8220;patched.&#8221;</li>



<li>Upgrading the package and never restarting FPM, Apache or long-running workers.</li>



<li>Treating <code>session.upload_progress.enabled=0</code> as the fix. It was circulated as a workaround before the release and only covers part of one CVE.</li>



<li>Setting mitigations with <code>ini_set()</code>, <code>.htaccess</code> overrides that apps can change, or <code>.user.ini</code>, which take effect too late.</li>



<li>Leaving <code>memory_limit=-1</code> in place long after patching, with no cgroup boundary behind it.</li>



<li>Forgetting the boxes nobody owns: the staging server, the old client site on PHP 7.4, the forgotten container in a side project.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Keep an inventory of every PHP install, its source (distro, panel, container, source build) and its branch. This week shows why.</li>



<li>Verify fixes by CVE in the package changelog, and verify the running process by start time.</li>



<li>Patch mod_php servers first, then public FPM sites, then internal tools.</li>



<li>Once patched, restore a sane <code>memory_limit</code>, but keep session upload progress off until PHP&#8217;s next regular release covers CVE-2026-115560.</li>



<li>Move servers off mod_php to PHP-FPM where you can. It would not have saved you here, but it narrowed the easiest path.</li>



<li>Stay on a supported PHP branch, or pay for a vendor that backports. An unsupported branch turns every emergency into a migration.</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">What is the PHP2Shell vulnerability?</h3>



<p class="wp-block-paragraph">It is the nickname for two memory-safety flaws in the PHP engine, CVE-2026-109618 and CVE-2026-109619, both rated 9.2 on CVSS v4.0. They can allow unauthenticated remote code execution on servers that run PHP for web requests.</p>



<h3 class="wp-block-heading">Which PHP versions fix PHP2Shell?</h3>



<p class="wp-block-paragraph">PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35 from php.net. Distribution and vendor packages may carry the fix under different, older-looking version numbers, so check the package changelog for the CVE.</p>



<h3 class="wp-block-heading">Is my WordPress site affected?</h3>



<p class="wp-block-paragraph">If it runs on an unpatched PHP build, yes, regardless of how up to date WordPress and its plugins are. The flaw is in PHP itself, so the fix comes from your server, host or container image.</p>



<h3 class="wp-block-heading">Is disabling session.upload_progress enough?</h3>



<p class="wp-block-paragraph">No. It blocks the confirmed session paths for one of the two bugs and covers the narrow third issue, CVE-2026-115560. It does nothing for CVE-2026-109618. Only an update is a complete fix.</p>



<h3 class="wp-block-heading">I&#8217;m on PHP 7.4 or 8.1. What now?</h3>



<p class="wp-block-paragraph">php.net has no release for you, but PHP published source backports for 7.4, 8.0 and 8.1, and several vendors and distributions shipped backported packages. Use one of those immediately, then plan the upgrade to a supported branch.</p>



<h3 class="wp-block-heading">Do I need to restart anything after updating PHP?</h3>



<p class="wp-block-paragraph">Yes. Restart every PHP-FPM pool, Apache if you use mod_php, LiteSpeed, and long-running PHP workers. Until they restart, they keep running the vulnerable code already loaded in memory.</p>



<h3 class="wp-block-heading">Should I rotate passwords and keys?</h3>



<p class="wp-block-paragraph">If an internet-facing server ran an unpatched build after details went public, it is a sensible precaution. Researchers showed worker memory could be read remotely, and that memory can hold credentials from other requests.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">The PHP2Shell vulnerability lives below your application, so the only real fix is a patched PHP build that is actually running. Everything else in this post is either a way to get there or a way to buy time. If you remember one thing, make it this: verify the fix by CVE in the package, and verify the process by its restart time. The version string will mislead you in both directions.</p>



<h2 class="wp-block-heading">Need help patching PHP2Shell across your servers?</h2>



<p class="wp-block-paragraph">If you have more PHP servers than certainty, I can help you close this out properly:</p>



<ul class="wp-block-list">
<li>Inventory every PHP install across your VPS, cPanel, DirectAdmin and container hosts, with branch and package source.</li>



<li>Patch each one through the right channel and prove the fix by CVE and process restart time.</li>



<li>Apply and later roll back the interim mitigations safely, with a memory boundary in place.</li>



<li>Rebuild and redeploy PHP container images, including pinned-digest pipelines.</li>



<li>Review logs for the published indicators and plan secret rotation without breaking production.</li>



<li>Plan the move off end-of-life PHP branches and off mod_php.</li>
</ul>



<p class="wp-block-paragraph">Send me the output of <code>php -v</code>, your package changelog check, and how PHP runs on the server (FPM, mod_php or LiteSpeed), and I&#8217;ll tell you where you stand.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/system-administration/php2shell-vulnerability-fix/">PHP2Shell Vulnerability: What Happened, How to Patch, and How to Prove It Worked</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Build a CRM Knowledge Assistant on Amazon Bedrock Without Leaking Salesforce Records</title>
		<link>https://john-nessime.com/blog/devops/salesforce-crm-knowledge-assistant-bedrock/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 23 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[AgentCore]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon Kendra]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Bedrock Knowledge Bases]]></category>
		<category><![CDATA[Connected App]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Embeddings]]></category>
		<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Metadata Filtering]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[OpenSearch Serverless]]></category>
		<category><![CDATA[RAG]]></category>
		<category><![CDATA[Row-Level Security]]></category>
		<category><![CDATA[Salesforce]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Vector Database]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=243</guid>

					<description><![CDATA[<p>Connecting Salesforce to Amazon Bedrock Knowledge Bases takes an afternoon. Making sure the assistant does not read people records they were never allowed to see takes longer. A practical walkthrough of the connector, the permission model it quietly drops, and the two architectures that actually hold up.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-crm-knowledge-assistant-bedrock/">Build a CRM Knowledge Assistant on Amazon Bedrock Without Leaking Salesforce Records</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A support lead types a question into the assistant you demoed last week: &#8220;What&#8217;s the renewal value on the Northwind account?&#8221; It answers in about two seconds, correctly, with a citation back to the source record. Everyone in the room is pleased.</p>



<p class="wp-block-paragraph">Then someone points out that this particular support lead has never had access to Opportunity records. Not through her profile, not through the role hierarchy, not through a sharing rule. And the assistant just read one out to her.</p>



<p class="wp-block-paragraph">That is the failure that kills most CRM assistant projects, and it almost never shows up in testing, because you test as an admin. Everything you ask works. Everything you ask is supposed to work. The gap only opens when a real user with a real, narrow profile starts asking questions.</p>



<p class="wp-block-paragraph">This post walks through building a CRM knowledge assistant on Salesforce and Amazon Bedrock Knowledge Bases: which of the two knowledge base paths you are actually allowed to use, how the Salesforce connector authenticates, what it does and does not carry across from your CRM, and the two architectures that survive a real permission model. The wiring is the easy part. The scoping is the work.</p>



<h2 class="wp-block-heading">The three moving parts</h2>



<p class="wp-block-paragraph">Strip away the marketing and a CRM knowledge assistant is three things bolted together:</p>



<ul class="wp-block-list">
<li><strong>Ingestion</strong> — something pulls Cases, Knowledge articles, Opportunities and Accounts out of Salesforce on a schedule, chunks them, embeds them, and writes the vectors somewhere.</li>

<li><strong>Retrieval</strong> — a query comes in, gets embedded, and the closest chunks come back. This is where filtering lives, and it is the security surface.</li>

<li><strong>Generation</strong> — a foundation model reads those chunks and writes an answer with citations.</li>
</ul>



<p class="wp-block-paragraph">Generation is the part everyone stares at and the part you should worry about least. It is a model choice and a prompt. Retrieval is where the assistant becomes either trustworthy or a liability.</p>



<h2 class="wp-block-heading">Decision one: which knowledge base you are allowed to use</h2>



<p class="wp-block-paragraph">Bedrock now has two shapes of knowledge base, and this decision gets made for you the moment you say &#8220;Salesforce&#8221;.</p>



<p class="wp-block-paragraph"><strong>Bedrock Managed Knowledge Base</strong> is the newer, opinionated one. AWS manages the vector store, picks the embedding and reranking models by default, handles parsing, and exposes agentic multi-hop retrieval. It ships with a small set of first-party connectors: Amazon S3, SharePoint, Confluence, Google Drive, OneDrive, Web Crawler, and a custom ingestion path. Salesforce is not on that list.</p>



<p class="wp-block-paragraph"><strong>Customer-managed knowledge bases</strong> are the older shape. You bring the vector store, you pick the embedding model, you own more of the pipeline. This is where the Salesforce connector lives, and AWS still flags that connector as a preview release subject to change. It also has a hard constraint worth reading twice: with a Salesforce data source, Amazon OpenSearch Serverless is currently the only supported vector store. If your organisation has standardised on Aurora pgvector or something else, that standard does not apply here.</p>



<p class="wp-block-paragraph">So the fork is real. Either you use the preview Salesforce connector on the customer-managed path, or you land Salesforce data in S3 yourself and use the managed path. I will come back to why the second option is often the right call, because it is not just a workaround.</p>



<p class="wp-block-paragraph">One piece of context that matters if you are inheriting an older design: Amazon Kendra entered maintenance mode on 30 June 2026 and stopped accepting new customers on 30 July 2026. Kendra was the standard answer to &#8220;search my Salesforce org&#8221; for years, and AWS now points people at Bedrock Knowledge Bases instead. Existing indexes keep running and no end-of-service date has been set, but Kendra will not gain features. If a proposal on your desk starts with a Kendra index, that is worth raising before anyone writes code.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The permission model is the whole problem</h2>



<p class="wp-block-paragraph">Salesforce spends enormous effort on who can see what. Profiles, permission sets, the role hierarchy, org-wide defaults, criteria-based sharing rules, field-level security, restriction rules. A given Opportunity is visible to a specific, computed set of users, and that computation happens on every query.</p>



<p class="wp-block-paragraph">None of that survives ingestion.</p>



<p class="wp-block-paragraph">The AWS documentation is blunt about it, and this is the single most important sentence in the whole feature: everything you sync from your data source becomes available to anyone holding <code>bedrock:Retrieve</code> on that knowledge base, including content that had controlled permissions at the source. Three layers of Salesforce scoping collapse into one IAM action.</p>



<p class="wp-block-paragraph">Worse, the connector authenticates as a single identity. You configure a Connected App using the OAuth 2.0 client credentials flow, and that flow runs as one named user. Whatever that user can see, the crawler ingests. Pick a System Administrator to &#8220;make sure the sync works&#8221; and you have just copied the entire org into a flat index that your whole application tier can read.</p>



<h3 class="wp-block-heading">Why this is invisible until it isn&#8217;t</h3>



<p class="wp-block-paragraph">Nothing errors. There is no warning in the console, no failed sync, no denied API call. Retrieval returns chunks with good similarity scores and the model writes a fluent, cited answer. The only signal that something is wrong is a human recognising a record they should not be looking at, and that human is usually a customer-facing employee, not you.</p>



<p class="wp-block-paragraph">Treat the knowledge base as a published dataset, not as a view over Salesforce. Ask the question you would ask about an S3 bucket: if every row of this landed in front of every user of the app, who gets hurt?</p>



<h3 class="wp-block-heading">The two architectures that actually hold up</h3>



<p class="wp-block-paragraph"><strong>Option A: ingest only what everyone is allowed to see.</strong> Run the Connected App as a deliberately low-privilege integration user whose profile grants read access to Knowledge articles, published solutions, product data and price books, and nothing else. Exclude Contact, Lead, Case and Opportunity entirely. You get a genuinely useful support and enablement assistant, the permission question disappears, and you can ship it without a security review turning into a project.</p>



<p class="wp-block-paragraph">This is the one I reach for first. Most of the value people want from a CRM knowledge assistant is &#8220;what do we know about this product, this policy, this recurring problem&#8221;, and none of that needs record-level data.</p>



<p class="wp-block-paragraph"><strong>Option B: export to S3 with your own access metadata.</strong> If you genuinely need per-record answers, stop using the connector. Pull the objects you need through the Salesforce API or a scheduled extract, write each record to S3 as a document, and write a sidecar metadata file next to it carrying the access attribute you will filter on. Then point a knowledge base at the bucket and apply an explicit metadata filter on every retrieval, derived from the caller&#8217;s identity rather than from the query text.</p>



<p class="wp-block-paragraph">The cost is honest: you now own an extraction pipeline, a metadata schema, and the job of keeping the access attribute correct when a record changes owner. That is real work. The benefit is that you can point at exactly where the boundary is enforced and test it.</p>



<p class="wp-block-paragraph">What you cannot do is enforce the boundary in the prompt. &#8220;Only answer about accounts the user owns&#8221; is a suggestion to a language model, not an access control. Retrieval already happened by then.</p>



<h2 class="wp-block-heading">Wiring the Salesforce connector</h2>



<p class="wp-block-paragraph">Assuming you have chosen Option A and want the connector, here is the shape of it.</p>



<h3 class="wp-block-heading">On the Salesforce side</h3>



<ol class="wp-block-list">
<li>Create a Connected App in your org and enable the OAuth 2.0 client credentials flow.</li>

<li>Under the client credentials flow settings, set the <strong>Run As</strong> field to your integration user. This is the step people skip, and the sync fails without it. It is also the step that decides your entire blast radius, so pick that user before you pick anything else.</li>

<li>Copy the consumer key and consumer secret from the app&#8217;s OAuth settings.</li>

<li>Note your instance URL.</li>
</ol>



<h3 class="wp-block-heading">On the AWS side</h3>



<p class="wp-block-paragraph">Store the credentials in AWS Secrets Manager. The connector looks for three specific keys, and it will not fall back to alternatives if you name them differently:</p>



<pre class="wp-block-code"><code>{
  "consumerKey": "your Connected App client ID",
  "consumerSecret": "your Connected App client secret",
  "authenticationUrl": "https://yourcompany.my.salesforce.com/"
}</code></pre>



<p class="wp-block-paragraph">The secret has to live in the same AWS region as the knowledge base. Cross-region reads are not supported here, and the resulting error is generic enough to waste an hour. Do not reuse this secret for any other data source; rotate it on the same schedule as your other integration credentials.</p>



<h3 class="wp-block-heading">Creating the data source</h3>



<p class="wp-block-paragraph">The data source configuration is a JSON document. The interesting part is <code>patternObjectFilter</code>, which is how you keep the crawler away from objects you never wanted:</p>



<pre class="wp-block-code"><code>{
  "salesforceConfiguration": {
    "sourceConfiguration": {
      "hostUrl": "https://yourcompany.my.salesforce.com/",
      "authType": "OAUTH2_CLIENT_CREDENTIALS",
      "credentialsSecretArn": "arn:aws:secretsmanager:REGION:ACCOUNT:secret:NAME"
    },
    "crawlerConfiguration": {
      "filterConfiguration": {
        "type": "PATTERN",
        "patternObjectFilter": {
          "filters": [
            {
              "objectType": "Knowledge__kav",
              "inclusionFilters": [".*"]
            },
            {
              "objectType": "Case",
              "exclusionFilters": [".*"]
            }
          ]
        }
      }
    }
  },
  "type": "SALESFORCE"
}</code></pre>



<p class="wp-block-paragraph">Two things to internalise about these filters. First, they are regular expressions matched against a per-object identifier, not against record contents: for <code>Knowledge__kav</code> that is the article title, for <code>Case</code> the case number, for <code>Attachment</code> the file name. Second, when an inclusion and an exclusion filter both match the same document, exclusion wins. That precedence is what makes an allow-list pattern safe to write.</p>



<p class="wp-block-paragraph">Then create the data source and attach it to an existing knowledge base:</p>



<pre class="wp-block-code"><code>aws bedrock create-data-source 
  --name "salesforce-knowledge" 
  --knowledge-base-id "YOUR_KB_ID" 
  --data-source-configuration file://salesforce-connector-config.json 
  --data-deletion-policy "DELETE" 
  --vector-ingestion-configuration '{"chunkingConfiguration":[{"chunkingStrategy":"FIXED_SIZE","fixedSizeChunkingConfiguration":[{"maxTokens":"300","overlapPercentage":"10"}]}]}'</code></pre>



<p class="wp-block-paragraph">A note on the command namespace: the agent-plane operations have historically lived under a separate CLI command group from the model-inference ones, and this has shifted between CLI versions. Run <code>aws bedrock help</code> against your installed version before assuming. The JSON payload is the stable part.</p>



<p class="wp-block-paragraph"><code>--data-deletion-policy DELETE</code> means the vectors get removed from the store when you delete the data source. The alternative retains them. Choose <code>DELETE</code> unless you have a specific reason not to, because orphaned vectors in an OpenSearch Serverless collection are both a cost line and a surprise waiting for whoever inherits this.</p>



<p class="wp-block-paragraph">After that, kick off a sync with <code>StartIngestionJob</code>, or hit <strong>Sync</strong> in the console. The connector does incremental syncs after the first run: added, updated and deleted content only. The first run crawls everything.</p>



<h2 class="wp-block-heading">Chunking is a one-way door</h2>



<p class="wp-block-paragraph">You cannot change the chunking strategy after connecting the data source. Not tune it, not swap it. You delete the data source and start over, which means re-crawling and re-embedding everything.</p>



<p class="wp-block-paragraph">So spend twenty minutes here rather than five. CRM content is not uniform. Knowledge articles are structured prose and behave like documentation. Case comment threads are short, conversational, and lose their meaning when split mid-exchange. Opportunity descriptions are often two sentences. A fixed-size chunk that suits one of these will mangle another.</p>



<p class="wp-block-paragraph">The practical move is to stop trying to find one strategy that fits everything. Create separate data sources for materially different content shapes, each with its own chunking configuration, all pointing at the same knowledge base. It costs you a bit of configuration and saves you a full re-index later.</p>



<p class="wp-block-paragraph">One more constraint: the Salesforce connector does not handle multimodal content. Tables, charts and diagrams embedded in your Knowledge articles are not going to be understood. If your best troubleshooting articles are mostly screenshots, this connector will quietly ingest the surrounding text and nothing else, and your retrieval quality will be worse than the article count suggests.</p>



<h2 class="wp-block-heading">Retrieval, filtering, and one trap</h2>



<p class="wp-block-paragraph">Bedrock gives you two kinds of metadata filter at query time, and confusing them is a security bug.</p>



<p class="wp-block-paragraph"><strong>Explicit filters</strong> are the ones you construct in code and pass in the retrieval configuration. Your application decides them from the authenticated caller. They are deterministic.</p>



<pre class="wp-block-code"><code>{
  "retrievalQuery": { "text": "renewal value on the Northwind account" },
  "retrievalConfiguration": {
    "vectorSearchConfiguration": {
      "numberOfResults": 8,
      "filter": {
        "andAll": [
          { "equals": { "key": "visibility_group", "value": "emea-enterprise" } },
          { "equals": { "key": "record_class", "value": "internal" } }
        ]
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph"><strong>Implicit filters</strong> are the other kind. You declare which metadata attributes exist and Bedrock uses a model to infer filters from the natural-language query. Someone asks about last quarter, it infers a date filter. This is a genuinely nice relevance feature and it is not an access control. A model inferring a filter from user-supplied text is a model that can be talked out of it. Use implicit filters to improve answers; use explicit filters to enforce boundaries.</p>



<p class="wp-block-paragraph">The trap: filters on the Salesforce connector operate on auto-detected fields from your CRM records. You cannot inject your own <code>visibility_group</code> attribute at ingestion. That capability comes from sidecar metadata files, which is an S3 data source feature. This is the concrete reason Option B routes through a bucket. If per-user scoping is a requirement, the connector cannot give it to you no matter how carefully you configure it.</p>



<h2 class="wp-block-heading">Where the money goes</h2>



<p class="wp-block-paragraph">Rates change, so learn the meters instead of memorising numbers.</p>



<ul class="wp-block-list">
<li><strong>Embedding at ingestion</strong> — billed per token processed. A full re-crawl re-embeds everything, which is exactly why the chunking decision is expensive to reverse.</li>

<li><strong>Vector storage</strong> — on the customer-managed path this is your OpenSearch Serverless collection, billed on provisioned compute units and storage, with a floor whether or not anyone queries it. This is the line item that surprises people on a proof of concept that nobody uses. Managed Knowledge Base bills on indexed data size and retrieval count instead, with no idle collection sitting underneath.</li>

<li><strong>Generation</strong> — input and output tokens on whichever foundation model you chose. Retrieved chunks are input tokens, so <code>numberOfResults</code> is a direct cost lever and worth tuning down until quality actually degrades.</li>

<li><strong>Everything around it</strong> — the API layer, the chat frontend, and the observability stack. A small VPS from a provider like InterServer or a Lambda behind API Gateway both work; the choice barely matters next to the Bedrock line. Push retrieval latency and citation counts into Grafana Cloud or Datadog early, because &#8220;the assistant feels worse this week&#8221; is otherwise unfalsifiable.</li>
</ul>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Sync fails immediately with an authentication error.</strong> Check the Run As user on the Connected App&#8217;s client credentials flow first. An empty Run As field is the most common cause. Then check that the secret keys are spelled exactly <code>consumerKey</code>, <code>consumerSecret</code>, <code>authenticationUrl</code>, and that the secret is in the knowledge base&#8217;s region.</li>

<li><strong>Sync succeeds but the document count is far lower than expected.</strong> Usually the integration user&#8217;s profile, not the filters. Query the object as that user in Salesforce and count. Also check the per-knowledge-base file count and file size quotas, which are separate limits and are silently enforced.</li>

<li><strong>Retrieval returns nothing for queries you know are covered.</strong> Test with no filter at all. If results appear, your filter keys do not match the auto-detected field names. If they do not, the content probably never made it past chunking; check whether those articles are mostly images.</li>

<li><strong>Deleted Salesforce records still turn up in answers.</strong> Incremental sync handles deletions, but only on the next sync. If you deleted something sensitive, run a sync immediately rather than waiting for the schedule.</li>

<li><strong>Answers are confidently wrong about numbers.</strong> Fixed-size chunking split a record away from its label. Retrieve the raw chunks for that query and read them. Nine times out of ten the problem is visible in the chunk, not in the model.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Running the Connected App as a System Administrator &#8220;just to get the sync working&#8221;. That decision is almost never revisited.</li>

<li>Assuming sharing rules and field-level security follow the data into the index. They do not.</li>

<li>Enforcing access in the system prompt. Retrieval has already happened by the time the model reads your instructions.</li>

<li>Ingesting Contact and Lead because they were there. Both carry personal data and both are usually irrelevant to the questions people actually ask.</li>

<li>Accepting default chunking, then discovering it is immutable.</li>

<li>Building on a preview connector without telling stakeholders it is preview. The configuration surface can change under you.</li>

<li>Testing only as an admin. Every acceptance test should run as the narrowest real profile you have.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Decide the integration user&#8217;s profile before you touch AWS. It defines the security boundary more than any IAM policy will.</li>

<li>Start with Knowledge articles only. Ship that, get real queries, then argue about whether Opportunity data is worth the architecture it requires.</li>

<li>Write filters as allow-lists. Exclusion beats inclusion, so an allow-list fails closed.</li>

<li>Split content shapes across separate data sources so chunking suits each one.</li>

<li>Keep a fixed evaluation set of about fifty real questions with known-good answers, and re-run it after every configuration change.</li>

<li>Log the retrieved chunk IDs alongside every generated answer. When someone reports a bad answer months from now, that log is the only thing that makes it debuggable.</li>

<li>Treat <code>bedrock:Retrieve</code> as a privileged permission and scope it to the specific knowledge base ARN, not to a wildcard.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does the Bedrock Salesforce connector respect sharing rules?</h3>



<p class="wp-block-paragraph">No. The crawler authenticates as one user and everything it can read becomes retrievable by anyone with <code>bedrock:Retrieve</code> on the knowledge base. Per-user scoping has to be built at the retrieval layer with explicit metadata filters, which in practice means routing through S3 rather than using the connector.</p>



<h3 class="wp-block-heading">Can I use Aurora or pgvector instead of OpenSearch Serverless?</h3>



<p class="wp-block-paragraph">Not with the Salesforce data source. AWS currently restricts that connector to Amazon OpenSearch Serverless. If your vector store is non-negotiable, extract to S3 and use a data source that supports your chosen store.</p>



<h3 class="wp-block-heading">Should I use Agentforce instead of building this on Bedrock?</h3>



<p class="wp-block-paragraph">If your questions are answerable entirely from Salesforce data and your users already live in Salesforce, Agentforce is the shorter path and it operates inside the platform&#8217;s existing permission model, which removes the hardest problem in this post. Bedrock earns its complexity when the assistant has to reason across CRM data plus sources that are not in Salesforce, or when you need a specific foundation model, or when the data must stay inside your AWS account. Cost profiles differ substantially and are worth modelling on your real query volume before committing.</p>



<h3 class="wp-block-heading">How current will the assistant&#8217;s answers be?</h3>



<p class="wp-block-paragraph">As current as your last sync. Ingestion is scheduled, not live. For rapidly changing fields such as case status or pipeline stage, do not answer from the index at all: have the assistant call the Salesforce API for the live value and use the knowledge base only for the surrounding context.</p>



<h3 class="wp-block-heading">What happens to the connector when it leaves preview?</h3>



<p class="wp-block-paragraph">Unknown, which is the point of the label. Preview means the configuration surface, supported objects and behaviour can change without the compatibility guarantees you would expect from a GA service. Keep your data source configuration in version control so a re-create is a rerun rather than an archaeology exercise.</p>



<h3 class="wp-block-heading">Is a CRM knowledge assistant worth building if we already have Salesforce search?</h3>



<p class="wp-block-paragraph">It depends on the question shape. Search is better when people know what record they want. A CRM knowledge assistant wins on synthesis: &#8220;what usually causes this error for enterprise customers&#8221; pulls from twenty articles and cases at once, which no search result page does well. If your users mostly navigate to known records, you will not get much return.</p>



<h3 class="wp-block-heading">Do I need Bedrock AgentCore for this?</h3>



<p class="wp-block-paragraph">Not for retrieval and generation. AgentCore matters when you want the assistant to take actions, chain tools, or be exposed to agent frameworks over MCP. A read-only question-answering assistant works fine against the knowledge base APIs directly, and skipping AgentCore removes a layer you would otherwise have to operate.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to remember</h2>



<p class="wp-block-paragraph">Connecting Salesforce to Amazon Bedrock takes an afternoon. The connector works, the syncs run, the answers are good. That speed is exactly what makes this dangerous, because nothing in the setup path forces you to think about who the answers are for.</p>



<p class="wp-block-paragraph">A CRM knowledge assistant is a publishing decision wearing an engineering costume. Every record you ingest, you are publishing to everyone who can call retrieve. Decide what that audience is allowed to see, encode it in the integration user&#8217;s profile and in your inclusion filters, and test every question as your narrowest real user. Do that first and the rest of the build is routine.</p>



<h2 class="wp-block-heading">Need help getting this into production?</h2>



<p class="wp-block-paragraph">I work with teams building retrieval systems on AWS, usually at the point where a working prototype has to survive a security review. Things I can help with on a CRM knowledge assistant:</p>



<ul class="wp-block-list">
<li>Reviewing an existing Bedrock knowledge base for records that should never have been ingested, and scoping the integration user&#8217;s Salesforce profile to match.</li>

<li>Designing the S3 extraction path with sidecar access metadata when per-user filtering is a hard requirement.</li>

<li>Choosing between the managed and customer-managed knowledge base paths, with the OpenSearch Serverless cost floor modelled against your actual query volume.</li>

<li>Building the chunking and data source split before it becomes an immutable mistake, plus a repeatable evaluation set to measure changes against.</li>

<li>Terraform or OpenTofu for the whole stack: knowledge base, data sources, Secrets Manager, IAM, and the vector store.</li>

<li>Retrieval observability so you can tell whether answer quality moved, and why.</li>

<li>Migration planning for teams currently on a Kendra index.</li>
</ul>



<p class="wp-block-paragraph">If you have a data source configuration, a failing sync, or a retrieval result that looks wrong, send it over and we can work out what it is doing.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-crm-knowledge-assistant-bedrock/">Build a CRM Knowledge Assistant on Amazon Bedrock Without Leaking Salesforce Records</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</title>
		<link>https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 17 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Cursor Pagination]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Data Quality]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[GraphQL]]></category>
		<category><![CDATA[Idempotency]]></category>
		<category><![CDATA[Incremental Sync]]></category>
		<category><![CDATA[Orchestration]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Schema Drift]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Shopify]]></category>
		<category><![CDATA[Step Functions]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=215</guid>

					<description><![CDATA[<p>A GraphQL source can hand you a 200 OK, a populated data block, and a quietly broken column in the same response. Here is how to build a GraphQL data ingestion pipeline on AWS that catches partial errors, respects cost-based rate limits, resumes cleanly from a cursor, and notices when the schema moves under you.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/">Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The row counts looked fine. That was the problem.</p>



<p class="wp-block-paragraph">A nightly sync from a partner&#8217;s GraphQL API had run for weeks without a single failed execution. Green across the board in the console. Then someone asked why one column had been empty since the previous Tuesday. Not sparse. Empty. Every row, every night, for six nights.</p>



<p class="wp-block-paragraph">The API had started returning a field-level error on that one field. Status code 200. A populated <code>data</code> block. An <code>errors</code> array sitting right next to it. The fetcher parsed <code>data</code>, wrote the records to S3, exited zero, and the orchestrator logged a success. Nothing in the stack was wrong, exactly. Everything in the stack was checking the wrong thing.</p>



<p class="wp-block-paragraph">This post covers how to build a GraphQL data ingestion pipeline on AWS that fails honestly. It is organised by failure family rather than by service, because the AWS wiring is the easy part. The hard part is that GraphQL breaks four assumptions that REST ingestion quietly relies on, and every one of them fails silently by default.</p>



<h2 class="wp-block-heading">Why GraphQL ingestion breaks the habits REST taught you</h2>



<p class="wp-block-paragraph">Most ETL code carries three inherited reflexes from years of REST work: the status code tells you whether it worked, the response shape is the server&#8217;s business, and rate limits are counted in requests. GraphQL invalidates all three.</p>



<ul class="wp-block-list">
<li><strong>The status code is not the contract.</strong> The GraphQL over HTTP specification says a server should return 200 even when field errors are raised during execution, because a partial response is still a successful execution.</li>

<li><strong>You own the response shape.</strong> The selection set is written by you, which means schema drift shows up as a query that stops being valid, not as a payload that changes underneath you.</li>

<li><strong>Cost is query-shaped, not endpoint-shaped.</strong> One endpoint can cost almost nothing or blow your entire budget depending on how deeply you nested it.</li>
</ul>



<p class="wp-block-paragraph">Each of those becomes a distinct failure family. Let&#8217;s take them in the order they bite.</p>



<h2 class="wp-block-heading">Failure one: the 200 OK that quietly ate a column</h2>



<p class="wp-block-paragraph">This is the one that costs the most and shows up the latest. GraphQL splits errors into two categories, and they behave completely differently on the wire.</p>



<ul class="wp-block-list">
<li><strong>Request errors</strong> happen before execution starts: syntax errors, validation failures, bad variable coercion. No <code>data</code> key is returned at all. Nothing executed.</li>

<li><strong>Field errors</strong> happen during execution: a resolver threw, a downstream service timed out, a permission check failed. Execution continues, the failed field goes null, and you get <code>data</code> and <code>errors</code> together.</li>
</ul>



<p class="wp-block-paragraph">Field errors are the dangerous ones for ingestion, because they look exactly like success to anything that only inspects the transport layer. Worse, GraphQL&#8217;s null propagation rules mean a single failing non-null field bubbles up to the nearest nullable ancestor, so one broken resolver can hollow out an entire nested object while the surrounding rows land normally.</p>



<p class="wp-block-paragraph">The fix is to make the response body the contract. Classify before you commit anything:</p>



<pre class="wp-block-code"><code>import json, urllib.request

class RequestError(Exception): pass
class FieldError(Exception): pass

def graphql(endpoint, token, query, variables):
    body = json.dumps({"query": query, "variables": variables}).encode()
    req = urllib.request.Request(
        endpoint,
        data=body,
        headers={
            "Content-Type": "application/json",
            "Accept": "application/graphql-response+json, application/json",
            "Authorization": f"Bearer {token}",
        },
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        payload = json.loads(resp.read())

    # The status code told us nothing. The body is the contract.
    if "data" not in payload:
        # Nothing executed. Query or auth problem. Do not retry blindly.
        raise RequestError(payload.get("errors"))

    if payload.get("errors"):
        # Partial response. data is populated but incomplete.
        raise FieldError(payload["errors"])

    return payload["data"]</code></pre>



<p class="wp-block-paragraph">Two things worth explaining. The <code>Accept</code> header asks for <code>application/graphql-response+json</code>, the media type the GraphQL over HTTP spec recommends, which lets a compliant server return a real 4xx for pre-execution failures instead of burying them in a 200. Servers that don&#8217;t support it fall back to <code>application/json</code>, so including both is free.</p>



<p class="wp-block-paragraph">The second is the split between the two exception types. They need different handling. A request error is a bug in your query or a credential problem, and retrying it just burns budget on the same failure. A field error is transient often enough to be worth retrying, but never worth committing.</p>



<h3 class="wp-block-heading">Deciding what &#8220;partial&#8221; means for your data</h3>



<p class="wp-block-paragraph">Failing the whole page on any error is the safe default and the one I reach for first. But it&#8217;s not always right. If you are pulling a hundred fields and one optional enrichment field is flaky, refusing every page forever is its own outage.</p>



<p class="wp-block-paragraph">The workable middle ground is to classify by path. Every error object carries a <code>path</code> array pointing at the exact field that failed. Maintain a small allowlist of paths that are permitted to be null, fail the page on anything outside it, and emit the error count as a CloudWatch metric so a rising trend is visible before someone in finance notices.</p>



<h2 class="wp-block-heading">Failure two: rate limits that never send a 429</h2>



<p class="wp-block-paragraph">Your retry logic almost certainly triggers on 429 and 5xx. Several major GraphQL APIs will throttle you without ever sending one.</p>



<p class="wp-block-paragraph">GitHub&#8217;s GraphQL API is explicit about this: exceeding the primary rate limit returns a 200 response with an error message in the body and <code>x-ratelimit-remaining</code> set to zero. Your Step Functions retrier sees a success. Your Lambda sees no data. The run completes, having ingested nothing, and reports green.</p>



<p class="wp-block-paragraph">The other half of the problem is that request counting is the wrong unit. Shopify&#8217;s GraphQL Admin API and several others price queries by <em>calculated cost</em>, using a leaky bucket. The server statically analyses your query before executing it, charges the bucket the worst-case <em>requested</em> cost, executes, then refunds the difference between requested and <em>actual</em> cost. That has a consequence people miss: a query asking for the maximum connection size gets charged as if every page were full, even when the last page returns three records.</p>



<p class="wp-block-paragraph">The good news is that these APIs tell you the state of the bucket on every response, under the <code>extensions</code> key:</p>



<pre class="wp-block-code"><code>{
  "extensions": {
    "cost": {
      "requestedQueryCost": 72,
      "actualQueryCost": 38,
      "throttleStatus": {
        "maximumAvailable": 1000.0,
        "currentlyAvailable": 962.0,
        "restoreRate": 100.0
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Read those numbers, never hardcode them. Bucket size and restore rate vary by plan tier, and a query that works fine against a large merchant&#8217;s store will fail instantly against a small one. The pattern that holds up is a pre-flight check: before dispatching, compare your estimated cost against <code>currentlyAvailable</code>, and if you&#8217;re short, sleep for the shortfall divided by <code>restoreRate</code>.</p>



<p class="wp-block-paragraph">On APIs that expose their own budget as a queryable field, ask for it in the same round trip rather than spending a second call to find out:</p>



<pre class="wp-block-code"><code>query BudgetCheck {
  rateLimit {
    limit
    cost
    remaining
    resetAt
  }
}</code></pre>



<p class="wp-block-paragraph">Two more things about concurrency. First, if you fan out across Lambda invocations, each worker&#8217;s local view of the bucket is wrong, because the bucket is shared. Multiple workers independently deciding they have headroom is the classic way to get throttled while every individual worker believes it&#8217;s behaving. Either centralise the token accounting or cap concurrency low enough that the arithmetic can&#8217;t go wrong.</p>



<p class="wp-block-paragraph">Second, if the source offers a bulk or async export operation, use it for backfills. Paginating fifty thousand records through the interactive endpoint burns the same budget your incremental syncs need, and it competes with itself.</p>



<h2 class="wp-block-heading">Failure three: cursors that don&#8217;t survive a restart</h2>



<p class="wp-block-paragraph">Most production GraphQL schemas follow the Relay cursor connections convention: <code>first</code> and <code>after</code> arguments, a <code>pageInfo</code> object with <code>hasNextPage</code> and <code>endCursor</code>, and edges carrying opaque per-item cursors.</p>



<pre class="wp-block-code"><code>query Orders($cursor: String) {
  orders(first: 100, after: $cursor) {
    pageInfo {
      hasNextPage
      endCursor
    }
    edges {
      cursor
      node {
        id
        updatedAt
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">The filter argument for incremental sync is not standardised. Some schemas take a query-string filter, some take a typed timestamp argument, some expose nothing at all. Run introspection against the schema and find out rather than guessing, because a wrong argument name is a request error and your run will fail before it starts.</p>



<p class="wp-block-paragraph">The failure here is subtle. Cursors are opaque and their validity is entirely up to the server. Some are stable positional pointers. Some encode a sort key and a snapshot identifier and expire. If a page-47 fetch fails and you retry an hour later with the stored cursor, you may get an error, or you may get results from a shifted window with a hole in the middle. The second outcome is worse, because it looks like it worked.</p>



<p class="wp-block-paragraph">Three habits make restarts safe:</p>



<ol class="wp-block-list">
<li><strong>Checkpoint after the write, not after the fetch.</strong> A cursor recorded before the page lands in S3 is a promise you can&#8217;t keep.</li>

<li><strong>Store a high watermark alongside the cursor.</strong> The cursor is your fast path. The watermark, taken from the newest record you actually committed, is your fallback when the cursor is rejected.</li>

<li><strong>Overlap the watermark deliberately.</strong> Restart a few minutes before the last committed timestamp and deduplicate on the primary key downstream. Re-reading a handful of rows is cheap. Missing rows that arrived during a clock skew is not.</li>
</ol>



<p class="wp-block-paragraph">A DynamoDB item per source stream is enough state for all of this:</p>



<pre class="wp-block-code"><code>{
  "source": "orders",
  "run_id": "01hq7k...",
  "cursor": "eyJsYXN0X2lkIjoxMjM0fQ==",
  "high_watermark": "&lt;updatedAt of newest committed record&gt;",
  "pages_committed": 47,
  "status": "IN_PROGRESS"
}</code></pre>



<p class="wp-block-paragraph">Overlap only works if downstream deduplication is real. Write raw pages with an ingest-time partition and a run identifier in the object key, then let the curated layer resolve duplicates by primary key and latest <code>updatedAt</code>. If you write straight into a merged table with no dedupe step, overlapping restarts will double-count and you&#8217;ll have traded silent loss for silent inflation.</p>



<h2 class="wp-block-heading">Failure four: the schema moved and nobody told you</h2>



<p class="wp-block-paragraph">GraphQL&#8217;s deprecation story is genuinely better than REST&#8217;s. Fields carry a <code>@deprecated</code> directive with a reason string, and introspection exposes it. The problem is that nothing forces you to look.</p>



<p class="wp-block-paragraph">The failure mode is a field that gets deprecated, stops being populated, then eventually gets removed. Stage one is invisible: your query still validates, the field returns null, and your warehouse fills with nulls. Stage two is loud but late: the query becomes invalid and the whole ingest breaks at 3am.</p>



<p class="wp-block-paragraph">Put an introspection diff in CI. Fetch the schema on a schedule, store it as an artifact, and diff against the last known good version. Fail the build on removals or type changes to fields your queries actually select, and warn on new deprecations. Tools in the Apollo and Hasura ecosystems ship schema-diff tooling for this, and rolling your own against the introspection result is not much work either. Either way, the point is that the check runs on a schedule rather than when the pager goes off.</p>



<p class="wp-block-paragraph">The complementary habit is to keep queries narrow. Every field you select is a field that can break you, a field that costs budget, and a field that widens your Glue table. Selecting a whole object because it was convenient during development is a liability you carry indefinitely.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Wiring the GraphQL data ingestion pipeline on AWS</h2>



<p class="wp-block-paragraph">With the failure families understood, the AWS assembly is mostly a set of trade-offs about where the fetch loop lives.</p>



<h3 class="wp-block-heading">Where the fetch loop runs</h3>



<p class="wp-block-paragraph">Step Functions can call an HTTPS endpoint directly with an HTTP Task, no Lambda involved. It uses an EventBridge connection to hold the credentials, which means your token lives in a Secrets Manager secret managed by EventBridge rather than in an environment variable:</p>



<pre class="wp-block-code"><code>"FetchPage": {
  "Type": "Task",
  "Resource": "arn:aws:states:::http:invoke",
  "Parameters": {
    "ApiEndpoint": "https://api.example.com/graphql",
    "Method": "POST",
    "InvocationConfig": {
      "ConnectionArn": "arn:aws:events:region:account-id:connection/name/id"
    },
    "Headers": { "Content-Type": "application/json" },
    "RequestBody": {
      "query.$": "$.query",
      "variables.$": "$.variables"
    }
  },
  "Retry": [{
    "ErrorEquals": [
      "States.Http.StatusCode.429",
      "States.Http.StatusCode.502",
      "States.Http.StatusCode.503",
      "States.Http.StatusCode.504"
    ],
    "IntervalSeconds": 1,
    "BackoffRate": 2,
    "MaxAttempts": 3,
    "JitterStrategy": "FULL"
  }],
  "Next": "InspectBody"
}</code></pre>



<p class="wp-block-paragraph">The execution role needs four permissions: <code>states:InvokeHTTPEndpoint</code>, <code>events:RetrieveConnectionCredentials</code>, and both <code>secretsmanager:GetSecretValue</code> and <code>secretsmanager:DescribeSecret</code> scoped to the EventBridge-managed secret.</p>



<p class="wp-block-paragraph">It&#8217;s a clean pattern, and I like it for small, high-value syncs. Two constraints decide whether it fits. HTTP Task requests time out after 60 seconds, which a deeply nested GraphQL query can exceed without being pathological. And Step Functions reads the response within its standard state payload ceiling of roughly 256 KB, so a page of a hundred rich records will not fit. Also note the retrier above: it fires on HTTP status codes, so it will not catch a throttle that arrives as a 200 with an error in the body. That check has to happen in the state after it.</p>



<p class="wp-block-paragraph">The alternative is a Lambda driver that paginates internally and writes pages straight to S3, returning only the cursor and a count to the state machine. This dodges both limits, gives you a real HTTP client with real backoff, and keeps large payloads out of the workflow&#8217;s state. It costs you per-page retry granularity, which you buy back with the checkpoint record.</p>



<p class="wp-block-paragraph">Distributed Map is worth mentioning for the fan-out case, such as one child execution per tenant or per shard. It gives you high concurrency and a per-item execution history, which is genuinely useful for debugging. Be aware of two things before committing: express child workflows cap at five minutes rather than Lambda&#8217;s fifteen, and practitioners have published benchmarks showing throughput well below what the headline concurrency numbers suggest for small, fast items. For a lot of ingestion work, SQS plus a DynamoDB job table remains the simpler answer.</p>



<p class="wp-block-paragraph">One case genuinely falls outside serverless: sources that require a long-lived session, a persistent websocket subscription, or a synchronisation window longer than fifteen minutes with no resumable cursor. A small always-on worker on a modest VPS from a provider like InterServer or Hetzner, feeding the same S3 landing zone, is less clever and more honest than contorting Lambda around the constraint.</p>



<h3 class="wp-block-heading">The landing zone</h3>



<p class="wp-block-paragraph">Land raw responses first, unmodified, including the <code>errors</code> and <code>extensions</code> keys. Storage is the cheapest thing in this architecture and the raw layer is what lets you reprocess after you discover a parsing bug six weeks later.</p>



<ul class="wp-block-list">
<li><strong>Partition by ingest time, not event time.</strong> Event-time partitioning requires reading the record before you know where to put it, which means you can&#8217;t write until you&#8217;ve parsed. Ingest-time keys let the raw write be dumb and fast.</li>

<li><strong>Put the run identifier in the key.</strong> When you need to unwind one bad run, being able to delete or ignore by prefix beats writing a surgical delete.</li>

<li><strong>Define Glue tables explicitly rather than crawling.</strong> You wrote the selection set, so you already know the schema. A crawler inferring types from JSON will surprise you the first time an optional field is absent from an entire page.</li>

<li><strong>Convert to columnar in the curated layer.</strong> Athena over raw JSON works and is fine for debugging. Parquet is what you want for anything anyone queries repeatedly.</li>
</ul>



<p class="wp-block-paragraph">If your source pushes rather than being polled, such as webhook deliveries carrying GraphQL payloads, Amazon Data Firehose (renamed from Kinesis Data Firehose, with APIs and IAM policies unchanged) will buffer, convert to Parquet and partition on the way to S3 without you running a consumer. For scheduled pull-based syncs it adds a hop you don&#8217;t need.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Runs succeed but a column is all null.</strong> Field errors. Pull a raw response from the landing zone and look at the <code>errors</code> array. The <code>path</code> tells you exactly which field.</li>

<li><strong>Runs succeed but ingest zero rows.</strong> Likely a throttle returned as a 200. Log the response body and the rate-limit headers on empty results, always.</li>

<li><strong>Intermittent failures partway through pagination.</strong> Cursor expiry. Fall back to the watermark and check whether the source documents a cursor lifetime.</li>

<li><strong>Query fails immediately with no <code>data</code> key.</strong> Request error. Validate against current introspection; something in the schema moved.</li>

<li><strong>Throttled despite staying under the documented limit.</strong> Either concurrent workers are sharing a bucket they each think they own, or you&#8217;re being charged requested cost on near-empty pages.</li>

<li><strong>HTTP Task fails on large pages.</strong> The response exceeds the state payload ceiling. Reduce page size or move the fetch into Lambda.</li>

<li><strong>Row counts drift upward over time.</strong> Overlap without deduplication. Check that the curated layer actually resolves on the primary key.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating a 200 as success and never reading the body.</li>

<li>Retrying request errors. They are deterministic; retrying just spends budget to fail identically.</li>

<li>Hardcoding bucket sizes and restore rates that differ per plan tier and change without notice.</li>

<li>Checkpointing the cursor before the data is durable.</li>

<li>Requesting the maximum page size on every call while being charged the worst-case cost for it.</li>

<li>Selecting entire objects because it was easier during development.</li>

<li>Alerting only on execution failure, so a job that succeeds while ingesting nothing never pages anyone.</li>

<li>Overlapping restart windows with no downstream deduplication.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Classify every response into clean, partial, or request-failed before anything downstream sees it.</li>

<li>Persist raw responses with <code>errors</code> and <code>extensions</code> intact, then parse from the raw layer.</li>

<li>Alarm on row-count deltas and partial-error rate, not just on execution status.</li>

<li>Read the throttle state the API hands you and back off before you hit the wall, not after.</li>

<li>Keep both a cursor and a watermark, and know which one is authoritative on restart.</li>

<li>Diff the schema on a schedule so deprecations reach you before they reach your data.</li>

<li>Hold credentials in Secrets Manager or an EventBridge connection, never in a Lambda environment variable.</li>

<li>Trace the pipeline end to end. CloudWatch covers the basics; Grafana or Datadog earn their keep once you&#8217;re correlating child executions across a fan-out.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can Step Functions call a GraphQL API without a Lambda function?</h3>



<p class="wp-block-paragraph">Yes. An HTTP Task using the <code>arn:aws:states:::http:invoke</code> resource posts your query directly, with an EventBridge connection handling authentication. It works well for small pages. The 60-second request timeout and the state payload size ceiling are what push most real ingestion work back into Lambda.</p>



<h3 class="wp-block-heading">How do I detect GraphQL errors when the status code is always 200?</h3>



<p class="wp-block-paragraph">Inspect the body. If there&#8217;s no <code>data</code> key, nothing executed and you have a request error. If <code>data</code> is present alongside a non-empty <code>errors</code> array, you have a partial response and the data is incomplete. Only a response with <code>data</code> and no errors is unambiguously clean.</p>



<h3 class="wp-block-heading">Does AWS AppSync fit into this?</h3>



<p class="wp-block-paragraph">Only if you&#8217;re on the serving side. AppSync is a managed GraphQL API for exposing your own data, so it&#8217;s what you&#8217;d reach for if you wanted clients to push events in via mutations. For pulling data out of somebody else&#8217;s GraphQL API, AppSync isn&#8217;t part of the picture.</p>



<h3 class="wp-block-heading">Should I store raw GraphQL responses or parsed records in S3?</h3>



<p class="wp-block-paragraph">Both, in separate layers. Raw responses are your audit trail and your reprocessing path, and they&#8217;re the only way to answer &#8220;was this field null in the source, or did my parser drop it?&#8221; weeks after the fact. Parsed and columnar output goes in a curated prefix that analysts query.</p>



<h3 class="wp-block-heading">How do I run an incremental sync from a GraphQL API?</h3>



<p class="wp-block-paragraph">Find the schema&#8217;s filter argument for a modification timestamp using introspection, pass your stored watermark into it, and page forward with cursors. Overlap the window slightly on each run and deduplicate on the primary key downstream. If the schema exposes no such filter, you&#8217;re limited to full pulls or webhook-driven change capture.</p>



<h3 class="wp-block-heading">Why am I throttled when I&#8217;m making very few requests?</h3>



<p class="wp-block-paragraph">Cost-based APIs charge for query complexity, not call count. A single deeply nested query pulling a connection inside a connection can consume a large fraction of the bucket by itself. Read the cost figures the API returns and flatten the query if the requested cost is consistently much higher than the actual cost.</p>



<h3 class="wp-block-heading">Is GraphQL a worse ingestion source than REST?</h3>



<p class="wp-block-paragraph">Different, not worse. You control the payload, which cuts transfer and lets you avoid the N+1 fan-out REST forces on nested data. Deprecation metadata is machine-readable, which REST rarely offers. The cost is that error signalling and rate limiting both moved into the body, so naive clients fail silently instead of loudly.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A GraphQL data ingestion pipeline on AWS fails differently from a REST one, and the difference is almost entirely about where the truth lives. In REST, the transport tells you what happened. In GraphQL, the transport tells you the request arrived, and the body tells you what happened. Every failure family in this post traces back to that: partial errors hidden behind a 200, throttles hidden behind a 200, cursors that fail by returning the wrong rows rather than an error.</p>



<p class="wp-block-paragraph">So build the pipeline so that a green run means something. Parse before you commit, alarm on row deltas rather than exit codes, and keep the raw responses so you can prove what the source actually said. Everything else is ordinary AWS plumbing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help with a GraphQL ingestion pipeline?</h2>



<p class="wp-block-paragraph">This is the kind of work I take on regularly. Specifically:</p>



<ul class="wp-block-list">
<li>Auditing an existing GraphQL sync for silent partial-error loss and telling you which columns are already affected</li>

<li>Building the fetch, checkpoint and land loop in Step Functions and Lambda with cursors that survive restarts</li>

<li>Fitting a cost-aware throttle to whatever budget model your source actually uses, including multi-worker coordination</li>

<li>Designing the raw and curated S3 layout, Glue table definitions and Athena partitioning so queries stay cheap</li>

<li>Adding introspection diffing to CI so schema deprecations surface in a pull request, not in a dashboard</li>

<li>Wiring CloudWatch metrics and alarms on row-count deltas and partial-error rates, so a green run is actually green</li>
</ul>



<p class="wp-block-paragraph">Send me a run log, a state machine definition, or a raw response with an <code>errors</code> array in it, and I&#8217;ll tell you what it&#8217;s costing you.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/graphql-data-ingestion-pipeline-aws/">Building a GraphQL Data Ingestion Pipeline on AWS That Doesn&#8217;t Lie to You</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>GitLab CI vs GitHub Actions vs Jenkins: Choosing Without Regretting It Later</title>
		<link>https://john-nessime.com/blog/devops/gitlab-ci-vs-github-actions-vs-jenkins/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 13 Aug 2026 10:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[CI/CD]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Deployment]]></category>
		<category><![CDATA[Git]]></category>
		<category><![CDATA[GitHub Actions]]></category>
		<category><![CDATA[GitLab CI]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Jenkins]]></category>
		<category><![CDATA[OIDC]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Platform Engineering]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Self Hosting]]></category>
		<category><![CDATA[Supply Chain Security]]></category>
		<category><![CDATA[Vendor Lock-In]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=201</guid>

					<description><![CDATA[<p>A practical comparison of GitLab CI, GitHub Actions and Jenkins that skips the feature table. What actually decides the choice is where your code lives, whether the runner can reach the deploy target, and who owns the control plane at 2am. Includes the cost mechanics, the security failure modes, and a decision procedure you can run in an afternoon.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/gitlab-ci-vs-github-actions-vs-jenkins/">GitLab CI vs GitHub Actions vs Jenkins: Choosing Without Regretting It Later</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question almost always arrives the same way. Someone pastes a long Jenkinsfile into a channel, adds a shrug emoji, and asks how hard it would be to move this to GitHub Actions. Nobody in the thread is asking the questions that will actually decide the outcome, because those questions are boring and the YAML is right there looking portable.</p>



<p class="wp-block-paragraph">Here is the thing that catches teams out: the pipeline definition is the cheapest part of a CI system to move. Rewriting stages and steps is a few days of tedious work for a normal service. What does not move is everything underneath it, and that is where a two-week migration turns into a two-quarter one.</p>



<p class="wp-block-paragraph">This post compares <strong>GitLab CI vs GitHub Actions vs Jenkins</strong> on the things that actually differ once you are running them in anger: who owns the runners, where secrets live, how the bill is calculated, and what breaks when an outside contributor opens a pull request. There is a profile of each with an honest &#8220;where it wins and where it doesn&#8217;t&#8221;, a section on cost mechanics, a section on the security failure modes, and a decision procedure at the end you can run in an afternoon.</p>



<h2 class="wp-block-heading">The part of the decision that actually bites</h2>



<p class="wp-block-paragraph">Draw a line under your pipeline file. Everything above the line is syntax. Everything below is platform.</p>



<p class="wp-block-paragraph">Below the line sits the runner fleet and how it is scaled, the secret store and how jobs authenticate to it, artifact and container registry storage with its retention rules, deploy credentials and the network path they travel, approval gates and who can click them, and the audit trail that someone in compliance will eventually ask about. None of that ports. All of it has to be rebuilt on the new platform, and each piece has a person or a team attached to it who has opinions.</p>



<p class="wp-block-paragraph">The failure mode this produces is specific and predictable. A team migrates the build and test half of the pipeline because that part is easy and demos well. Deploy stays on the old system &#8220;for now&#8221;, because deploy touches production and nobody wants to be the one who broke it during a migration. Six months later there are two CI systems, two secret stores, two sets of runners to patch, two audit trails, and a deploy path that nobody can explain end to end. The migration never technically failed. It just never finished, and the running cost doubled permanently.</p>



<p class="wp-block-paragraph">So the real question is not which YAML dialect you prefer. It is whether you are prepared to move the whole thing, and which platform makes the &#8220;below the line&#8221; half cheapest for your specific situation.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">What each platform is really optimising for</h2>



<p class="wp-block-paragraph">Feature tables make these three look more similar than they are. They are optimising for genuinely different things, and the differences show up as soon as you leave the happy path.</p>



<h3 class="wp-block-heading">GitHub Actions: shortest path from push to green</h3>



<p class="wp-block-paragraph">Actions is built around events in the GitHub repository. Something happens, a workflow fires, jobs land on hosted runners, and you did not have to provision anything. The marketplace means most common tasks are already someone else&#8217;s problem. Reusable workflows let a platform team define a pipeline once and have every service call it.</p>



<p class="wp-block-paragraph">A minimal hardened job header looks like this, and every line in it is there for a reason:</p>



<pre class="wp-block-code"><code>name: ci

on:
  pull_request:
    branches: [main]

permissions:
  contents: read      # start read-only, raise per job where needed
  id-token: write     # only for jobs that federate to a cloud provider

concurrency:
  group: ci-${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@PIN_TO_A_40_CHAR_COMMIT_SHA
      - name: Use the PR title safely
        env:
          PR_TITLE: ${{ github.event.pull_request.title }}
        run: echo "$PR_TITLE"</code></pre>



<p class="wp-block-paragraph">The <code>permissions</code> block caps what the automatically provided job token can do, so a compromised step cannot push to your branches. The <code>concurrency</code> block cancels superseded runs when someone pushes three times in a minute, which is the single easiest compute saving available. Pinning the action to a commit SHA rather than a tag matters because tags are mutable and have been repointed to malicious commits in real supply chain incidents. And binding the pull request title to an environment variable instead of interpolating it directly into the shell is what stops an attacker-controlled string from being executed as a command.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> your code is already on GitHub, you want CI running today, and your deploy targets are cloud APIs you can reach over the internet with a federated identity token. For open source it is close to unbeatable, since standard hosted runners are free for public repositories.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> anything needing deep control of the execution environment, long-running or stateful builds, deploys into networks the internet cannot reach, or a compliance posture that requires build logs and secrets to stay inside your perimeter. Self-hosted runners solve the network problem but hand you back the maintenance you were trying to avoid.</p>



<h3 class="wp-block-heading">GitLab CI: one platform owning the whole lifecycle</h3>



<p class="wp-block-paragraph">GitLab&#8217;s pitch is that source control, CI, package registry, container registry, security scanning and issue tracking are one product with one permission model and one audit trail. That is a real architectural advantage, not marketing. When a scanner finding, the merge request that introduced it, the pipeline that caught it and the registry image that contains it are all the same system, you spend a lot less time gluing tools together.</p>



<p class="wp-block-paragraph">The pipeline language rewards you for using its graph features rather than treating stages as a queue:</p>



<pre class="wp-block-code"><code>workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

default:
  interruptible: true

stages: [build, test, deploy]

build:
  stage: build
  script:
    - make build
  artifacts:
    paths: [dist/]
    expire_in: 1 week

unit:
  stage: test
  needs: [build]
  script:
    - make test

deploy:
  stage: deploy
  needs: [unit]
  environment: production
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual
  script:
    - ./deploy.sh</code></pre>



<p class="wp-block-paragraph">The <code>workflow:rules</code> block is the one people skip and then wonder why every push runs two pipelines. It decides whether a pipeline is created at all. <code>needs</code> turns the stage list into a directed graph, so a job starts the moment its own dependency finishes rather than waiting for every job in the previous stage. <code>interruptible: true</code> makes jobs eligible for automatic cancellation when a newer pipeline supersedes them, which you then have to enable in the project&#8217;s CI settings; the keyword alone does nothing. And <code>expire_in</code> is worth setting deliberately, because artifact storage is billed separately from compute and quietly accumulates.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> you want one vendor for the whole lifecycle, you have compliance requirements that benefit from a single auth model and audit trail, or you need self-managed hosting without giving up the integrated experience. The self-managed edition with your own runners is genuinely one of the better value propositions in this space at moderate to high build volume.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> your code is not in GitLab. The CI is tightly coupled to GitLab repositories, and running it against code hosted elsewhere is possible but perpetually awkward. The third-party ecosystem is also smaller, so more of what you need you will write yourself. And self-managing GitLab is a real operational commitment, not a Docker Compose file you forget about.</p>



<h3 class="wp-block-heading">Jenkins: whatever you need, on hardware you control</h3>



<p class="wp-block-paragraph">Jenkins gets dismissed as legacy by people who have never had a requirement it was the only tool to meet. It runs on your hardware, in your network, with no dependency on a vendor&#8217;s availability or pricing decisions. It talks to source control systems the other two do not. It can drive hardware test rigs, mainframe jobs, and build agents on operating systems nobody else offers. If your pipeline data legally cannot leave your network, this is a short list and Jenkins is on it.</p>



<pre class="wp-block-code"><code>pipeline {
  agent { label 'linux-ephemeral' }

  options {
    timeout(time: 30, unit: 'MINUTES')
    disableConcurrentBuilds()
    buildDiscarder(logRotator(numToKeepStr: '30'))
    timestamps()
  }

  stages {
    stage('Build') {
      steps {
        sh 'make build'
      }
    }
    stage('Deploy') {
      when { branch 'main' }
      steps {
        withCredentials([string(credentialsId: 'deploy-token', variable: 'DEPLOY_TOKEN')]) {
          sh './deploy.sh'
        }
      }
    }
  }

  post {
    always {
      cleanWs()
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Two things in that file are worth pointing at. <code>timeout</code> and <code>buildDiscarder</code> are the difference between a controller that runs for years and one that fills its disk and hangs on stuck jobs; set them on every pipeline, not as an afterthought. And <code>timestamps()</code> and <code>cleanWs()</code> both come from plugins rather than core. That is Jenkins in a sentence: the capability exists, and it arrives as a dependency you now own, patch, and eventually discover is unmaintained.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong> air-gapped or heavily restricted networks, source control that is not GitHub or GitLab, orchestration of things that are not code builds, and organisations with genuinely unusual workflows that no hosted product models. If you already have working Jenkins infrastructure and a person who understands it, the case for ripping it out is weaker than the migration slides suggest.</p>



<p class="wp-block-paragraph"><strong>Where it doesn&#8217;t:</strong> the controller is a stateful, single-point-of-failure server that someone has to own. Plugins are the extensibility model and also the attack surface, and the project publishes security advisories regularly enough that patching is a standing chore rather than an event. The Java baseline also moves; recent LTS lines have dropped support for older Java versions, so an upgrade is periodically a JVM migration too. If you cannot name the person responsible for that work, do not choose Jenkins.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">GitLab CI vs GitHub Actions vs Jenkins on cost</h2>



<p class="wp-block-paragraph">Published rates change often enough that quoting them ages a post badly. The billing mechanics change far more slowly, and the mechanics are what you actually need to model.</p>



<ul class="wp-block-list"><li><strong>GitHub Actions.</strong> Standard hosted runners are free for public repositories. Private repositories draw from a monthly pool of included minutes tied to your plan, and those minutes are Linux-equivalent: Windows and macOS jobs drain the pool at a multiplier, with macOS by far the steepest. Larger runners do not draw from the included pool at all and bill per minute from the first second. Billing is per job rounded up to the minute, minutes do not roll over, and usage is charged to the repository owner. Self-hosted runner usage carries no per-minute fee, though GitHub has floated changing that and then walked it back, so verify against current billing docs before you build a model on it.</li><li><strong>GitLab CI.</strong> Jobs on GitLab-hosted instance runners consume compute minutes, calculated as job duration multiplied by a cost factor for the machine size. The included pool is per top-level namespace, not per seat, which is the detail that surprises people: a five-person group and a fifty-person group on the same tier start from the same pool. Self-managed runners consume zero quota on every tier including Free, which is why &#8220;pay for seats, run your own runners&#8221; is such a common GitLab shape.</li><li><strong>Jenkins.</strong> No licence cost, which is where most cost comparisons stop and where the interesting part starts. You are paying for the controller, the agent fleet, the storage, and the engineering time to patch all three. That last item is the largest line and the one nobody puts in the spreadsheet.</li></ul>



<p class="wp-block-paragraph">Two observations that matter more than the platform choice itself. First, the levers with the biggest effect on a CI bill are the same everywhere: cancel superseded runs, cache dependencies properly, skip workflows on paths that did not change, and stop running the full matrix on every draft commit. A team that does none of these will pay more on the cheapest platform than a disciplined team pays on the most expensive one.</p>



<p class="wp-block-paragraph">Second, self-hosted runners are cheaper per minute and are not free. A pool of runners on a VPS provider like Hetzner, DigitalOcean or InterServer is inexpensive to rent and still needs image maintenance, autoscaling, disk cleanup and a security model. The break-even against hosted minutes is real, but it arrives later than people expect once you price the engineering hours honestly.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Security is where the differences stop being cosmetic</h2>



<p class="wp-block-paragraph">Every CI system is a machine that holds production credentials and executes code on demand. The platforms differ mostly in which mistakes they make easy.</p>



<ul class="wp-block-list"><li><strong>Untrusted pull requests.</strong> This is the classic GitHub Actions trap. The <code>pull_request</code> trigger runs fork code without access to your secrets. The <code>pull_request_target</code> trigger runs in the privileged context of the base repository, and if a workflow using it checks out the pull request&#8217;s code, an outside contributor can execute arbitrary code with your secrets in scope. GitHub&#8217;s own guidance is to avoid that trigger unless the workflow genuinely needs the privileged context, and never to check out untrusted code in it.</li><li><strong>Expression injection.</strong> Any attacker-controlled string, such as a branch name, issue title or commit message, that gets interpolated directly into a shell command is a code execution path. Bind it to an environment variable and reference the variable instead. This applies equally to GitLab, where the same values arrive as predefined CI variables.</li><li><strong>Self-hosted runners on public repositories.</strong> Don&#8217;t. A persistent runner that executes fork code is compromised for every subsequent job on it, including jobs that do have secrets. If you must self-host, use ephemeral runners that are destroyed after a single job, and keep public repositories on hosted runners.</li><li><strong>Mutable action and image references.</strong> Pin third-party actions to a full commit SHA. Tag repointing has been used in real supply chain compromises to reach thousands of repositories at once. The same logic applies to base images referenced by tag in GitLab jobs.</li><li><strong>Credential scope.</strong> Prefer short-lived federated credentials over long-lived secrets wherever the target supports it. GitHub and GitLab both issue OIDC tokens that cloud providers and HashiCorp Vault can exchange for short-lived access, and a token that expires in an hour is a much smaller incident than a static key that nobody rotates. Where you do use variables, mark them protected so they are only exposed on protected branches.</li><li><strong>Jenkins specifically.</strong> The controller holds the credentials store, so builds should never execute on it. Restrict agent-to-controller access, keep the plugin list as short as you can defend, and subscribe to the project&#8217;s security advisories, because plugin vulnerabilities are the most common way Jenkins installations get owned.</li></ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">A decision procedure you can run in an afternoon</h2>



<ol class="wp-block-list"><li><strong>Follow platform gravity first.</strong> Where does the code live, and where will it live in three years? CI that lives next to the code is cheaper to operate than CI that has to be integrated with it. If that single question has a clear answer, it decides most cases and the rest of this list is confirmation.</li><li><strong>Check whether a runner can reach the deploy target.</strong> Not the build. The deploy. If production sits behind a private network with no inbound path, you need self-hosted runners, a tunnel, or a system that already lives inside the perimeter. This is the requirement that most often overrides answer one.</li><li><strong>Name the on-call owner.</strong> Write down who gets paged when the CI system itself is down at 2am on a Sunday. If the honest answer is nobody, choose a hosted platform. Jenkins and self-managed GitLab both need that name to exist.</li><li><strong>Model the compute, not the sticker price.</strong> Take last month&#8217;s build minutes, split them by operating system and machine size, and apply each platform&#8217;s mechanics. macOS-heavy and Windows-heavy workloads change the answer significantly.</li><li><strong>Inventory what the plugins actually do.</strong> If you are leaving Jenkins, go through the plugin list and mark each one as replaced by a native feature, replaced by a marketplace action, needs writing, or can be dropped. The &#8220;needs writing&#8221; pile is your real migration estimate.</li><li><strong>Trial with your ugliest pipeline.</strong> Not the greenfield service. The one with the flaky integration test, the hardware dependency and the manual approval. Every platform handles a clean Node build beautifully. Only one of them will handle yours.</li></ol>



<h2 class="wp-block-heading">The hybrid pattern nobody puts on the slide</h2>



<p class="wp-block-paragraph">A very common real-world arrangement: GitHub Actions or GitLab CI runs build, test, scan and image publish, then a retained Jenkins instance handles deployment into restricted environments. It works, and for regulated organisations it is sometimes the only thing that does.</p>



<p class="wp-block-paragraph">Be honest about what it costs, though. You now have two systems to patch, two places secrets live, and a handoff between them that needs its own monitoring. It should be a deliberate architecture with a documented boundary, not the accidental result of a migration that stalled. The tell is whether anyone can draw the handoff on a whiteboard without hedging.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<ul class="wp-block-list"><li><strong>&#8220;Jenkins is legacy.&#8221;</strong> It is old, which is not the same thing. It is actively maintained and still the only option for several categories of requirement. The real objection is the maintenance cost, so make that argument instead; it is stronger and it is true.</li><li><strong>&#8220;We&#8217;ll save money with self-hosted runners.&#8221;</strong> On per-minute compute, yes. Once you price the image maintenance, autoscaling and patching, the saving is much smaller than the spreadsheet suggested, and it only exists at volume.</li><li><strong>&#8220;GitLab CI is just GitHub Actions with different YAML.&#8221;</strong> The pipeline languages are comparable. The products are not. One is an event platform around a repository, the other is an integrated lifecycle platform. The difference shows up in permissions, scanning and audit, not in syntax.</li><li><strong>&#8220;We&#8217;ll stay vendor-neutral by keeping all logic in shell scripts.&#8221;</strong> Good instinct, incomplete. The scripts port. The triggers, permissions, secret injection, artifact handling and approval gates do not, and those are most of the migration work.</li><li><strong>&#8220;The bigger marketplace wins.&#8221;</strong> Marketplace size counts unreviewed third-party code you are about to execute with your credentials. It is a convenience and a supply chain liability at the same time. Treat it as both.</li></ul>



<h2 class="wp-block-heading">How I&#8217;d decide</h2>



<ul class="wp-block-list"><li>Code on GitHub, cloud deploy targets, no unusual compliance requirement: GitHub Actions, hosted runners, OIDC to the cloud provider, actions pinned to SHAs. This is the one I reach for first because it is the least infrastructure to own.</li><li>Wanting one vendor for source, CI, registry and scanning, especially with compliance pressure: GitLab, and take self-managed seriously if data residency matters.</li><li>Restricted networks, unusual hardware, or existing Jenkins that works and has an owner: keep Jenkins, but put the pipeline in a Jenkinsfile, the controller config in Configuration as Code, and the plugin list under version control. Jenkins pain is almost always click-configured Jenkins pain.</li><li>Whichever you pick, wire pipeline duration and failure rate into whatever you already use for metrics, whether that is Grafana, Datadog or something homegrown. A CI system you cannot see is one you cannot tune, and slow pipelines are a cost problem and a morale problem at the same time.</li><li>Do not migrate for aesthetics. Migrate because a specific requirement is unmet, and write that requirement down before you start.</li></ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is Jenkins still worth using?</h3>



<p class="wp-block-paragraph">Yes, for specific situations. Air-gapped networks, source control that is not GitHub or GitLab, hardware-attached builds, and orchestration of things that are not software builds are all cases where Jenkins is the practical answer. For a standard cloud-deployed web service with code on GitHub, a hosted platform will cost you less operationally. The deciding factor is whether you have someone who owns the controller.</p>



<h3 class="wp-block-heading">Which is cheaper, GitLab CI or GitHub Actions?</h3>



<p class="wp-block-paragraph">It depends on your operating system mix and team size, because the two use different mechanics. GitHub applies multipliers so Windows and especially macOS jobs drain the included pool much faster than Linux. GitLab&#8217;s included compute pool is per namespace rather than per seat, so it does not grow as you hire. Both make self-hosted runners free of per-minute charges. Model your own last month of build minutes rather than trusting a general answer.</p>



<h3 class="wp-block-heading">Can I use GitLab CI with a GitHub repository?</h3>



<p class="wp-block-paragraph">There are mirroring and integration options, but the pipeline features are built around GitLab-hosted repositories and the experience degrades away from that. If your code is staying on GitHub, treat GitLab CI as a poor fit rather than a configuration challenge.</p>



<h3 class="wp-block-heading">How long does migrating from Jenkins to GitHub Actions or GitLab CI take?</h3>



<p class="wp-block-paragraph">Translating pipeline stages is usually days per service. The schedule is set by the parts that are not the pipeline: replacing plugin functionality, rebuilding the secret and identity model, provisioning runners with network access to deploy targets, and reproducing approval gates. Inventory those first, because that inventory is the estimate.</p>



<h3 class="wp-block-heading">Are self-hosted runners safe?</h3>



<p class="wp-block-paragraph">They are safe for private repositories with trusted contributors, provided the runner is ephemeral and destroyed after each job. They are not safe for public repositories, because a fork pull request can execute code on a persistent runner and reach everything that runs on it afterwards. Keep public repositories on hosted runners.</p>



<h3 class="wp-block-heading">Do I need to pin GitHub Actions to a commit SHA?</h3>



<p class="wp-block-paragraph">For any third-party action that runs with access to secrets, yes. Tags are mutable, and repointing a widely used tag to a malicious commit is a demonstrated attack path that has affected large numbers of repositories at once. Pinning costs you a dependency update per release and removes an entire class of compromise.</p>



<h3 class="wp-block-heading">Is it reasonable to run more than one CI system?</h3>



<p class="wp-block-paragraph">Reasonable if it is deliberate, expensive if it is accidental. A documented split where hosted CI builds and tests while an internal system deploys into restricted networks is a legitimate architecture. A split that exists because a migration stalled halfway is a permanent tax with no owner.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Comparing <strong>GitLab CI vs GitHub Actions vs Jenkins</strong> on syntax and feature checklists produces a decision that feels rigorous and predicts almost nothing. The pipeline file is the part that ports. The runner fleet, the secret and identity model, the artifact storage and the network path to your deploy target are the parts that get rebuilt, and they are where the schedule and the risk actually live.</p>



<p class="wp-block-paragraph">Pick the platform closest to where your code lives, unless a hard requirement about network access, data residency or unusual execution environments overrides it. Then spend your effort on the layer beneath the YAML, because that is the layer you will be living with.</p>



<h2 class="wp-block-heading">Need help choosing or moving a CI pipeline?</h2>



<p class="wp-block-paragraph">This is a large part of what I do as a freelance DevOps engineer. Typical engagements around CI/CD platform decisions look like:</p>



<ul class="wp-block-list"><li>A platform evaluation with your actual build minutes, operating system mix and deploy targets modelled, so the recommendation is arithmetic rather than opinion.</li><li>Jenkins to GitHub Actions or GitLab CI migrations, including the plugin inventory, the secret and identity rebuild, and a staged cutover that does not leave you running two systems forever.</li><li>Pipeline security review: fork pull request triggers, expression injection paths, action pinning, runner isolation and token permission scoping.</li><li>Replacing long-lived cloud keys with OIDC federation on GitHub Actions or GitLab CI, including the trust policy and claim conditions on the cloud side.</li><li>Self-hosted runner fleets that autoscale, stay ephemeral, and do not quietly fill their disks.</li><li>CI cost reduction: caching, concurrency cancellation, path filters and matrix pruning, measured before and after.</li><li>Jenkins hardening and Configuration as Code, so the controller is reproducible instead of a server nobody wants to touch.</li></ul>



<p class="wp-block-paragraph">If you want a second opinion, send me the pipeline file, a plugin list or a month of build minutes and I will tell you what I would actually do with it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/gitlab-ci-vs-github-actions-vs-jenkins/">GitLab CI vs GitHub Actions vs Jenkins: Choosing Without Regretting It Later</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Automating Server Provisioning for New Clients Without Building a Snowflake Farm</title>
		<link>https://john-nessime.com/blog/devops/automate-server-provisioning-new-clients/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 12 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Ansible]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[Client Onboarding]]></category>
		<category><![CDATA[cloud-init]]></category>
		<category><![CDATA[Freelancing]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Infrastructure as Code]]></category>
		<category><![CDATA[OpenTofu]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Server Provisioning]]></category>
		<category><![CDATA[SOPS]]></category>
		<category><![CDATA[SSH]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Terraform]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=192</guid>

					<description><![CDATA[<p>Hand-built client servers fail quietly: you only find out they were never reproducible on the day you need to rebuild one. Here is how to automate server provisioning for new clients using cloud-init for first boot, Ansible for convergence, per-client secrets and a verification gate that refuses to hand over a half-built box.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/automate-server-provisioning-new-clients/">Automating Server Provisioning for New Clients Without Building a Snowflake Farm</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The email is usually short. &#8220;Can you spin up a staging box that matches production?&#8221; You open your notes from eight months ago, find a half-finished shell history and a text file called <code>setup-notes-final2.txt</code>, and the honest answer is no. You can build something close. You cannot build the same machine.</p>



<p class="wp-block-paragraph">That is the failure that actually bites when you provision servers by hand for a handful of clients. It is invisible for months. Nothing alerts. Nothing breaks. Then a disk dies, or a client asks for a second region, or you want to hand the whole thing over, and the gap between &#8220;the server works&#8221; and &#8220;the server is reproducible&#8221; turns into a weekend.</p>



<p class="wp-block-paragraph">This post covers how to automate server provisioning for new clients in a way that survives that moment: what to put in the image, what to put in first boot, what to leave to configuration management, how to keep one client&#8217;s credentials from ever touching another client&#8217;s box, and how to prove a build is finished instead of assuming it.</p>



<h2 class="wp-block-heading">The four failure families worth designing against</h2>



<p class="wp-block-paragraph">Automation is not the goal. Automation is what you reach for once you have decided which of these you refuse to keep living with. Pick the ones that apply to you and let them shape the build, because trying to solve all four at once is how provisioning projects stall.</p>



<h3 class="wp-block-heading">1. You cannot rebuild it</h3>



<p class="wp-block-paragraph">The test is blunt: destroy the machine and rebuild it from your repository, with no shell access to the old one. If you cannot do that, you do not have a server, you have an artefact. Every hand-run <code>apt install</code>, every file edited over SSH at 1am, every &#8220;I&#8217;ll document that later&#8221; is a piece of the machine that exists nowhere but the machine.</p>



<p class="wp-block-paragraph">This is the one that stays invisible longest. A hand-built box can run happily for two years. The cost lands entirely in one bad afternoon.</p>



<h3 class="wp-block-heading">2. Every client is a slightly different server</h3>



<p class="wp-block-paragraph">Client A got fail2ban. Client B got it two months later with different jail settings. Client C never got it because that build happened on a Friday. Now a CVE lands in a package you installed on four of six boxes and you have to go look at each one to find out.</p>



<p class="wp-block-paragraph">Drift is not just untidy. It destroys your ability to reason about the fleet as a fleet. Any statement that starts &#8220;all our clients have&#8230;&#8221; becomes a guess.</p>



<h3 class="wp-block-heading">3. One credential, too much blast radius</h3>



<p class="wp-block-paragraph">This one is specific to consulting and it is the one people get wrong most often. If the same SSH key opens every client&#8217;s server, and the same API token provisions every client&#8217;s infrastructure, then a compromise of your laptop is a compromise of everybody. You are a shared dependency with none of the controls a shared dependency is supposed to have.</p>



<p class="wp-block-paragraph">Worse, it is contractually awkward. Plenty of client agreements have language about access control that a single reused key quietly violates.</p>



<h3 class="wp-block-heading">4. The handover you cannot perform</h3>



<p class="wp-block-paragraph">Engagements end. When one does, the client should be able to take a repository, run it, and get their infrastructure. If the only way to reproduce their environment is to hire you again, that is not leverage, it is a liability. It also makes you slower to say yes to the next client, because your capacity is tied up in machines only you understand.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">How to automate server provisioning for new clients</h2>



<p class="wp-block-paragraph">The stack that holds up in practice has four layers, and the important part is the boundary between them. Most broken provisioning setups I see are broken because someone pushed work into the wrong layer.</p>



<h3 class="wp-block-heading">Layer 1: the machine itself</h3>



<p class="wp-block-paragraph">Terraform or OpenTofu talks to your provider&#8217;s API and creates the server, the firewall rules, the volumes and the DNS records. Whether you run a VPS at InterServer, Hetzner, Vultr or DigitalOcean, or full cloud on AWS, the shape is the same: a declarative file that says what should exist.</p>



<p class="wp-block-paragraph">Worth knowing before you commit: HashiCorp moved Terraform from the MPL to the Business Source License, which is source-available rather than open source, and the community forked the last MPL release as OpenTofu under the Linux Foundation. For an independent engineer building client infrastructure, either is workable. Terraform has the larger provider and module ecosystem and the commercial backing. OpenTofu gives you a licence that will not be renegotiated under you, which matters more when the code you write ends up in a client&#8217;s repository and out of your control. The HCL, provider protocol and state model are shared, so the switching cost is low in both directions. I lean OpenTofu for client work for the licence certainty, and that is a judgement call, not a technical verdict.</p>



<p class="wp-block-paragraph">Keep state remote and locked from day one, and keep one state file per client. Shared state is the fastest way to turn a mistake on one engagement into an outage on another.</p>



<h3 class="wp-block-heading">Layer 2: first boot with cloud-init</h3>



<p class="wp-block-paragraph">cloud-init runs once, on first boot, before you have any way in. Its job is narrow: create the account you will connect as, install its key, lock down password and root login, and get out of the way. Everything else belongs in the next layer.</p>



<pre class="wp-block-code"><code>#cloud-config
hostname: web01
fqdn: web01.acme.internal
timezone: Etc/UTC

users:
  - name: deploy
    groups: [sudo]
    shell: /bin/bash
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3Nz... provisioning key for this client only

# Refuse passwords and direct root logins from the very first boot,
# so there is no window where the box is reachable with weaker settings.
ssh_pwauth: false
disable_root: true

package_update: true
packages:
  - python3
  - sudo</code></pre>



<p class="wp-block-paragraph">Two deliberate omissions. There is no <code>package_upgrade: true</code>, because a full upgrade at first boot can run for minutes, holds the package lock, and is the single most common cause of the race described in the troubleshooting section below. Do the upgrade in Ansible where you can see it fail. And there is no application setup, because anything in <code>runcmd</code> is a script you will never test again.</p>



<p class="wp-block-paragraph"><code>python3</code> is there so Ansible has an interpreter to work with. Some images ship it already; installing it is cheap insurance.</p>



<p class="wp-block-paragraph">Validate the file before it ever reaches a provider, because a YAML error in user-data fails silently and you find out by not being able to log in:</p>



<pre class="wp-block-code"><code>cloud-init schema --config-file user-data.yaml</code></pre>



<h3 class="wp-block-heading">Layer 3: convergence with Ansible</h3>



<p class="wp-block-paragraph">This is where the actual build lives: packages, users, firewall, TLS, web server, database, backup agent, monitoring agent, log shipping. Ansible is agentless and runs over SSH, which fits consulting well because you are not asking a client to accept a permanent daemon on their box.</p>



<p class="wp-block-paragraph">The layout that scales across clients separates the shared build from the per-client differences:</p>



<pre class="wp-block-code"><code>infra/
  roles/                  # shared, versioned, the same for everyone
    base/
    hardening/
    web/
    backup/
  clients/
    acme/
      inventory/hosts.ini
      group_vars/all.yml
      secrets/vars.sops.yml
      terraform/
    globex/
      inventory/hosts.ini
      group_vars/all.yml
      secrets/vars.sops.yml
      terraform/
  site.yml</code></pre>



<p class="wp-block-paragraph">Roles are the product. Client directories are configuration. If you find yourself copying a role into a client directory and editing it, that is a variable you have not extracted yet.</p>



<p class="wp-block-paragraph">Hand the inventory over from Terraform rather than maintaining it by hand. The simplest reliable version reads the outputs as JSON:</p>



<pre class="wp-block-code"><code>cd clients/acme/terraform

tofu output -json client_hosts 
  | jq -r '.value[] | "(.name) ansible_host=(.ipv4)"' 
  &gt; ../inventory/hosts.ini</code></pre>



<p class="wp-block-paragraph">There are richer options, including inventory plugins that read Terraform state directly, but a generated file has a property worth keeping: you can read it, diff it, and commit it. When a provisioning run targets the wrong host, that diff is how you find out.</p>



<h3 class="wp-block-heading">Layer 4: the verification gate</h3>



<p class="wp-block-paragraph">This is the layer almost everyone skips, and it is the one that turns provisioning from a script into a process. A playbook that finishes without errors has told you that no task raised an exception. It has not told you the server works.</p>



<p class="wp-block-paragraph">Write a separate play whose only job is to assert the finished state, and run it as the last step of every provision:</p>



<pre class="wp-block-code"><code>- name: Verify the host is genuinely ready
  hosts: all
  gather_facts: true
  tasks:
    # raw, not command: this may run before Python is guaranteed present
    - name: Wait for cloud-init to finish
      ansible.builtin.raw: cloud-init status --wait
      changed_when: false

    - name: Collect service state
      ansible.builtin.service_facts:

    - name: Core services are running
      ansible.builtin.assert:
        that:
          - ansible_facts.services['nginx.service'].state == 'running'
          - ansible_facts.services[ssh_unit].state == 'running'
        fail_msg: "Core services are not up on {{ inventory_hostname }}"

    - name: Site answers over TLS
      ansible.builtin.uri:
        url: "https://{{ client_domain }}/"
        status_code: 200
      delegate_to: localhost</code></pre>



<p class="wp-block-paragraph">Note <code>ssh_unit</code> as a variable rather than a literal: the SSH unit is <code>ssh.service</code> on Debian and Ubuntu but <code>sshd.service</code> on RHEL-family systems. Hard-coding it is how a verification play passes on one distro and fails confusingly on another.</p>



<p class="wp-block-paragraph">The point of the gate is not that it catches exotic bugs. It is that it catches the ordinary ones: the service that installed but did not enable, the certificate that issued but did not get loaded, the firewall rule that opened the wrong port.</p>



<h2 class="wp-block-heading">Keeping clients apart: secrets and access</h2>



<p class="wp-block-paragraph">Generate a fresh SSH keypair per client engagement. It costs you thirty seconds and it means revoking access to one client&#8217;s infrastructure is a single, contained action.</p>



<p class="wp-block-paragraph">For secrets in the repository you have two credible options. Ansible Vault is built in and needs nothing extra, but it encrypts whole files, so your diffs become meaningless blobs and rotating a key across several clients is tedious. SOPS encrypts values while leaving keys readable, which means a code review can still see that a variable changed without seeing what it changed to, and it supports age, PGP and the cloud KMS services. For a repo that a client might eventually inherit, that readability is worth the extra tool.</p>



<pre class="wp-block-code"><code># .sops.yaml at the repo root: different key per client
creation_rules:
  - path_regex: clients/acme/secrets/.*.ya?ml$
    age: age1acme...
  - path_regex: clients/globex/secrets/.*.ya?ml$
    age: age1globex...</code></pre>



<p class="wp-block-paragraph">Separate keys per client is the whole point. One shared key and you are back to a single blast radius with extra ceremony.</p>



<p class="wp-block-paragraph">For handing the initial credentials to a client, use something with an audit trail and an expiry. A shared vault item in 1Password or Bitwarden is fine. Email and chat are not, because the secret then lives in two message histories neither of you controls.</p>



<p class="wp-block-paragraph">If you want ongoing access without exposing SSH to the internet at all, an overlay network like Tailscale installed by the base role is a clean way to do it. The provisioning path still uses a public address for the first run, then the firewall closes it.</p>



<h2 class="wp-block-heading">Troubleshooting a provisioning run</h2>



<h3 class="wp-block-heading">Ansible connects before cloud-init has finished</h3>



<p class="wp-block-paragraph">The most common failure by a wide margin. SSH comes up before cloud-init is done, Ansible connects, and then the first package task fails because cloud-init still holds the package manager lock. The error points at apt or dnf, which sends you looking in entirely the wrong place.</p>



<p class="wp-block-paragraph">Fix it by blocking explicitly at the top of the first play, as in the verification snippet above. Check the state manually with:</p>



<pre class="wp-block-code"><code>cloud-init status --wait --long
sudo grep -i Traceback /var/log/cloud-init.log</code></pre>



<p class="wp-block-paragraph">One caveat that catches people: never put <code>cloud-init status --wait</code> inside a <code>runcmd</code> or <code>bootcmd</code> block. It will deadlock waiting for the process that is running it.</p>



<h3 class="wp-block-heading">cloud-init appears to have done nothing</h3>



<p class="wp-block-paragraph">Two usual causes. Either the user-data was malformed, in which case cloud-init skipped it rather than erroring loudly, or the modules already ran on a previous boot and the semaphore files in <code>/var/lib/cloud/instance/sem/</code> are preventing a rerun.</p>



<pre class="wp-block-code"><code># what actually ran, and how long each stage took
cloud-init analyze show

# the human-readable output of scripts and package installs
sudo less /var/log/cloud-init-output.log

# on a throwaway test box only: wipe state and start over
sudo cloud-init clean --logs --reboot</code></pre>



<p class="wp-block-paragraph">Never run <code>cloud-init clean</code> on a production host. It removes cached state and the following boot will behave as a first boot.</p>



<h3 class="wp-block-heading">Host key verification failed on a rebuilt server</h3>



<p class="wp-block-paragraph">Rebuild a machine on the same IP and every client with the old key in <code>known_hosts</code> refuses to connect. The tempting fix is to disable host key checking in <code>ansible.cfg</code>. Do not. That turns a real integrity check into a permanent no-op across every client you manage.</p>



<p class="wp-block-paragraph">Remove the stale entry and pin the new one as part of the provisioning flow:</p>



<pre class="wp-block-code"><code>ssh-keygen -R 203.0.113.10
ssh-keyscan -H 203.0.113.10 &gt;&gt; ~/.ssh/known_hosts</code></pre>



<p class="wp-block-paragraph">Better still, if your provider exposes the host key fingerprint through its API or console output, compare it before you accept. That closes the trust-on-first-use gap properly.</p>



<h3 class="wp-block-heading">The playbook reports changes on every single run</h3>



<p class="wp-block-paragraph">Almost always a <code>shell</code> or <code>command</code> task with no <code>creates</code>, <code>changed_when</code> or module equivalent. Ansible cannot know whether a raw command changed anything, so it assumes yes. Real changed counts are what let you spot drift, and a playbook that always reports twelve changes tells you nothing.</p>



<p class="wp-block-paragraph">Run against an already-provisioned host in check mode. Anything that reports a change is either genuine drift or a task you need to fix:</p>



<pre class="wp-block-code"><code>ansible-playbook -i clients/acme/inventory/hosts.ini site.yml --check --diff</code></pre>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Putting the application build in <code>runcmd</code>. It runs once, never gets tested, and is invisible to every later change.</li>

<li>One shared SSH key and one shared API token across all clients. Convenient right up to the moment it is not.</li>

<li>Copying a role into a client directory to make one small change, instead of extracting a variable.</li>

<li>Treating &#8220;the playbook exited zero&#8221; as proof the server is ready. It is proof no task threw.</li>

<li>Local Terraform state on your laptop. It is a single point of failure that also cannot be handed over.</li>

<li>Provisioning the server but not the DNS, TLS, monitoring and backups. A box nobody watches and nobody can restore is not finished.</li>

<li>Automating the build and then fixing production by hand. One SSH session and you are back to a snowflake.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li><strong>Rebuild before you need to.</strong> Destroy and rebuild a client&#8217;s staging host on a schedule. It is the only honest test of reproducibility.</li>

<li><strong>Version your roles and pin them per client.</strong> A shared role that everyone tracks on main means one commit changes six clients at once.</li>

<li><strong>Keep first boot minimal.</strong> If it can wait for Ansible, it should.</li>

<li><strong>Make monitoring part of provisioning.</strong> Whether that is Prometheus you run yourself, Grafana Cloud, or a lighter uptime service, the agent goes on in the base role, not as a follow-up ticket.</li>

<li><strong>Same for backups.</strong> An offsite target such as Backblaze B2 configured at build time, plus a restore test, or you have a backup you have never proven.</li>

<li><strong>Separate credentials per client, always.</strong> Keys, tokens, encryption keys, state backends.</li>

<li><strong>Write the runbook as you build.</strong> The provisioning repo&#8217;s README is the handover document. If it is accurate, the engagement can end cleanly.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need Terraform if I only manage a few VPS instances?</h3>



<p class="wp-block-paragraph">Not necessarily. If you are creating three servers a year through a provider&#8217;s control panel, the honest answer is that Terraform adds a state file to look after for very little return. Start with cloud-init and Ansible, which give you most of the reproducibility. Add Terraform or OpenTofu when the infrastructure around the server, such as firewalls, volumes, load balancers and DNS records, starts to be the part you keep getting wrong.</p>



<h3 class="wp-block-heading">Should I build custom images instead of provisioning at boot?</h3>



<p class="wp-block-paragraph">Golden images cut boot time and remove dependency on package mirrors being up during a build. They also add a pipeline you have to maintain and rebuild for every security update. For client work at small scale, provisioning from a stock distribution image is usually the better trade, because the build stays readable and there is no stale image to forget about. Images earn their cost when you are launching machines often enough that boot time matters.</p>



<h3 class="wp-block-heading">Terraform or OpenTofu for client projects?</h3>



<p class="wp-block-paragraph">Both work, and the configuration language is shared, so this is not a lock-in decision in the way it looks. Terraform has the deeper ecosystem and the commercial support path. OpenTofu is under the Linux Foundation with a licence that is unlikely to change, which is the stronger argument when the code will live in a client repository long after you have moved on. Ask the client whether they have a standard before you pick.</p>



<h3 class="wp-block-heading">How do I automate server provisioning without exposing SSH to the internet?</h3>



<p class="wp-block-paragraph">The first run generally needs a reachable address, so scope the firewall rule to your own source IP for the duration of provisioning and close it in the same run. After that, reach the host through a bastion or an overlay network installed during the base role. Provider firewalls and host firewalls should both be in the automation, because relying on only one of them means a provider console change can silently open the box.</p>



<h3 class="wp-block-heading">How do I stop configuration drift once the server is live?</h3>



<p class="wp-block-paragraph">Run the playbook in check mode on a schedule and look at what it reports. Any change on a host you have not touched is drift, and it is far cheaper to find it that way than during an incident. This only works if your tasks are properly idempotent, which is why the always-reports-changed problem above matters more than it looks.</p>



<h3 class="wp-block-heading">What belongs in cloud-init versus Ansible?</h3>



<p class="wp-block-paragraph">cloud-init handles what must exist before you can connect: hostname, the login account, its SSH key, disabled password authentication, and a Python interpreter. Ansible handles everything you will ever want to change again. The dividing line is repeatability. cloud-init runs once and you cannot re-run it safely on a live host, so anything you might revisit belongs on the Ansible side.</p>



<h3 class="wp-block-heading">How long should provisioning a new client server take?</h3>



<p class="wp-block-paragraph">Wall-clock time is the wrong measure. The useful measure is how much of it needs you. A build that takes twenty minutes but runs unattended from one command is in far better shape than a five-minute build with four manual steps, because the manual steps are where the divergence between clients gets introduced.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">When you automate server provisioning for new clients, the deliverable is not the script. It is the ability to answer &#8220;can you rebuild this?&#8221; with yes, on any day, without opening a shell on the existing machine.</p>



<p class="wp-block-paragraph">Everything else follows from that. Reproducibility gives you consistency across the fleet, because there is one definition. It gives you a clean handover, because the repository is the environment. It gives you the ability to take the next client on without dreading the build. And it costs the same afternoon that a single unplanned rebuild would have cost you anyway, just spent on your own schedule instead of at 2am.</p>



<p class="wp-block-paragraph">Start with one client. Get their staging box rebuilding from scratch. Then make the second client&#8217;s build reuse the roles from the first. That is the whole path.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a provisioning setup you can actually hand over?</h2>



<p class="wp-block-paragraph">Most of the provisioning work I take on is not greenfield. It is a set of hand-built servers that work fine and nobody can reproduce. Things I can help with:</p>



<ul class="wp-block-list">
<li>Turning existing hand-built client servers into an Ansible role set, without a rebuild and without downtime</li>

<li>Building a client onboarding repository: Terraform or OpenTofu, cloud-init, roles, per-client inventory and secrets</li>

<li>Splitting shared credentials into per-client keys, tokens and state backends, with a revocation path that works</li>

<li>Writing the verification playbook that decides whether a build is finished, and wiring it into CI</li>

<li>Fixing idempotency so check-mode runs become a usable drift report instead of noise</li>

<li>Preparing an environment for handover: runbook, access review, and a documented rebuild that the client can run themselves</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion on something specific, send the cloud-config, the playbook output, or a check-mode diff you cannot explain, and I will tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/automate-server-provisioning-new-clients/">Automating Server Provisioning for New Clients Without Building a Snowflake Farm</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Keep Kubernetes Secrets Out of Git Without Kidding Yourself</title>
		<link>https://john-nessime.com/blog/devops/keep-kubernetes-secrets-out-of-git/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 09 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Kubernetes]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Argo CD]]></category>
		<category><![CDATA[CI/CD]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[External Secrets Operator]]></category>
		<category><![CDATA[Flux CD]]></category>
		<category><![CDATA[Git]]></category>
		<category><![CDATA[GitOps]]></category>
		<category><![CDATA[HashiCorp Vault]]></category>
		<category><![CDATA[Infrastructure as Code]]></category>
		<category><![CDATA[Sealed Secrets]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[SOPS]]></category>
		<category><![CDATA[Supply Chain Security]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=174</guid>

					<description><![CDATA[<p>Encrypting a Secret before you commit it only closes one of the five places that value comes to rest. Here is how to keep Kubernetes secrets out of Git properly: Sealed Secrets, SOPS and External Secrets compared honestly, the key-backup problem nobody plans for, and what to actually do when a credential is already in your history.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/keep-kubernetes-secrets-out-of-git/">Keep Kubernetes Secrets Out of Git Without Kidding Yourself</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually arrives on a quiet afternoon. A secret scanner has flagged your infrastructure repo, and the finding is a Postgres connection string sitting in a <code>values.yaml</code> that someone committed a long time ago and deleted a few weeks later. The file is gone from the working tree. The commit is not.</p>



<p class="wp-block-paragraph">The reflex at that point is to reach for history rewriting. That is the wrong first move, and I will come back to why. But the bigger problem is the one that shows up six months after you clean up: teams migrate to Sealed Secrets or SOPS, tick the box, and quietly assume the whole class of problem is now solved. It is not. Encrypting a value before it enters Git closes exactly one door out of about five.</p>



<p class="wp-block-paragraph">This post covers how to <strong>keep Kubernetes secrets out of Git</strong> in a way that survives contact with a real cluster: the three families of solution and where each one genuinely wins, the disaster-recovery trap that bites hardest, how to stop plaintext reaching the repo in the first place, and what to actually do about credentials already in your history.</p>



<h2 class="wp-block-heading">The failure that actually bites: encrypted in Git, wide open in the cluster</h2>



<p class="wp-block-paragraph">Sealed Secrets and SOPS both terminate in the same place. The controller decrypts, the API server accepts, and what lands in etcd is an ordinary Kubernetes Secret. Ordinary means base64-encoded, which is an encoding, not a cipher. Anyone who can run <code>kubectl get secret -o yaml</code> in that namespace reads the value in one pipe through <code>base64 -d</code>.</p>



<p class="wp-block-paragraph">So the honest description of what these tools buy you is narrower than the marketing suggests. They stop the credential appearing in a public object store with permanent history and a CDN in front of it. They do nothing about the cluster&#8217;s own blast radius.</p>



<p class="wp-block-paragraph">Three things need to be true alongside whichever tool you pick:</p>



<ul class="wp-block-list">
<li><strong>Encryption at rest is on.</strong> A stock Kubernetes control plane writes Secrets to etcd unencrypted unless you configure an <code>EncryptionConfiguration</code>, ideally with a KMS provider. Managed distributions vary. On EKS, envelope encryption is something you enable, not something you inherit. Check yours rather than assuming.</li>

<li><strong>RBAC on secrets is actually restrictive.</strong> A depressing number of clusters grant blanket <code>get</code> and <code>list</code> on secrets to service accounts that only ever needed one. That single verb turns a namespace compromise into a credential dump.</li>

<li><strong>Secrets are mounted as files, not shoved into environment variables where you can help it.</strong> Environment variables leak through <code>/proc/&lt;pid&gt;/environ</code>, through anything that dumps the environment on crash, and through every debug endpoint an application framework has ever shipped. A projected volume with tight permissions is the quieter option.</li>
</ul>



<p class="wp-block-paragraph">Quick sanity check on the RBAC side, which takes about ten seconds and surprises people:</p>



<pre class="wp-block-code"><code># Can the default service account in a namespace read every secret in it?
kubectl auth can-i get secrets 
  --namespace production 
  --as system:serviceaccount:production:default</code></pre>



<p class="wp-block-paragraph">If that returns <code>yes</code>, your Git hygiene is not the weakest link in the chain.</p>



<h2 class="wp-block-heading">Already leaked? Rotate first, rewrite second, and understand what rewriting does not do</h2>



<p class="wp-block-paragraph">Order matters here more than technique. GitHub&#8217;s own guidance is blunt about it: once a credential has been pushed, revoke or rotate it first, because that step alone removes the attacker&#8217;s ability to use it. History surgery is the optional follow-up, not the fix.</p>



<p class="wp-block-paragraph">The reason is that a force-push does not reach everywhere the commit went. Objects survive in forks, which inherit the full object graph at fork time. They survive in colleagues&#8217; clones. On GitHub they remain reachable by SHA through cached views and through pull requests that referenced them, and purging those caches is a support request, not a git command. If the repo was public even briefly, treat the credential as burned and move on.</p>



<p class="wp-block-paragraph">When you do rewrite, use <code>git filter-repo</code>. It has replaced <code>filter-branch</code> as the recommended tool and it is dramatically faster on any repo with real history.</p>



<pre class="wp-block-code"><code># Work on a bare mirror, never your day-to-day clone.
git clone --mirror git@github.com:org/infra.git infra-clean.git
cd infra-clean.git

# Strip one path from every commit that ever touched it.
git filter-repo --path clusters/production/values.yaml --invert-paths

# filter-repo deletes the origin remote on purpose, so you cannot
# force-push a half-finished rewrite by muscle memory. Add it back.
git remote add origin git@github.com:org/infra.git
git push --force --all
git push --force --tags</code></pre>



<p class="wp-block-paragraph">Then the unglamorous part: close or merge open pull requests before you start, because every SHA downstream of the rewrite changes. Tell every collaborator to delete their clone and clone again. A <code>git pull</code> will happily drag the old objects back in.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Two families of solution, and how to tell which one you are in</h2>



<p class="wp-block-paragraph">Every approach to this problem is either <em>encrypt the value and commit the ciphertext</em>, or <em>commit a pointer and fetch the value at runtime</em>. Everything else is implementation detail. The choice is mostly about whether you already run a secret store, and whether you can tolerate the CLI step in the middle of your workflow.</p>



<h3 class="wp-block-heading">Sealed Secrets: the lowest-friction way to start</h3>



<p class="wp-block-paragraph">A controller in the cluster holds an RSA private key and publishes the matching certificate. The <code>kubeseal</code> CLI encrypts a Secret manifest against that certificate and produces a <code>SealedSecret</code> custom resource. Only that controller can decrypt it, so the resulting YAML is safe to commit.</p>



<pre class="wp-block-code"><code># Fetch the controller's public certificate. This is safe to commit;
# it can only encrypt, never decrypt.
kubeseal --fetch-cert 
  --controller-name=sealed-secrets 
  --controller-namespace=kube-system 
  &gt; pub-cert.pem

# Build a Secret manifest locally. --dry-run=client means it is rendered
# and never sent to the API server.
kubectl create secret generic db-auth 
  --from-literal=password='&lt;value&gt;' 
  --dry-run=client -o yaml &gt; db-auth.yaml

# Encrypt against the cert. Offline, so no cluster access needed here.
kubeseal --cert pub-cert.pem --format yaml &lt; db-auth.yaml &gt; db-auth-sealed.yaml

rm db-auth.yaml</code></pre>



<p class="wp-block-paragraph">Because you can hand out <code>pub-cert.pem</code>, developers can seal secrets without any cluster credentials at all. That is the underrated part, and it is the reason this is the one I reach for first on a small team with no existing vault.</p>



<p class="wp-block-paragraph">Two details that catch people. First, sealing is scoped by default: a <code>SealedSecret</code> is bound to its exact name and namespace, and moving the file to a different namespace makes it undecryptable. That is deliberate, it stops a developer sealing a secret into a namespace they should not have access to, and you can relax it with <code>--scope namespace-wide</code> or <code>--scope cluster-wide</code> if you genuinely need to. Second, the Helm chart names the controller <code>sealed-secrets</code> while the CLI looks for <code>sealed-secrets-controller</code> by default, so <code>--controller-name</code> is not optional in practice.</p>



<p class="wp-block-paragraph">Where it stops being the right answer: multi-cluster. Each cluster generates its own key pair, so a sealed file is not portable, and a fleet means either copying private keys around or maintaining per-cluster ciphertext for the same value. Rotation is also manual. There is no dynamic secret story here at all.</p>



<h3 class="wp-block-heading">SOPS with age: encrypt files, not just Kubernetes objects</h3>



<p class="wp-block-paragraph">SOPS encrypts the values inside a structured file and leaves the keys readable, so a diff still tells you which field changed even if it cannot tell you what it changed to. Paired with <code>age</code>, which uses short X25519 keys instead of the GPG keyring experience, it is pleasant to live with.</p>



<p class="wp-block-paragraph">A <code>.sops.yaml</code> at the repo root drives everything through path matching, which is how you get per-environment key separation without anyone having to remember flags:</p>



<pre class="wp-block-code"><code>creation_rules:
  - path_regex: clusters/production/.*.yaml$
    encrypted_regex: '^(data|stringData)$'
    age: age1...   # production recipient
  - path_regex: clusters/staging/.*.yaml$
    encrypted_regex: '^(data|stringData)$'
    age: age1...   # staging recipient</code></pre>



<p class="wp-block-paragraph">The <code>encrypted_regex</code> line is the one worth understanding rather than copying. Without it SOPS encrypts everything including <code>metadata</code>, and you lose the readable-diff property that made this approach attractive. Restricting it to <code>data</code> and <code>stringData</code> keeps names, namespaces and labels reviewable in a pull request.</p>



<p class="wp-block-paragraph">Flux decrypts SOPS natively: you point a <code>Kustomization</code> at a Secret holding the age private key via <code>decryption.provider: sops</code> and it handles the rest. Argo CD does not have equivalent built-in support, so you bolt it on with a config management plugin such as KSOPS or the Argo CD Vault Plugin. Worth confirming against current Argo CD docs before you commit to it, because plugin mechanics there have changed more than once.</p>



<p class="wp-block-paragraph">SOPS also handles files that are not Kubernetes manifests: Terraform variable files, Ansible vars, plain <code>.env</code>. If you have config living outside the cluster, that reach is a real advantage over Sealed Secrets.</p>



<h3 class="wp-block-heading">External Secrets Operator: stop putting the value in Git at all</h3>



<p class="wp-block-paragraph">The third approach sidesteps encryption entirely. You commit a reference, and a controller resolves it against a real secret store: HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, Google Secret Manager, or one of the newer hosted options like Doppler, Infisical or 1Password. The repo contains a path, never a value.</p>



<pre class="wp-block-code"><code>apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: db-auth
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: db-auth
  data:
    - secretKey: password
      remoteRef:
        key: production/postgres
        property: password</code></pre>



<p class="wp-block-paragraph">This is the only one of the three that gives you rotation without a commit. Change the value in Vault, wait for <code>refreshInterval</code>, and the cluster Secret updates. It also gives you a real audit trail, because reads happen against a system built to log them.</p>



<p class="wp-block-paragraph">The cost is honest and worth stating. You have introduced a runtime dependency: if the store is unreachable, new workloads cannot get their credentials. You still have a bootstrap credential problem, because the operator needs to authenticate to the store somehow, and that root of trust has to get into the cluster by some other path. And Git is no longer the complete description of your system, which is a genuine philosophical cost if GitOps purity matters to you.</p>



<p class="wp-block-paragraph"><strong>One live upgrade trap.</strong> The <code>external-secrets.io/v1beta1</code> API was removed in ESO 0.17.0. Before that, in the 0.16 series, a conversion webhook served both versions and rewrote stored objects to <code>v1</code>, which produced permanent drift in Argo CD for anyone whose Git manifests still said <code>v1beta1</code>. Argo sees <code>v1</code> in the cluster, Git says <code>v1beta1</code>, and it reconciles forever. Update your manifests to <code>v1</code> before you cross that boundary, not after.</p>



<h3 class="wp-block-heading">Secrets Store CSI Driver: when a Secret object is one object too many</h3>



<p class="wp-block-paragraph">Worth knowing about even if you do not use it. The CSI driver mounts values from an external store directly into the pod as a volume, so no Kubernetes Secret object needs to exist at all. That closes the etcd exposure completely. It costs you the ability to consume the value as an environment variable, and it means the secret is only available to pods that mount it, which is either exactly what you want or a nuisance depending on the workload. Use it where the compliance posture demands that etcd never sees the value.</p>



<h2 class="wp-block-heading">The key you forgot to back up</h2>



<p class="wp-block-paragraph">This is the failure I would put money on, and it is the one nobody rehearses. Every encrypt-into-Git approach concentrates all your risk into one small piece of key material. Lose it and your repository turns into a directory of well-organised noise.</p>



<p class="wp-block-paragraph">Concretely: rebuild a cluster from scratch, install a fresh Sealed Secrets controller, and it generates a new key pair. Every <code>SealedSecret</code> in your repo is now undecryptable. Your GitOps repo, the one that was supposed to let you recreate everything, cannot recreate anything that needed a credential. You find this out during the incident, not before it.</p>



<ul class="wp-block-list">
<li><strong>Back up the sealing key out of band.</strong> The controller&#8217;s private key lives in a Secret in its namespace. Export it, encrypt it, and store it somewhere that is not the cluster and not the repo. A password manager or an offline copy is fine. Test the restore.</li>

<li><strong>Same for age.</strong> The private key is one line of text. Its size makes it feel unimportant. It is the entire thing.</li>

<li><strong>Add a second recipient for anything that matters.</strong> SOPS encrypts the data key once per recipient, so listing both an age key and a KMS key means either path can decrypt. The developer works with age locally, the cluster decrypts through KMS, and no single lost key is fatal.</li>

<li><strong>Know your re-key procedure before you need it.</strong> For SOPS that is <code>sops updatekeys</code> across every file matched by the rule. For Sealed Secrets it means re-sealing everything against the new certificate. Neither is hard. Both are miserable to work out under pressure.</li>
</ul>



<h2 class="wp-block-heading">Stopping plaintext from reaching the repo in the first place</h2>



<p class="wp-block-paragraph">All of the above assumes the encryption step happens. The commit that leaks is always the one where somebody skipped it, usually while debugging at the end of a long day.</p>



<ol class="wp-block-list">
<li><strong>Put a scanner in the pre-commit hook.</strong> <code>gitleaks</code> and <code>trufflehog</code> both work well here. Local hooks are bypassable, which is fine, because their job is catching accidents rather than stopping a determined person.</li>

<li><strong>Run the same scan in CI, on the whole history, not just the diff.</strong> The diff-only scan misses everything that predates the day you added the scanner.</li>

<li><strong>Turn on server-side push protection.</strong> GitHub&#8217;s push protection rejects known credential patterns at push time. GitLab and platforms like GitGuardian offer equivalents. This is the layer that actually holds, because it does not depend on anyone&#8217;s local setup.</li>

<li><strong>Make the plaintext path short-lived by construction.</strong> Never write a plaintext manifest to a tracked path. Use <code>/tmp</code>, or pipe straight into the encrypt step so the value never lands on disk at all.</li>

<li><strong>Add <code>*.dec.yaml</code>, <code>secret.yaml</code> and friends to <code>.gitignore</code> as a convention.</strong> Not a control, just one less way to be careless.</li>

<li><strong>Remember your shell history.</strong> A <code>--from-literal</code> with a real password ends up in <code>~/.bash_history</code> in cleartext. Use <code>--from-file</code>, or read from a variable, or prefix the command with a space if your shell is configured to skip those.</li>
</ol>



<h2 class="wp-block-heading">Troubleshooting the four things that go wrong</h2>



<h3 class="wp-block-heading">The SealedSecret applies cleanly but no Secret appears</h3>



<p class="wp-block-paragraph">Almost always a scope mismatch. The resource was sealed for one name or namespace and applied to another. The API server accepts the custom resource happily; the controller is where the failure surfaces.</p>



<pre class="wp-block-code"><code># The controller logs are the only place this error lives.
kubectl logs -n kube-system deploy/sealed-secrets --tail=50

# Confirm what the CRD actually thinks its name and namespace are.
kubectl get sealedsecret db-auth -n production -o yaml</code></pre>



<p class="wp-block-paragraph">The second common cause: the file was sealed against a certificate from a different cluster, or from before a key rotation. Re-fetch the cert and re-seal.</p>



<h3 class="wp-block-heading">The ExternalSecret sits there doing nothing</h3>



<p class="wp-block-paragraph">The status conditions carry the real message, and the events under <code>describe</code> usually name the provider error directly.</p>



<pre class="wp-block-code"><code>kubectl get externalsecret db-auth -n production
kubectl describe externalsecret db-auth -n production
kubectl describe clustersecretstore vault-backend</code></pre>



<p class="wp-block-paragraph">Check the store before the secret. If the <code>SecretStore</code> is not <code>Ready</code>, nothing referencing it will ever sync, and the error you want is on the store, not the <code>ExternalSecret</code>. When using a <code>ClusterSecretStore</code>, remember that any secret reference inside it needs an explicit <code>namespace</code>, because a cluster-scoped object has no namespace of its own to fall back on.</p>



<h3 class="wp-block-heading">Argo CD or Flux reports permanent drift</h3>



<p class="wp-block-paragraph">Two usual causes. Either an API version conversion is rewriting stored objects to a version your Git manifests do not use, as with the ESO <code>v1beta1</code> removal above, or a controller is adding fields to the generated Secret that your reconciler then tries to remove. Compare the live object against the rendered manifest field by field before you start changing anything.</p>



<h3 class="wp-block-heading">SOPS decryption fails only in the cluster</h3>



<p class="wp-block-paragraph">You encrypted for one recipient and the cluster holds a different key. This is the classic outcome of listing multiple recipients in <code>.sops.yaml</code> but running <code>sops --encrypt</code> with an explicit <code>--age</code> flag that overrides the rule. Decrypt locally to prove your key works, then check which recipients are actually listed in the file&#8217;s SOPS metadata block. The recipient list is stored in the file, in the clear, which makes this a fast thing to diagnose.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Rewriting history before rotating the credential, and treating the rewrite as the remediation.</li>

<li>Assuming a private repo is safe enough. Private repos get forked internally, cloned onto laptops, and made public by accident.</li>

<li>Committing the encrypted file and the plaintext source next to it, because the encrypt step wrote to a new filename and nobody deleted the original.</li>

<li>Never backing up the sealing key or the age identity.</li>

<li>Leaving encryption at rest off, so the credential moves from a Git leak to an etcd snapshot leak.</li>

<li>Encrypting the entire manifest with SOPS instead of just <code>data</code> and <code>stringData</code>, and losing every useful diff.</li>

<li>Using one age key or one sealing controller for all environments, so a staging compromise reaches production ciphertext.</li>

<li>Piping a Helm release&#8217;s rendered output into a debug artefact in CI. Helm renders secrets in the clear, and CI artefacts are frequently world-readable inside an organisation.</li>
</ul>



<h2 class="wp-block-heading">How I would actually decide</h2>



<p class="wp-block-paragraph">Skip the feature matrix and answer three questions in order.</p>



<ol class="wp-block-list">
<li><strong>Do you already run a secret store?</strong> If Vault or a cloud secret manager is already in your environment and someone owns it, use External Secrets Operator. You get rotation and audit for almost no additional conceptual load, and the argument is over.</li>

<li><strong>Do you have config outside Kubernetes?</strong> Terraform variables, Ansible vars, application <code>.env</code> files. If yes, SOPS with age, because one tool covers all of it and the per-path rules give you environment separation for free.</li>

<li><strong>Neither?</strong> Sealed Secrets. One cluster, a handful of secrets, a team that wants this solved this afternoon. It is the shortest distance to a repo with no plaintext in it, and migrating away later is straightforward because the plaintext values are recoverable from a running cluster.</li>
</ol>



<p class="wp-block-paragraph">Whichever you land on, the cluster-side work is the same and it is not optional: encryption at rest, tight RBAC on secrets, files rather than environment variables, and a tested backup of whatever key material the scheme depends on. On a managed control plane from a provider like DigitalOcean, Linode or a hyperscaler, the encryption-at-rest configuration is partly theirs and partly yours, so read the specific documentation rather than trusting a general answer. On a self-managed cluster running on your own VPS instances, for instance a k3s setup on InterServer or similar, all of it is yours and none of it is on by default.</p>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Is base64 in a Kubernetes Secret a form of encryption?</h3>



<p class="wp-block-paragraph">No. Base64 is an encoding that exists so binary values survive YAML parsing. It is reversed with a single command and provides no confidentiality whatsoever. Anyone reading the manifest reads the secret.</p>



<h3 class="wp-block-heading">Should I use Sealed Secrets or External Secrets Operator?</h3>



<p class="wp-block-paragraph">If you already run Vault, AWS Secrets Manager or an equivalent, use External Secrets Operator: you get rotation without commits and a proper audit trail. If you have no secret store and no appetite to run one, Sealed Secrets is a faster path with fewer moving parts. The trade-off is that Sealed Secrets has no rotation story and its keys are per-cluster.</p>



<h3 class="wp-block-heading">What happens if I lose the Sealed Secrets private key?</h3>



<p class="wp-block-paragraph">Every <code>SealedSecret</code> in your repository becomes permanently undecryptable, and you regenerate every credential from source. This is the single most common way this setup fails in practice. Back the key up somewhere outside the cluster and the repo, and test restoring it.</p>



<h3 class="wp-block-heading">Do I still need to rewrite Git history after rotating a leaked credential?</h3>



<p class="wp-block-paragraph">Often not. Once the credential is revoked it cannot be used, which usually resolves the actual risk. Rewriting is worth the disruption when the value is not rotatable, when it is personal data rather than a credential, or when a compliance process requires it. Weigh it against breaking every open pull request and forcing everyone to re-clone.</p>



<h3 class="wp-block-heading">Can Argo CD decrypt SOPS files on its own?</h3>



<p class="wp-block-paragraph">Not natively, unlike Flux, which has SOPS decryption built into its Kustomize controller. With Argo CD you add a config management plugin such as KSOPS or the Argo CD Vault Plugin. Check the current Argo CD documentation for the supported plugin mechanism before building around it.</p>



<h3 class="wp-block-heading">Is it safe to commit the Sealed Secrets certificate or the age public key?</h3>



<p class="wp-block-paragraph">Yes, and committing them is the point. Both are public halves of an asymmetric pair and can only encrypt. Putting them in the repo is what lets developers seal new secrets without cluster access. The private halves never go anywhere near Git.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">If you take one idea away, make it this: the work to <strong>keep Kubernetes secrets out of Git</strong> and the work to keep them safe in the cluster are two separate jobs, and finishing the first one feels a lot like finishing both. It is not. A repo full of <code>SealedSecret</code> resources sitting in front of an unencrypted etcd with permissive RBAC has moved the exposure, not removed it.</p>



<p class="wp-block-paragraph">Pick the approach that matches what you already run. Back up the key on day one, not after the first rebuild. Turn on encryption at rest. And when something does leak, rotate before you reach for <code>filter-repo</code>, because rotation is the part that actually stops the bleeding.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help sorting out secrets in your cluster?</h2>



<p class="wp-block-paragraph">This is work I do regularly, and most of it is less dramatic than it sounds once someone has done it before. Things I can help with:</p>



<ul class="wp-block-list">
<li>Auditing a GitOps repo for committed credentials, including full history, and producing a prioritised rotation list rather than a wall of scanner output</li>

<li>Setting up Sealed Secrets, SOPS with age, or External Secrets Operator against Vault or a cloud secret manager, including the key backup and restore procedure</li>

<li>Migrating an existing repo off plaintext or off a scheme that no longer fits, without a big-bang cutover</li>

<li>Fixing Argo CD or Flux reconciliation loops caused by secret controllers and API version conversions</li>

<li>Enabling and verifying etcd encryption at rest, and tightening RBAC on secrets across namespaces</li>

<li>Adding pre-commit and CI secret scanning that people will not immediately disable because of false positives</li>
</ul>



<p class="wp-block-paragraph">If you have a controller log, a stuck <code>ExternalSecret</code>, or a scanner report you are not sure how to triage, send it over and I will tell you what I make of it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/keep-kubernetes-secrets-out-of-git/">Keep Kubernetes Secrets Out of Git Without Kidding Yourself</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
