<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>HashiCorp Vault | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/hashicorp-vault/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/hashicorp-vault/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Tue, 03 Nov 2026 07:06:02 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>HashiCorp Vault | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/hashicorp-vault/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>AWS Secrets Manager vs Parameter Store: Stop Deciding on Price</title>
		<link>https://john-nessime.com/blog/devops/aws-secrets-manager-vs-parameter-store/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 03 Nov 2026 07:06:02 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Amazon ECS]]></category>
		<category><![CDATA[Amazon RDS]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[AWS Secrets Manager]]></category>
		<category><![CDATA[CloudTrail]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[HashiCorp Vault]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[Infrastructure as Code]]></category>
		<category><![CDATA[Key Rotation]]></category>
		<category><![CDATA[Least Privilege]]></category>
		<category><![CDATA[Parameter Store]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Terraform]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=442</guid>

					<description><![CDATA[<p>Most teams pick between AWS Secrets Manager and SSM Parameter Store on price, then get bitten months later by rotation semantics, a shared throughput quota, or a recovery window that blocks a redeploy. Here are the four levers that actually decide it, per value, plus a decision procedure you can run against a real inventory.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-secrets-manager-vs-parameter-store/">AWS Secrets Manager vs Parameter Store: Stop Deciding on Price</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The pull request comment was one line: why is the database password in Parameter Store and the Stripe key in Secrets Manager?</p>



<p class="wp-block-paragraph">Nobody had a good answer. Both were added in a hurry, months apart, by different people, and each choice made sense on the day it was made. That is how most AWS accounts end up with secrets in two places. Not by design, by accumulation.</p>



<p class="wp-block-paragraph">The comparison usually gets framed as a pricing question: Parameter Store&#8217;s standard tier has no storage charge, Secrets Manager bills per secret, so use Parameter Store until somebody complains. That framing is wrong, and it stays invisible for months before it costs you anything. What separates these two services is what happens when a value <em>changes</em>, not what it costs to sit there.</p>



<p class="wp-block-paragraph">This post covers the failure modes that actually bite, the four levers that decide <strong>AWS Secrets Manager vs Parameter Store</strong> for any given value, a decision procedure you can run against a real inventory, and the arguments about this that do not survive contact with a running system.</p>



<h2 class="wp-block-heading">The two failure modes that actually bite</h2>



<h3 class="wp-block-heading">Rotation that succeeds while your application quietly breaks</h3>



<p class="wp-block-paragraph">Secrets Manager&#8217;s single user rotation strategy changes the password on the existing database user. AWS documents a short window between the password changing in the database and the secret being updated, during which calls using the rotated credentials can be denied. Retry logic covers that window.</p>



<p class="wp-block-paragraph">That is the documented gap. The one that catches people is quieter.</p>



<p class="wp-block-paragraph">Your application fetched the credential once at boot, cached it in a module-level variable, and never looked again. Rotation completes. The rotation function logs success. Nothing breaks. Then a container recycles, picks up the new value and works fine, while older containers keep running on connections opened before the change. It looks healthy right up until a connection pool churns and part of your fleet starts failing authentication against a service you have not deployed in weeks.</p>



<p class="wp-block-paragraph">The alternating users strategy exists for exactly this. Instead of changing one user&#8217;s password, Secrets Manager clones the user and alternates which one it updates, so a valid credential exists on both sides of every change. The price is a second database user and a separate superuser secret with permission to do the cloning. AWS recommends single user for most cases and for interactive or one-off credentials, and alternating users where the application needs high availability.</p>



<p class="wp-block-paragraph">The rule I use: if consumers cache and you rotate on a schedule, you need alternating users or an explicit refresh path. A shorter cache TTL narrows the window, it does not close it.</p>



<h3 class="wp-block-heading">A throughput ceiling you do not own</h3>



<p class="wp-block-paragraph">Parameter Store read throughput is a quota on the account and Region, not on the parameter and not on the caller. Every cold start, every task boot, every CI job in that account draws from the same budget. The default is sized for low-volume workloads, and there is an opt-in higher throughput setting that raises the ceiling for an additional charge.</p>



<p class="wp-block-paragraph">The failure presents as an application bug. A scale-out event or a chatty pipeline fans out, <code>GetParameter</code> starts returning <code>ThrottlingException</code>, and services fail at boot for reasons unrelated to the code anybody just shipped. Nothing in the console tells you that you are near the line.</p>



<p class="wp-block-paragraph">Both services have quotas. The difference is blast radius: Secrets Manager throttling is annoying, Parameter Store throttling is a shared resource problem where one team&#8217;s new batch job degrades another team&#8217;s deploys. Alarm on throttling for both before you need to. CloudWatch does it natively, and if your dashboards live in Grafana Cloud or a self-hosted Grafana, pull the metric across so the alert lands where people actually look.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Where Parameter Store wins, and where it hurts</h2>



<p class="wp-block-paragraph">Parameter Store is not primarily a secrets store. It is a hierarchical configuration store that encrypts when you ask it to. Hold that framing and its behaviour becomes predictable.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong></p>



<ul class="wp-block-list">
<li><strong>Configuration that is not confidential.</strong> AMI IDs, feature flags, queue names, tuning values. Putting these in Secrets Manager means paying a per-secret charge for data with no confidentiality requirement.</li>



<li><strong>Hierarchies and bulk reads.</strong> Path-based naming plus <code>GetParametersByPath</code> pulls a whole subtree in one call, which is what a service wants at startup.</li>



<li><strong>No storage charge on the standard tier,</strong> so nobody is incentivized to cram unrelated values together to save money.</li>



<li><strong>Parameter policies and AWS public parameters</strong> live here, including read-only values AWS publishes for things like current AMI IDs.</li>
</ul>



<p class="wp-block-paragraph"><strong>Where it hurts:</strong></p>



<ul class="wp-block-list">
<li><strong>No rotation.</strong> You write the function, schedule it, and own verification and rollback yourself.</li>



<li><strong>CloudFormation cannot create an encrypted parameter.</strong> The <code>AWS::SSM::Parameter</code> resource accepts <code>String</code> and <code>StringList</code> only. You can reference an existing encrypted parameter through the <code>ssm-secure</code> dynamic reference, and only on the resource properties that support that pattern.</li>



<li><strong>No recovery window.</strong> Delete is delete, and parameter history does not survive deleting the parameter.</li>



<li><strong>Cross-account sharing requires the advanced tier,</strong> through Resource Access Manager.</li>
</ul>



<h2 class="wp-block-heading">Where AWS Secrets Manager wins, and where it hurts</h2>



<p class="wp-block-paragraph">Secrets Manager is a lifecycle service. Storage is the least interesting thing it does.</p>



<p class="wp-block-paragraph"><strong>Where it wins:</strong></p>



<ul class="wp-block-list">
<li><strong>Managed rotation</strong> for Amazon RDS, Aurora, Redshift and DocumentDB, using AWS-published rotation functions and two documented strategies.</li>



<li><strong>Versioning with staging labels.</strong> <code>AWSCURRENT</code>, <code>AWSPENDING</code> and <code>AWSPREVIOUS</code> give rotation a defined state machine and give you a last known good version to fall back to.</li>



<li><strong>A recovery window on delete,</strong> between seven and thirty days, defaulting to thirty.</li>



<li><strong>Cross-region replication and resource policies</strong> as first-class features, so multi-Region and cross-account access are properties of the secret rather than things you build.</li>
</ul>



<p class="wp-block-paragraph"><strong>Where it hurts:</strong></p>



<ul class="wp-block-list">
<li><strong>Cost scales by multiplication.</strong> Storage bills per secret per month, prorated hourly, and every cross-region replica bills as its own secret. The multiplier is secrets times environments times Regions.</li>



<li><strong>The recovery window is a deployment trap.</strong> Destroy an environment, apply again, and the create fails because the name is still reserved.</li>



<li><strong>Managed rotation only covers databases.</strong> Anything else means writing your own function, so you carry the operational burden without the free tier.</li>



<li><strong>The pricing model pushes you toward bundling.</strong> One secret per value is the clean design and the expensive one, so teams pack fields into JSON and lose per-value IAM granularity.</li>
</ul>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Four levers that decide AWS Secrets Manager vs Parameter Store</h2>



<h3 class="wp-block-heading">1. Does the value have to change on a schedule?</h3>



<p class="wp-block-paragraph">If yes, and it is a credential for RDS, Aurora, Redshift or DocumentDB, use Secrets Manager and stop deliberating. You are not paying for storage. You are paying to avoid writing, testing and maintaining a rotation function, and to inherit a staging-label state machine that makes a failed rotation recoverable instead of an outage.</p>



<p class="wp-block-paragraph">If the value never changes on a schedule, an AMI ID, a webhook URL, a vendor key you replace manually once a year, then rotation machinery buys you nothing.</p>



<h3 class="wp-block-heading">2. How many independent gates guard the read path?</h3>



<p class="wp-block-paragraph">This is the lever people skip, and it matters more than the service choice.</p>



<p class="wp-block-paragraph">Both services encrypt through KMS. The question is whether KMS is a second, independent authorization decision or a formality. An encrypted parameter under the AWS managed key for Systems Manager is decrypted through that key on behalf of the service, so in practice the SSM permission is the gate. Encrypt under a customer managed key instead and the caller must also be allowed <code>kms:Decrypt</code> on that specific key. Now two policies in two services both have to say yes, and you can scope the key policy to a named set of roles independently of who may call Systems Manager. The first gate protects the API surface, the second protects the plaintext.</p>



<p class="wp-block-paragraph">The same split exists in Secrets Manager, between the AWS managed key and one you create.</p>



<pre class="wp-block-code"><code># Store an encrypted parameter under a customer managed key
# instead of the AWS managed default.
aws ssm put-parameter 
  --name "/prod/billing/db-password" 
  --type SecureString 
  --key-id "alias/prod-app-secrets" 
  --value "REPLACE_ME" 
  --overwrite

# Reading it now needs ssm:GetParameter AND
# kms:Decrypt on alias/prod-app-secrets.
aws ssm get-parameter 
  --name "/prod/billing/db-password" 
  --with-decryption 
  --query "Parameter.Value" 
  --output text</code></pre>



<p class="wp-block-paragraph">There is a second payoff. A customer managed key makes the decrypt call attributable in CloudTrail against a key policy you control, which turns &#8220;someone read a secret&#8221; into &#8220;this role decrypted this value with this key at this time&#8221;. That is usually what an auditor is asking for, and it is independent of which store you land on.</p>



<h3 class="wp-block-heading">3. What does the retrieval path look like under load?</h3>



<p class="wp-block-paragraph">Neither service is designed to be called on every request. Both punish you for it in different currencies: Parameter Store with a shared throughput ceiling, Secrets Manager with a per-call charge.</p>



<p class="wp-block-paragraph">The answer for both is caching, and AWS ships the mechanism rather than making you build it. The Secrets Manager Agent, now published under the name AWS Workload Credentials Provider, runs as a sidecar or local service, holds secrets in memory, and serves them to your application over a local HTTP endpoint, so the app makes one call instead of thousands and needs no AWS SDK dependency to do it. For Lambda, the AWS Parameters and Secrets Lambda Extension does the equivalent inside the execution environment and covers both services.</p>



<p class="wp-block-paragraph">Caching moves the problem rather than removing it. A cached credential is a stale credential during rotation, which loops straight back to lever one.</p>



<h3 class="wp-block-heading">4. How does the value get created and destroyed?</h3>



<p class="wp-block-paragraph">Creation first. CloudFormation cannot create an encrypted parameter, so if your pipeline is CloudFormation or CDK and the value must be encrypted from the moment it exists, Secrets Manager is the path of least resistance. The usual workarounds, a Lambda-backed custom resource or creating the parameter out of band, both move the secret outside the template that claims to own it. That is a real cost, not a detail.</p>



<p class="wp-block-paragraph">Destruction is where the surprise lives. Secrets Manager schedules deletion after a recovery window with a minimum of seven days and a default of thirty, and the name stays reserved for the duration.</p>



<pre class="wp-block-code"><code># Shorten the recovery window to the documented minimum.
aws secretsmanager delete-secret 
  --secret-id prod/billing/db 
  --recovery-window-in-days 7

# Or skip the window entirely. There is no undo after this.
aws secretsmanager delete-secret 
  --secret-id ephemeral/pr-1482/db 
  --force-delete-without-recovery</code></pre>



<p class="wp-block-paragraph">AWS notes the actual deletion runs as an asynchronous background process, so deleting and immediately recreating a secret with the same name needs backoff and retry even with the force flag. This is why per-branch environments built with Terraform or OpenTofu fail on their second run. Use the force flag deliberately, put a unique suffix in the name, or keep ephemeral values in Parameter Store where deletion is immediate.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Cost levers, without quoting numbers that will age</h2>



<p class="wp-block-paragraph">Published rates change. The billing mechanisms are stable, and the mechanisms are what let you predict the bill.</p>



<ul class="wp-block-list">
<li><strong>Parameter Store standard tier</strong> carries no storage charge. The advanced tier is charged per parameter, and the higher throughput setting is charged separately as an account and Region level switch.</li>



<li><strong>Secrets Manager</strong> charges per stored secret per month, prorated hourly, plus a charge per block of API calls. Replicas count as separate stored secrets.</li>



<li><strong>KMS</strong> charges a monthly fee per customer managed key plus a request charge on cryptographic operations. This applies to both services and it is the line most estimates forget.</li>



<li><strong>Rotation</strong> has no separate line item, but the rotation function bills as ordinary Lambda usage.</li>
</ul>



<p class="wp-block-paragraph">Two consequences. The Secrets Manager multiplier is not the number of secrets, it is secrets across environments across Regions, and replicas are the sneaky term because nobody remembers enabling them. And the advanced parameter tier is a one-way door: you can promote a standard parameter to advanced, but you cannot demote it, because reverting would truncate the value. Undoing that means delete and recreate.</p>



<p class="wp-block-paragraph">If you cannot see which term is growing, break the bill down by service and tag before you migrate anything. Cost Explorer handles the single-account case; for cross-account views without building the aggregation yourself, tools like Vantage or CloudZero will answer &#8220;is this secret count, replication, or retrieval frequency&#8221; in a few minutes. That is a better use of an afternoon than guessing.</p>



<h2 class="wp-block-heading">A decision procedure you can run today</h2>



<p class="wp-block-paragraph">Take an actual list of values and walk each one through this. It takes about a minute each.</p>



<ol class="wp-block-list">
<li><strong>Is it confidential at all?</strong> If no, Parameter Store, plain <code>String</code> type, done. Do not encrypt configuration for the aesthetics.</li>



<li><strong>Is it a database credential that should rotate on a schedule?</strong> Secrets Manager with managed rotation. Choose single user or alternating users based on whether consumers cache.</li>



<li><strong>Does it need scheduled rotation but is not a supported database?</strong> You are writing a function either way. Secrets Manager if you want versioning, staging labels and a recovery window; Parameter Store if the free tier matters more and you accept owning the lifecycle.</li>



<li><strong>Does it need to exist in more than one Region or account?</strong> Secrets Manager gives you replication and resource policies directly. Parameter Store shares across accounts through Resource Access Manager, advanced tier only.</li>



<li><strong>Does CloudFormation or CDK create it?</strong> Encrypted parameters are not creatable there, so it is Secrets Manager or an accepted out-of-band step.</li>



<li><strong>Is it read frequently?</strong> Cache it either way, then return to step two, because caching and rotation interact and that interaction is where outages come from.</li>



<li><strong>Everything else</strong> goes to Parameter Store, encrypted under a customer managed key.</li>
</ol>



<p class="wp-block-paragraph">Most accounts land with the large majority of values in Parameter Store and a short list of rotating database credentials in Secrets Manager. That split is healthy. What is not healthy is being unable to say why a particular value is where it is.</p>



<p class="wp-block-paragraph">Once you accept you will use both, the operational work is making them look like one system: one naming convention across both stores, IAM scoped to the path rather than the service, KMS keys separated by trust boundary rather than by service, and the rule written down where the next person will find it. Two lines in a repository README beats a policy document nobody opens.</p>



<h2 class="wp-block-heading">Arguments that do not survive contact</h2>



<p class="wp-block-paragraph"><strong>&#8220;Parameter Store is free, so use it for everything.&#8221;</strong> The standard tier has no storage charge, which is not the same as free. KMS requests bill, the higher throughput setting bills, advanced parameters bill, and the rotation function you write instead of using a managed one has a real cost in engineering hours and incident risk.</p>



<p class="wp-block-paragraph"><strong>&#8220;Secrets Manager is expensive.&#8221;</strong> Per secret it is small. The bill grows through multiplication, not unit price. Before migrating anything, count secrets across environments and Regions and find out how many replicas exist that nobody asked for. Deleting unused replicas is usually a bigger win than a migration project.</p>



<p class="wp-block-paragraph"><strong>&#8220;Just use Vault and skip the argument.&#8221;</strong> HashiCorp Vault is a better answer if you are multi-cloud, need dynamic short-lived credentials, or want secrets engines AWS does not offer. It also hands you a cluster to run, upgrade, seal and unseal, back up and monitor, plus a licensing decision. On a single-cloud AWS estate with a handful of rotating credentials, the AWS services are less capable and less work. Both statements are true at once, and which dominates depends on how much platform team you have.</p>



<p class="wp-block-paragraph"><strong>&#8220;It is encrypted at rest, so it is secure.&#8221;</strong> Encryption at rest protects against someone reading the storage layer, which is not the threat you have. Your threat is an over-permissive role and a compromised workload. That is an IAM policy and a KMS key policy question, and the answer is the same on either service.</p>



<p class="wp-block-paragraph"><strong>&#8220;We will add rotation later.&#8221;</strong> Rotation is not a feature flag, it is a coordination problem between the store, the target system and every consumer&#8217;s caching behaviour. Deciding that a credential will eventually rotate changes how you write the consumer today. Retrofitting it means touching every consumer anyway.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I migrate a value between the two without downtime?</h3>



<p class="wp-block-paragraph">Yes, with a dual read. Write the value to the new location, deploy consumers that read the new location and fall back to the old one, then use CloudTrail to confirm nothing still reads the old path before deleting it. Do not skip the CloudTrail step; the consumer you forgot is always a job that runs monthly. If the value passed through a local file during the move, erase that file properly rather than with <code>rm</code>, using <code>shred</code> on Linux or a secure deletion tool such as those from O&amp;O Software on Windows.</p>



<h3 class="wp-block-heading">Does Parameter Store support rotation?</h3>



<p class="wp-block-paragraph">Not as a built-in feature. You write a function and invoke it on a schedule, typically with an EventBridge rule. You also inherit everything managed rotation handles for you: verifying the new credential works before committing to it, rolling back cleanly when it does not, and giving consumers an overlap window.</p>



<h3 class="wp-block-heading">Which one should I use for third-party API keys?</h3>



<p class="wp-block-paragraph">It depends on whether you rotate them. Most vendor keys get rotated manually and rarely, in which case an encrypted parameter under a customer managed key is enough. If the provider supports programmatic key rotation and you want that automated, Secrets Manager&#8217;s version staging is what makes the swap safe.</p>



<h3 class="wp-block-heading">Can workloads outside AWS read from either service?</h3>



<p class="wp-block-paragraph">Both, over the API, with AWS credentials. That means a long-lived access key or a federated identity, and you should federate wherever the platform supports it. If part of your estate runs on VPS hosts at a provider like Contabo or InterServer, this is the boundary to think hardest about, because you have just placed an AWS credential on a machine whose network you do not control. Scope the role narrowly, apply condition keys where the network path allows, and treat that credential as the one most likely to leak.</p>



<h3 class="wp-block-heading">What happens if I delete something by accident?</h3>



<p class="wp-block-paragraph">Secrets Manager lets you restore during the recovery window, which is the single best argument for putting genuinely important credentials there. Parameter Store has no equivalent. Once the parameter is gone your options are whatever exists in state files, backups or someone&#8217;s terminal history, and none of those is a plan.</p>



<h3 class="wp-block-heading">Should ECS or EKS workloads pull secrets differently?</h3>



<p class="wp-block-paragraph">ECS can inject values from either service directly into a task&#8217;s environment, which removes SDK code from the container but reads the value at task start, so a rotation does not reach a running task. On EKS you have both the Secrets Store CSI driver with the AWS provider, which mounts values as files, and the agent sidecar pattern, which serves them over localhost and can refresh without restarting the pod. Choose based on whether your application can reload a credential mid-life. If it cannot, mounting and injection are equivalent, and rotation means a rollout.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">The <strong>AWS Secrets Manager vs Parameter Store</strong> decision is a lifecycle decision wearing a pricing costume. What separates the two is not what they store, it is what they do when the value changes, when it gets deleted, when a Region falls over, and when ten thousand containers ask for it at once.</p>



<p class="wp-block-paragraph">Decide per value, not per project. Configuration and rarely-changing keys belong in Parameter Store. Database credentials that rotate on a schedule belong in Secrets Manager. Put a customer managed key in front of both so the second gate does real work, cache reads so you are not fighting a quota, and make sure whoever wrote the caching understands what happens the next time a rotation runs. If you cannot explain why a given value lives where it does, that is the value that will surprise you.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your secrets setup?</h2>



<p class="wp-block-paragraph">Most of the work here is not choosing a service, it is untangling what already exists. Things I can help with:</p>



<ul class="wp-block-list">
<li>Auditing an AWS account for where secrets actually live, which roles can decrypt them, and which values are duplicated across environments or replicated into Regions nobody uses</li>



<li>Designing a naming hierarchy and path-scoped IAM so one policy covers a service instead of one policy per value</li>



<li>Setting up managed rotation for RDS and Aurora credentials, including the alternating users pattern when the application caches and cannot be changed</li>



<li>Writing and testing custom rotation functions for non-database secrets, with the rollback path actually exercised rather than assumed</li>



<li>Moving values between the two stores with a dual-read cutover and CloudTrail verification that nothing is left reading the old path</li>



<li>Working out whether a growing Secrets Manager line item is secret count, replication or retrieval frequency, and fixing the one actually responsible</li>
</ul>



<p class="wp-block-paragraph">Send me a redacted IAM policy, a CloudTrail query result, or the output of listing your parameters by path, and I will tell you what I would change first and why.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-secrets-manager-vs-parameter-store/">AWS Secrets Manager vs Parameter Store: Stop Deciding on Price</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Keep Kubernetes Secrets Out of Git Without Kidding Yourself</title>
		<link>https://john-nessime.com/blog/devops/keep-kubernetes-secrets-out-of-git/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 09 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Kubernetes]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Argo CD]]></category>
		<category><![CDATA[CI/CD]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[External Secrets Operator]]></category>
		<category><![CDATA[Flux CD]]></category>
		<category><![CDATA[Git]]></category>
		<category><![CDATA[GitOps]]></category>
		<category><![CDATA[HashiCorp Vault]]></category>
		<category><![CDATA[Infrastructure as Code]]></category>
		<category><![CDATA[Sealed Secrets]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[SOPS]]></category>
		<category><![CDATA[Supply Chain Security]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=174</guid>

					<description><![CDATA[<p>Encrypting a Secret before you commit it only closes one of the five places that value comes to rest. Here is how to keep Kubernetes secrets out of Git properly: Sealed Secrets, SOPS and External Secrets compared honestly, the key-backup problem nobody plans for, and what to actually do when a credential is already in your history.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/keep-kubernetes-secrets-out-of-git/">Keep Kubernetes Secrets Out of Git Without Kidding Yourself</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The message usually arrives on a quiet afternoon. A secret scanner has flagged your infrastructure repo, and the finding is a Postgres connection string sitting in a <code>values.yaml</code> that someone committed a long time ago and deleted a few weeks later. The file is gone from the working tree. The commit is not.</p>



<p class="wp-block-paragraph">The reflex at that point is to reach for history rewriting. That is the wrong first move, and I will come back to why. But the bigger problem is the one that shows up six months after you clean up: teams migrate to Sealed Secrets or SOPS, tick the box, and quietly assume the whole class of problem is now solved. It is not. Encrypting a value before it enters Git closes exactly one door out of about five.</p>



<p class="wp-block-paragraph">This post covers how to <strong>keep Kubernetes secrets out of Git</strong> in a way that survives contact with a real cluster: the three families of solution and where each one genuinely wins, the disaster-recovery trap that bites hardest, how to stop plaintext reaching the repo in the first place, and what to actually do about credentials already in your history.</p>



<h2 class="wp-block-heading">The failure that actually bites: encrypted in Git, wide open in the cluster</h2>



<p class="wp-block-paragraph">Sealed Secrets and SOPS both terminate in the same place. The controller decrypts, the API server accepts, and what lands in etcd is an ordinary Kubernetes Secret. Ordinary means base64-encoded, which is an encoding, not a cipher. Anyone who can run <code>kubectl get secret -o yaml</code> in that namespace reads the value in one pipe through <code>base64 -d</code>.</p>



<p class="wp-block-paragraph">So the honest description of what these tools buy you is narrower than the marketing suggests. They stop the credential appearing in a public object store with permanent history and a CDN in front of it. They do nothing about the cluster&#8217;s own blast radius.</p>



<p class="wp-block-paragraph">Three things need to be true alongside whichever tool you pick:</p>



<ul class="wp-block-list">
<li><strong>Encryption at rest is on.</strong> A stock Kubernetes control plane writes Secrets to etcd unencrypted unless you configure an <code>EncryptionConfiguration</code>, ideally with a KMS provider. Managed distributions vary. On EKS, envelope encryption is something you enable, not something you inherit. Check yours rather than assuming.</li>

<li><strong>RBAC on secrets is actually restrictive.</strong> A depressing number of clusters grant blanket <code>get</code> and <code>list</code> on secrets to service accounts that only ever needed one. That single verb turns a namespace compromise into a credential dump.</li>

<li><strong>Secrets are mounted as files, not shoved into environment variables where you can help it.</strong> Environment variables leak through <code>/proc/&lt;pid&gt;/environ</code>, through anything that dumps the environment on crash, and through every debug endpoint an application framework has ever shipped. A projected volume with tight permissions is the quieter option.</li>
</ul>



<p class="wp-block-paragraph">Quick sanity check on the RBAC side, which takes about ten seconds and surprises people:</p>



<pre class="wp-block-code"><code># Can the default service account in a namespace read every secret in it?
kubectl auth can-i get secrets 
  --namespace production 
  --as system:serviceaccount:production:default</code></pre>



<p class="wp-block-paragraph">If that returns <code>yes</code>, your Git hygiene is not the weakest link in the chain.</p>



<h2 class="wp-block-heading">Already leaked? Rotate first, rewrite second, and understand what rewriting does not do</h2>



<p class="wp-block-paragraph">Order matters here more than technique. GitHub&#8217;s own guidance is blunt about it: once a credential has been pushed, revoke or rotate it first, because that step alone removes the attacker&#8217;s ability to use it. History surgery is the optional follow-up, not the fix.</p>



<p class="wp-block-paragraph">The reason is that a force-push does not reach everywhere the commit went. Objects survive in forks, which inherit the full object graph at fork time. They survive in colleagues&#8217; clones. On GitHub they remain reachable by SHA through cached views and through pull requests that referenced them, and purging those caches is a support request, not a git command. If the repo was public even briefly, treat the credential as burned and move on.</p>



<p class="wp-block-paragraph">When you do rewrite, use <code>git filter-repo</code>. It has replaced <code>filter-branch</code> as the recommended tool and it is dramatically faster on any repo with real history.</p>



<pre class="wp-block-code"><code># Work on a bare mirror, never your day-to-day clone.
git clone --mirror git@github.com:org/infra.git infra-clean.git
cd infra-clean.git

# Strip one path from every commit that ever touched it.
git filter-repo --path clusters/production/values.yaml --invert-paths

# filter-repo deletes the origin remote on purpose, so you cannot
# force-push a half-finished rewrite by muscle memory. Add it back.
git remote add origin git@github.com:org/infra.git
git push --force --all
git push --force --tags</code></pre>



<p class="wp-block-paragraph">Then the unglamorous part: close or merge open pull requests before you start, because every SHA downstream of the rewrite changes. Tell every collaborator to delete their clone and clone again. A <code>git pull</code> will happily drag the old objects back in.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Two families of solution, and how to tell which one you are in</h2>



<p class="wp-block-paragraph">Every approach to this problem is either <em>encrypt the value and commit the ciphertext</em>, or <em>commit a pointer and fetch the value at runtime</em>. Everything else is implementation detail. The choice is mostly about whether you already run a secret store, and whether you can tolerate the CLI step in the middle of your workflow.</p>



<h3 class="wp-block-heading">Sealed Secrets: the lowest-friction way to start</h3>



<p class="wp-block-paragraph">A controller in the cluster holds an RSA private key and publishes the matching certificate. The <code>kubeseal</code> CLI encrypts a Secret manifest against that certificate and produces a <code>SealedSecret</code> custom resource. Only that controller can decrypt it, so the resulting YAML is safe to commit.</p>



<pre class="wp-block-code"><code># Fetch the controller's public certificate. This is safe to commit;
# it can only encrypt, never decrypt.
kubeseal --fetch-cert 
  --controller-name=sealed-secrets 
  --controller-namespace=kube-system 
  &gt; pub-cert.pem

# Build a Secret manifest locally. --dry-run=client means it is rendered
# and never sent to the API server.
kubectl create secret generic db-auth 
  --from-literal=password='&lt;value&gt;' 
  --dry-run=client -o yaml &gt; db-auth.yaml

# Encrypt against the cert. Offline, so no cluster access needed here.
kubeseal --cert pub-cert.pem --format yaml &lt; db-auth.yaml &gt; db-auth-sealed.yaml

rm db-auth.yaml</code></pre>



<p class="wp-block-paragraph">Because you can hand out <code>pub-cert.pem</code>, developers can seal secrets without any cluster credentials at all. That is the underrated part, and it is the reason this is the one I reach for first on a small team with no existing vault.</p>



<p class="wp-block-paragraph">Two details that catch people. First, sealing is scoped by default: a <code>SealedSecret</code> is bound to its exact name and namespace, and moving the file to a different namespace makes it undecryptable. That is deliberate, it stops a developer sealing a secret into a namespace they should not have access to, and you can relax it with <code>--scope namespace-wide</code> or <code>--scope cluster-wide</code> if you genuinely need to. Second, the Helm chart names the controller <code>sealed-secrets</code> while the CLI looks for <code>sealed-secrets-controller</code> by default, so <code>--controller-name</code> is not optional in practice.</p>



<p class="wp-block-paragraph">Where it stops being the right answer: multi-cluster. Each cluster generates its own key pair, so a sealed file is not portable, and a fleet means either copying private keys around or maintaining per-cluster ciphertext for the same value. Rotation is also manual. There is no dynamic secret story here at all.</p>



<h3 class="wp-block-heading">SOPS with age: encrypt files, not just Kubernetes objects</h3>



<p class="wp-block-paragraph">SOPS encrypts the values inside a structured file and leaves the keys readable, so a diff still tells you which field changed even if it cannot tell you what it changed to. Paired with <code>age</code>, which uses short X25519 keys instead of the GPG keyring experience, it is pleasant to live with.</p>



<p class="wp-block-paragraph">A <code>.sops.yaml</code> at the repo root drives everything through path matching, which is how you get per-environment key separation without anyone having to remember flags:</p>



<pre class="wp-block-code"><code>creation_rules:
  - path_regex: clusters/production/.*.yaml$
    encrypted_regex: '^(data|stringData)$'
    age: age1...   # production recipient
  - path_regex: clusters/staging/.*.yaml$
    encrypted_regex: '^(data|stringData)$'
    age: age1...   # staging recipient</code></pre>



<p class="wp-block-paragraph">The <code>encrypted_regex</code> line is the one worth understanding rather than copying. Without it SOPS encrypts everything including <code>metadata</code>, and you lose the readable-diff property that made this approach attractive. Restricting it to <code>data</code> and <code>stringData</code> keeps names, namespaces and labels reviewable in a pull request.</p>



<p class="wp-block-paragraph">Flux decrypts SOPS natively: you point a <code>Kustomization</code> at a Secret holding the age private key via <code>decryption.provider: sops</code> and it handles the rest. Argo CD does not have equivalent built-in support, so you bolt it on with a config management plugin such as KSOPS or the Argo CD Vault Plugin. Worth confirming against current Argo CD docs before you commit to it, because plugin mechanics there have changed more than once.</p>



<p class="wp-block-paragraph">SOPS also handles files that are not Kubernetes manifests: Terraform variable files, Ansible vars, plain <code>.env</code>. If you have config living outside the cluster, that reach is a real advantage over Sealed Secrets.</p>



<h3 class="wp-block-heading">External Secrets Operator: stop putting the value in Git at all</h3>



<p class="wp-block-paragraph">The third approach sidesteps encryption entirely. You commit a reference, and a controller resolves it against a real secret store: HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, Google Secret Manager, or one of the newer hosted options like Doppler, Infisical or 1Password. The repo contains a path, never a value.</p>



<pre class="wp-block-code"><code>apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: db-auth
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: db-auth
  data:
    - secretKey: password
      remoteRef:
        key: production/postgres
        property: password</code></pre>



<p class="wp-block-paragraph">This is the only one of the three that gives you rotation without a commit. Change the value in Vault, wait for <code>refreshInterval</code>, and the cluster Secret updates. It also gives you a real audit trail, because reads happen against a system built to log them.</p>



<p class="wp-block-paragraph">The cost is honest and worth stating. You have introduced a runtime dependency: if the store is unreachable, new workloads cannot get their credentials. You still have a bootstrap credential problem, because the operator needs to authenticate to the store somehow, and that root of trust has to get into the cluster by some other path. And Git is no longer the complete description of your system, which is a genuine philosophical cost if GitOps purity matters to you.</p>



<p class="wp-block-paragraph"><strong>One live upgrade trap.</strong> The <code>external-secrets.io/v1beta1</code> API was removed in ESO 0.17.0. Before that, in the 0.16 series, a conversion webhook served both versions and rewrote stored objects to <code>v1</code>, which produced permanent drift in Argo CD for anyone whose Git manifests still said <code>v1beta1</code>. Argo sees <code>v1</code> in the cluster, Git says <code>v1beta1</code>, and it reconciles forever. Update your manifests to <code>v1</code> before you cross that boundary, not after.</p>



<h3 class="wp-block-heading">Secrets Store CSI Driver: when a Secret object is one object too many</h3>



<p class="wp-block-paragraph">Worth knowing about even if you do not use it. The CSI driver mounts values from an external store directly into the pod as a volume, so no Kubernetes Secret object needs to exist at all. That closes the etcd exposure completely. It costs you the ability to consume the value as an environment variable, and it means the secret is only available to pods that mount it, which is either exactly what you want or a nuisance depending on the workload. Use it where the compliance posture demands that etcd never sees the value.</p>



<h2 class="wp-block-heading">The key you forgot to back up</h2>



<p class="wp-block-paragraph">This is the failure I would put money on, and it is the one nobody rehearses. Every encrypt-into-Git approach concentrates all your risk into one small piece of key material. Lose it and your repository turns into a directory of well-organised noise.</p>



<p class="wp-block-paragraph">Concretely: rebuild a cluster from scratch, install a fresh Sealed Secrets controller, and it generates a new key pair. Every <code>SealedSecret</code> in your repo is now undecryptable. Your GitOps repo, the one that was supposed to let you recreate everything, cannot recreate anything that needed a credential. You find this out during the incident, not before it.</p>



<ul class="wp-block-list">
<li><strong>Back up the sealing key out of band.</strong> The controller&#8217;s private key lives in a Secret in its namespace. Export it, encrypt it, and store it somewhere that is not the cluster and not the repo. A password manager or an offline copy is fine. Test the restore.</li>

<li><strong>Same for age.</strong> The private key is one line of text. Its size makes it feel unimportant. It is the entire thing.</li>

<li><strong>Add a second recipient for anything that matters.</strong> SOPS encrypts the data key once per recipient, so listing both an age key and a KMS key means either path can decrypt. The developer works with age locally, the cluster decrypts through KMS, and no single lost key is fatal.</li>

<li><strong>Know your re-key procedure before you need it.</strong> For SOPS that is <code>sops updatekeys</code> across every file matched by the rule. For Sealed Secrets it means re-sealing everything against the new certificate. Neither is hard. Both are miserable to work out under pressure.</li>
</ul>



<h2 class="wp-block-heading">Stopping plaintext from reaching the repo in the first place</h2>



<p class="wp-block-paragraph">All of the above assumes the encryption step happens. The commit that leaks is always the one where somebody skipped it, usually while debugging at the end of a long day.</p>



<ol class="wp-block-list">
<li><strong>Put a scanner in the pre-commit hook.</strong> <code>gitleaks</code> and <code>trufflehog</code> both work well here. Local hooks are bypassable, which is fine, because their job is catching accidents rather than stopping a determined person.</li>

<li><strong>Run the same scan in CI, on the whole history, not just the diff.</strong> The diff-only scan misses everything that predates the day you added the scanner.</li>

<li><strong>Turn on server-side push protection.</strong> GitHub&#8217;s push protection rejects known credential patterns at push time. GitLab and platforms like GitGuardian offer equivalents. This is the layer that actually holds, because it does not depend on anyone&#8217;s local setup.</li>

<li><strong>Make the plaintext path short-lived by construction.</strong> Never write a plaintext manifest to a tracked path. Use <code>/tmp</code>, or pipe straight into the encrypt step so the value never lands on disk at all.</li>

<li><strong>Add <code>*.dec.yaml</code>, <code>secret.yaml</code> and friends to <code>.gitignore</code> as a convention.</strong> Not a control, just one less way to be careless.</li>

<li><strong>Remember your shell history.</strong> A <code>--from-literal</code> with a real password ends up in <code>~/.bash_history</code> in cleartext. Use <code>--from-file</code>, or read from a variable, or prefix the command with a space if your shell is configured to skip those.</li>
</ol>



<h2 class="wp-block-heading">Troubleshooting the four things that go wrong</h2>



<h3 class="wp-block-heading">The SealedSecret applies cleanly but no Secret appears</h3>



<p class="wp-block-paragraph">Almost always a scope mismatch. The resource was sealed for one name or namespace and applied to another. The API server accepts the custom resource happily; the controller is where the failure surfaces.</p>



<pre class="wp-block-code"><code># The controller logs are the only place this error lives.
kubectl logs -n kube-system deploy/sealed-secrets --tail=50

# Confirm what the CRD actually thinks its name and namespace are.
kubectl get sealedsecret db-auth -n production -o yaml</code></pre>



<p class="wp-block-paragraph">The second common cause: the file was sealed against a certificate from a different cluster, or from before a key rotation. Re-fetch the cert and re-seal.</p>



<h3 class="wp-block-heading">The ExternalSecret sits there doing nothing</h3>



<p class="wp-block-paragraph">The status conditions carry the real message, and the events under <code>describe</code> usually name the provider error directly.</p>



<pre class="wp-block-code"><code>kubectl get externalsecret db-auth -n production
kubectl describe externalsecret db-auth -n production
kubectl describe clustersecretstore vault-backend</code></pre>



<p class="wp-block-paragraph">Check the store before the secret. If the <code>SecretStore</code> is not <code>Ready</code>, nothing referencing it will ever sync, and the error you want is on the store, not the <code>ExternalSecret</code>. When using a <code>ClusterSecretStore</code>, remember that any secret reference inside it needs an explicit <code>namespace</code>, because a cluster-scoped object has no namespace of its own to fall back on.</p>



<h3 class="wp-block-heading">Argo CD or Flux reports permanent drift</h3>



<p class="wp-block-paragraph">Two usual causes. Either an API version conversion is rewriting stored objects to a version your Git manifests do not use, as with the ESO <code>v1beta1</code> removal above, or a controller is adding fields to the generated Secret that your reconciler then tries to remove. Compare the live object against the rendered manifest field by field before you start changing anything.</p>



<h3 class="wp-block-heading">SOPS decryption fails only in the cluster</h3>



<p class="wp-block-paragraph">You encrypted for one recipient and the cluster holds a different key. This is the classic outcome of listing multiple recipients in <code>.sops.yaml</code> but running <code>sops --encrypt</code> with an explicit <code>--age</code> flag that overrides the rule. Decrypt locally to prove your key works, then check which recipients are actually listed in the file&#8217;s SOPS metadata block. The recipient list is stored in the file, in the clear, which makes this a fast thing to diagnose.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Rewriting history before rotating the credential, and treating the rewrite as the remediation.</li>

<li>Assuming a private repo is safe enough. Private repos get forked internally, cloned onto laptops, and made public by accident.</li>

<li>Committing the encrypted file and the plaintext source next to it, because the encrypt step wrote to a new filename and nobody deleted the original.</li>

<li>Never backing up the sealing key or the age identity.</li>

<li>Leaving encryption at rest off, so the credential moves from a Git leak to an etcd snapshot leak.</li>

<li>Encrypting the entire manifest with SOPS instead of just <code>data</code> and <code>stringData</code>, and losing every useful diff.</li>

<li>Using one age key or one sealing controller for all environments, so a staging compromise reaches production ciphertext.</li>

<li>Piping a Helm release&#8217;s rendered output into a debug artefact in CI. Helm renders secrets in the clear, and CI artefacts are frequently world-readable inside an organisation.</li>
</ul>



<h2 class="wp-block-heading">How I would actually decide</h2>



<p class="wp-block-paragraph">Skip the feature matrix and answer three questions in order.</p>



<ol class="wp-block-list">
<li><strong>Do you already run a secret store?</strong> If Vault or a cloud secret manager is already in your environment and someone owns it, use External Secrets Operator. You get rotation and audit for almost no additional conceptual load, and the argument is over.</li>

<li><strong>Do you have config outside Kubernetes?</strong> Terraform variables, Ansible vars, application <code>.env</code> files. If yes, SOPS with age, because one tool covers all of it and the per-path rules give you environment separation for free.</li>

<li><strong>Neither?</strong> Sealed Secrets. One cluster, a handful of secrets, a team that wants this solved this afternoon. It is the shortest distance to a repo with no plaintext in it, and migrating away later is straightforward because the plaintext values are recoverable from a running cluster.</li>
</ol>



<p class="wp-block-paragraph">Whichever you land on, the cluster-side work is the same and it is not optional: encryption at rest, tight RBAC on secrets, files rather than environment variables, and a tested backup of whatever key material the scheme depends on. On a managed control plane from a provider like DigitalOcean, Linode or a hyperscaler, the encryption-at-rest configuration is partly theirs and partly yours, so read the specific documentation rather than trusting a general answer. On a self-managed cluster running on your own VPS instances, for instance a k3s setup on InterServer or similar, all of it is yours and none of it is on by default.</p>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Is base64 in a Kubernetes Secret a form of encryption?</h3>



<p class="wp-block-paragraph">No. Base64 is an encoding that exists so binary values survive YAML parsing. It is reversed with a single command and provides no confidentiality whatsoever. Anyone reading the manifest reads the secret.</p>



<h3 class="wp-block-heading">Should I use Sealed Secrets or External Secrets Operator?</h3>



<p class="wp-block-paragraph">If you already run Vault, AWS Secrets Manager or an equivalent, use External Secrets Operator: you get rotation without commits and a proper audit trail. If you have no secret store and no appetite to run one, Sealed Secrets is a faster path with fewer moving parts. The trade-off is that Sealed Secrets has no rotation story and its keys are per-cluster.</p>



<h3 class="wp-block-heading">What happens if I lose the Sealed Secrets private key?</h3>



<p class="wp-block-paragraph">Every <code>SealedSecret</code> in your repository becomes permanently undecryptable, and you regenerate every credential from source. This is the single most common way this setup fails in practice. Back the key up somewhere outside the cluster and the repo, and test restoring it.</p>



<h3 class="wp-block-heading">Do I still need to rewrite Git history after rotating a leaked credential?</h3>



<p class="wp-block-paragraph">Often not. Once the credential is revoked it cannot be used, which usually resolves the actual risk. Rewriting is worth the disruption when the value is not rotatable, when it is personal data rather than a credential, or when a compliance process requires it. Weigh it against breaking every open pull request and forcing everyone to re-clone.</p>



<h3 class="wp-block-heading">Can Argo CD decrypt SOPS files on its own?</h3>



<p class="wp-block-paragraph">Not natively, unlike Flux, which has SOPS decryption built into its Kustomize controller. With Argo CD you add a config management plugin such as KSOPS or the Argo CD Vault Plugin. Check the current Argo CD documentation for the supported plugin mechanism before building around it.</p>



<h3 class="wp-block-heading">Is it safe to commit the Sealed Secrets certificate or the age public key?</h3>



<p class="wp-block-paragraph">Yes, and committing them is the point. Both are public halves of an asymmetric pair and can only encrypt. Putting them in the repo is what lets developers seal new secrets without cluster access. The private halves never go anywhere near Git.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">If you take one idea away, make it this: the work to <strong>keep Kubernetes secrets out of Git</strong> and the work to keep them safe in the cluster are two separate jobs, and finishing the first one feels a lot like finishing both. It is not. A repo full of <code>SealedSecret</code> resources sitting in front of an unencrypted etcd with permissive RBAC has moved the exposure, not removed it.</p>



<p class="wp-block-paragraph">Pick the approach that matches what you already run. Back up the key on day one, not after the first rebuild. Turn on encryption at rest. And when something does leak, rotate before you reach for <code>filter-repo</code>, because rotation is the part that actually stops the bleeding.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help sorting out secrets in your cluster?</h2>



<p class="wp-block-paragraph">This is work I do regularly, and most of it is less dramatic than it sounds once someone has done it before. Things I can help with:</p>



<ul class="wp-block-list">
<li>Auditing a GitOps repo for committed credentials, including full history, and producing a prioritised rotation list rather than a wall of scanner output</li>

<li>Setting up Sealed Secrets, SOPS with age, or External Secrets Operator against Vault or a cloud secret manager, including the key backup and restore procedure</li>

<li>Migrating an existing repo off plaintext or off a scheme that no longer fits, without a big-bang cutover</li>

<li>Fixing Argo CD or Flux reconciliation loops caused by secret controllers and API version conversions</li>

<li>Enabling and verifying etcd encryption at rest, and tightening RBAC on secrets across namespaces</li>

<li>Adding pre-commit and CI secret scanning that people will not immediately disable because of false positives</li>
</ul>



<p class="wp-block-paragraph">If you have a controller log, a stuck <code>ExternalSecret</code>, or a scanner report you are not sure how to triage, send it over and I will tell you what I make of it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/keep-kubernetes-secrets-out-of-git/">Keep Kubernetes Secrets Out of Git Without Kidding Yourself</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
